Solved

Can we get access to the live usage API?



Show first post

204 replies

hi, Guys,

May I request access. Part of the reason I joined OVO was this very forum article. Whilst I am not on-board yet, on Oct 5 I switch and Oct 20th is my smart meter installation, I am keen to get access to this data as I am looking to create multiple apps that I can query with Alexa, integrate with home-assistant etc. I am keen to try and pre-empt energy usage based on weather, historical heating settings from the thermostat etc. and Shift hours. And finally, break down energy usage by correlating the information from my smart plugs.

There are probably loads of ideas out there for this type of data and it is great to see OVO embracing this way of working. As more third-party products come out with their own exposure, OVO will be seen as a leader and will hopefully bring more customers to you as a side benefit.

Can't wait to see what comes of this.

Chris
Userlevel 3
@ErTnEc, this is exactly what I want to be able to do. I use NodeRED to poll darksky.net for current weather readings. Can you send me your python script to play with?

Sorry I've had a hectic time lately and not had chance to pick this back up.

https://gist.github.com/PaulW/7f3ce29c3469757c26504b572597a45c

I still need to add in some extra error handling (specifically around initial auth requests as it takes a while for the authtoken to be fully active it seems) but it's a fairly simple process which people can easily build upon if need be.
+1 for this – or at least being able to download your raw data to be able to analyse and explore ourselves. Would be really useful and interesting.
Userlevel 7
Badge +5

Well we could ask @Jequinlan to interpret for you @Tim_OVO :wink:

Userlevel 1

can you help me understand if the links above allow users to access raw data as if from an API?

I can certainly give it a go.

 

Firstly, big thanks to Timmo and other contributors for developing the library. All thanks go to them and links to buy them a coffee for their effort can be found on PyPi or GitHub:

https://pypi.org/project/ovoenergy/
https://github.com/timmo001/ovoenergy

 

I assume the way the library was developed this was to reverse engineer the My OVO website. That is, browse the website as normal and and monitor the traffic to see what OVO URLs and APIs it uses. Assuming this is the case, one of these URLs (get account IDs) already seems to be different but the library is still working for the moment. The APIs being used are not officially supported outside of the website, so first, a warning that any changes to the My OVO website may break the library.

 

For anyone using Python, you should be able to include the ovoenergy library and look at https://github.com/timmo001/ovoenergy/blob/master/ovoenergy/cli.py for how to use it.

 

For replicating this in other languages, the process seems to be as follows:

  1. Login by sending a POST request to https://my.ovoenergy.com/api/v2/auth/login with the following JSON in the body:
    {
    "username": "...",
    "password": "...",
    "rememberMe": true
    }

    From the response, save the cookies to use in all future requests.

  2. Next, send a GET request to https://smartpaym.ovoenergy.com/api/customer-and-account-ids . Be sure to save the items in the “accountIds” property (usually only one item).
  3. Finally, make a GET request to https://smartpaym.ovoenergy.com/api/energy-usage/daily/{accountId}?date={yyyy-MM} where {accountId} is the value from the previous request and {yyyy-MM} is the month in the format 2020-08.

You can also get monthly or half-hourly usage using the following URLs similar to step 3:

As mentioned above, if this is wanted for Home Assistant, the integration was added in version 0.114 (https://www.home-assistant.io/integrations/ovo_energy/).


Unfortunately the data only goes up to the end of the previous day as OVO no longer provide real-time data:

 

I’m just looking at switching energy providers and access to this data is a prerequisite.

If this data exists then surely there is a right to be able to access it (and not a simplified form) under GDPR?
I’ve got a Chameleon Technology IHD6-CAD-PPMID smart meter which can send the real time data via wifi.
Bulb offer an integration with smarthings suggesting that the realtime data is accessible.
But I’m hoping to find another provider that are slightly cheaper. Hence why I looked at OVO.

 

Userlevel 3
@SianiAnni At the moment, I cludged this by running the query in my browser after creating my session interactively and then using a small perl script to upload this into my nodered db.

I haven' bother to retro-engineer the sequence of POST requests to establish fresh set of session cookies for each AJAX callback to work as retro-engineering this is all a bit of a dog, and iIt could take me a day to do this so there isn't the payback: one day's work to save a minute or so every month.

Which is why we really need to wait for the proper API if you want to automate this fully.
Userlevel 3

Advanced programmers could no doubt script the whole session connection to recover the session cookies. I am one, but I haven't bothered.


Further to @ErTnEc's git contribution, I have updated this to do the V2 API for historic consumption. I run the following script daily to download the previous day's half-hourly consumption into the MySQL DB that underpins my NodeRED installation: load_readings.py.

There's an issue with NodeRED HTTPrequest which causes logon problems so I haven't got the native NodeRED version working. Sorry. It's not worth my spending time on this because I will moving supplier when my contract ends. I really like OVO, but a 30+% price hike is just too much, when other suppliers offer better deals.
Userlevel 3
works a treat.
@jdenver, If you check, I've updated the script slightly. What I've found is that the Smart Meter sends its SMS message once a day about 02:00, so the consumption figures for the previous day are only available after that. The message occasionally fails for whatever reason, so you can miss a day, but it does retry at some point. So what I've done is to query the max DTS from my readings table and set the FROM to 30 mins after the last end. That way, the reading will catch up and you don't end up dropping data in your table.

I am note sure why the NodeRED native version fails to authenticate, but it is just easier to trigger the flow by a cron injector (actually I use node-red-contrib-pkjq as this is better maintained) and then exec the python script.
Userlevel 7
Badge +2

So I have installed the HomeAssistant on a virtual Pi and am indeed able to get some OVO data! As I am not a programmer, actually getting the data from the API isn’t my expertise but if someone can help then I AM the right (type of) person to be taking that data and doing something useful with it. @g-de can you help?

 

Thanks.

Userlevel 7

@Jequinlan I will have a read of this, but without an OVO account I won’t be able to test :pensive: Sounds like amazing customer-led action though!

 

Please be advised that from OVO’s point of view, I neither condone or condemn this workaround. It’s untested and possibly insecure, either way it’s not made alongside OVO. However I know there's been lots of interest, and (whilst OVO/Kaluza developers are tabling an idea that will be of interest to everyone here in November) so far we haven't been able to deliver anything in house. 

 

So I’m tagging everyone that's commented here to make sure you're aware of this work from @Jequinlan 

 

Let us know how it goes please! 

 

@Gary @GarethMarlow @rgphilpott @atilagiant @Deanohol @Peter_Mat @didge05 @sam_cat @jmontalvo @adec @Pramsbottom @ryck @CeeJayCee @alexbaulch @Random @maccordingley @sfgreenwood @johnk @jadam @Fordy @Steveo67 @Siggy0 @CerebralDigits @deanjames @ChrisB @jadam @DaveJenYearsley @ErTnEc @Marshcroft @jordanharper @Simonfea @obiwanzamora @AdamJ @dorinho @Ross2411 @Chrherms @MikeWilliams @blakedrayson @TerryE @VibroAxe @jdenver @maurorappa @adec @100903609 @Riscy @r3m0dul8 @MyZoe.Uk @Ajk @AlanN51 @CloudCircus @r3m0dul8 @technogeoff @resoai @Fuzzysteve @mattc @andsee @uvarvu @Transparent @g-de 

Userlevel 3
@jdenver, I have a passive-class house which uses electricity only, so I haven't needed to work out how the API works for gas MPRNs.

The logon function established a new session with the OVO server and collects the valid session cookies for your account in the returned Session object. The getMPAN function issues a /api/paym/accounts request which returns a JSON object containing all of your account info.

If you use the debugger mode in FF or Chrome, then you can walk this object interactively to see the field that you need to pick. I infer from what @SianiAnni says, that there is a corresponding mprn field that you could access in line 67 as data[0]['consumers'][0]['mprm'] to fetch your MPRN, and that this can be substituted in the remaining code to get your gas half-hourly figures. (Perhaps she could clarify.)

As I said, I don't have a gas meter so I can't check this myself.

Hope this helps ☺️
Userlevel 4
Badge

@Tim_OVO Given that some of Ovo’s customers have been asking for this and more for many years, I think @Jequinlan deserves a big round of applause for sharing his code. I agree that without the backing of Ovo, it should be used with a degree of caution. It could just stop working one day, when Ovo next change their api? interface, as happened in the Hive community recently. 

Userlevel 7
Badge +2

@Tim_OVO Given that some of Ovo’s customers have been asking for this and more for many years, I think @Jequinlan deserves a big round of applause for sharing his code. I agree that without the backing of Ovo, it should be used with a degree of caution. It could just stop working one day, when Ovo next change their api? interface, as happened in the Hive community recently. 

Yes, if the api changes it could stop, but also the apis are soft coded in the config file so if a trivial change the config will just need updating. The risk of any 3rd party integration exists if any 1st party changes an apinin any situation.

 

If ovo want to see the code , then i am happy to discuss.

Userlevel 1
Hey Tim, I am the creator of MyZoe.UK and an EV Driver I'de love this information to combine with my EV Tracking.

I'll PM you in the hopes of too being added to this list.
Userlevel 3

Further to @ErTnEc's git contribution, I have updated this to do the V2 API for historic consumption. I run the following script daily to download the previous day's half-hourly consumption into the MySQL DB that underpins my NodeRED installation: load_readings.py.
 

 

The new online interface has broken this unofficialt API :(  @Tim_OVO , any traction with the dev team.   Under the old scheme, the cookies were generated server-side, so you could simply script the logon, plus the API request. Now some of the session cookies are generated in obfuscated JS code client-side.  I can log on with the browser and capture these session cookies to part automate this, but this is a total PITA. 

Any chance of opening a dialogue with one of the dev team?  There is no reason why OVO could publish a Python script to make this API easily accessible to advanced users.

Userlevel 7
Badge +2

I Whatsapped my friend, a programmer, who likes little challenges, he is just getting back from holiday but read this post and reckons its under an hours work so it taking it on for me as a customisable and sharable dev! Fingers xed we will all ve able to download and analyse the info in detail!

Userlevel 7
Badge +2

@Transparent we work on helping each other ;)

Userlevel 7
Badge +5

Well in that case it’s :beers:  :tropical_drink:   :wine_glass:  :champagne:  :champagne_glass:  

Userlevel 6

We’ll take that away, @TerryE, and get a response from our dev team. We’ll be in touch when we have more info! :)

Userlevel 3

I’ve just spent another 4 fours re-engineering the Smart Meter API, and updated my python automatic script.

Anyone that is interested can use this python script; just update the user name and password.  This queries against the new API and unlike the previous version which output to a MySQL schema, this simply produces a CSV so users can also paste the O/P into Excel, Calc or whatever.  I run this as a cmd from NodeRED and parse the response in a NodeRED function.  Enjoy.  :relaxed:

@Eva_OVO this is unlicensed Open Source and my original work.

Userlevel 7
Badge +2

On a side note, a v2 with fixed year field coming next week.

Gave up when I saw the file extension, was hoping for something written that I could run on my Pi.   I tried to decompile it using Visual Studio but didn’t get far.

Sorry for another spamming comment but I’m keen to hear if others have checked out this work done by @Jequinlan this week? 

 

@Gary @GarethMarlow @rgphilpott @atilagiant @Deanohol @Peter_Mat @didge05 @sam_cat @jmontalvo @adec @Pramsbottom @ryck @CeeJayCee @alexbaulch @Random @maccordingley @sfgreenwood @johnk @jadam @Fordy @Steveo67 @Siggy0 @CerebralDigits @deanjames @ChrisB @jadam @DaveJenYearsley @ErTnEc @Marshcroft @jordanharper @Simonfea @obiwanzamora @AdamJ @dorinho @Ross2411 @Chrherms  @blakedrayson @TerryE @VibroAxe @jdenver @maurorappa @adec @100903609 @Riscy @r3m0dul8 @MyZoe.Uk @Ajk @AlanN51 @CloudCircus @r3m0dul8 @technogeoff @resoai @Fuzzysteve @mattc @andsee @uvarvu  @g-de 

 

 

No longer with Ovo, but as a sysadmin I wouldn't trust a compiled exe distributed via gdrive.

That in itself is very bad practice.

Source needs publishing through something like github, and you should host the exe under the releases section of some of that. 

Without it, no one can trust the security of it, especially when you have to put account information in it. 

Userlevel 7

Update: I’m still looking into this, @TerryE  - what we can offer, what we can’t and why. It would be good for everyone interested to get some clarity. Leave this with me!

Reply