To homepage To homepage
Solved

Can we get access to the live usage API?

Gary
Rank
Userlevel 2
  • Gary
  • Carbon Cutter****
  • 7 replies

Hi Ovo,

I wondered whether the API that powers your live and historical usage page in the account menu is something you could look into opening up a bit so we can freely access our raw data and also perhaps it may stimulate some interesting community projects?

For instance (more sport related), Strava and Fitbit.

icon

Best answer by g-de 27 August 2020, 19:02

can you help me understand if the links above allow users to access raw data as if from an API?

I can certainly give it a go.

 

Firstly, big thanks to Timmo and other contributors for developing the library. All thanks go to them and links to buy them a coffee for their effort can be found on PyPi or GitHub:

https://pypi.org/project/ovoenergy/
https://github.com/timmo001/ovoenergy

 

I assume the way the library was developed this was to reverse engineer the My OVO website. That is, browse the website as normal and and monitor the traffic to see what OVO URLs and APIs it uses. Assuming this is the case, one of these URLs (get account IDs) already seems to be different but the library is still working for the moment. The APIs being used are not officially supported outside of the website, so first, a warning that any changes to the My OVO website may break the library.

 

For anyone using Python, you should be able to include the ovoenergy library and look at https://github.com/timmo001/ovoenergy/blob/master/ovoenergy/cli.py for how to use it.

 

For replicating this in other languages, the process seems to be as follows:

  1. Login by sending a POST request to https://my.ovoenergy.com/api/v2/auth/login with the following JSON in the body: 
    {
        "username": "...",
        "password": "...",
        "rememberMe": true
    }

    From the response, save the cookies to use in all future requests.

  2. Next, send a GET request to https://smartpaym.ovoenergy.com/api/customer-and-account-ids . Be sure to save the items in the “accountIds” property (usually only one item).
  3. Finally, make a GET request to https://smartpaym.ovoenergy.com/api/energy-usage/daily/{accountId}?date={yyyy-MM} where {accountId} is the value from the previous request and {yyyy-MM} is the month in the format 2020-08.

You can also get monthly or half-hourly usage using the following URLs similar to step 3:

As mentioned above, if this is wanted for Home Assistant, the integration was added in version 0.114 (https://www.home-assistant.io/integrations/ovo_energy/).


Unfortunately the data only goes up to the end of the previous day as OVO no longer provide real-time data:

 

View original

203 replies

T
Rank
Userlevel 3

Further to @ErTnEc's git contribution, I have updated this to do the V2 API for historic consumption. I run the following script daily to download the previous day's half-hourly consumption into the MySQL DB that underpins my NodeRED installation: load_readings.py.
 

 

The new online interface has broken this unofficialt API :(  @Tim_OVO , any traction with the dev team.   Under the old scheme, the cookies were generated server-side, so you could simply script the logon, plus the API request. Now some of the session cookies are generated in obfuscated JS code client-side.  I can log on with the browser and capture these session cookies to part automate this, but this is a total PITA. 

Any chance of opening a dialogue with one of the dev team?  There is no reason why OVO could publish a Python script to make this API easily accessible to advanced users.

Eva_OVO
Rank
Userlevel 6

We’ll take that away, @TerryE, and get a response from our dev team. We’ll be in touch when we have more info! :)

My friend told me how electricity is measured and I was like Watt!
T
Rank
Userlevel 3

I’ve just spent another 4 fours re-engineering the Smart Meter API, and updated my python automatic script.

Anyone that is interested can use this python script; just update the user name and password.  This queries against the new API and unlike the previous version which output to a MySQL schema, this simply produces a CSV so users can also paste the O/P into Excel, Calc or whatever.  I run this as a cmd from NodeRED and parse the response in a NodeRED function.  Enjoy.  :relaxed:

@Eva_OVO this is unlicensed Open Source and my original work.

Tim_OVO
Rank
Userlevel 7

Update: I’m still looking into this, @TerryE  - what we can offer, what we can’t and why. It would be good for everyone interested to get some clarity. Leave this with me!

Carbon neutral - we need a community to get there!
M
Rank
Userlevel 1

Just make an official API already. Sorry to get short but people hacking their way to relevant data shows there is a need for it and you’re an energy company, I doubt you’re getting rich cos of your apps... 

https://segment.com/blog/why-we-re-transitioning-to-api-first/

Tim_OVO
Rank
Userlevel 7

I’ve already been speaking to the dev team this week (most recently yesterday) about what is required, how feasible, and when etc  @mattc 

 

I’ll be able to get an update shared next week. Thanks for bearing with - this query has not been forgotten. :)

Carbon neutral - we need a community to get there!
Tim_OVO
Rank
Userlevel 7

I’ve got our official stance on this from our dev team:

At the moment I’m afraid that we’re not planning to be offering something like this in the near future. It’s quite a complex undertaking and our product teams are focused on building a great online experience and working with our members towards our Plan Zero initiative. When this changes in the future we’ll announce it on this topic first.

I’ll stick this as the ‘best answer’ so others know. 

Thoughts below please! 

Carbon neutral - we need a community to get there!
S
Rank
Userlevel 1

That’s a great shame.  Ovo used to be leading in the area of engaging digitally with its customers and this seems to be a backwards step.

M
Rank
Userlevel 1

I am also working towards a plan zero initiative:

I’d like to be able to track energy usage through the day, not just a daily, weekly, monthly graph so I know where to optimize.

I’d like to be able to create charts using something like Graphana and map the energy data against light levels, temperature, humidity and other atmospheric conditions so I know when there are anomolies.

I’d like to combine the energy usage data with my smart thermostat to better understand the relationship between temperature and cost.

I’d like to set up an alert the moment my smart meter stops posting data so that I’m not sent an email months later requesting a manual submission.

I am not alone having these ideas and being capable of implementing them. But I will not scrape the data. I will simply move to a provider who does support an API.

F
Rank

That is really disappointing to hear, that you’re not going to make it easy for us to get our data from your system. 

 

I don’t think anyone is wanting anything particularly complex. a url which we can hit, which gives us the last 30 days of data (ideally with something like a 5 minute granularity) would be perfect. No requirement to go for an oauth tie-in or anything like that. A simple long token which can be reset. 

 

 

Either that, or a way to monitor our smart meters directly. I’ve heard something about them using zigbee, though I have no idea how valid that is. Any thoughts on that?

T
Rank
Userlevel 3

 

At the moment I’m afraid that we’re not planning to be offering something like this in the near future. It’s quite a complex undertaking

 

Thoughts below please! 

I’ve just spent another 4 fours re-engineering the Smart Meter API, and updated my python automatic script.

Anyone that is interested can use this python script; just update the user name and password.  This queries against the new API and unlike the previous version which output to a MySQL schema, this simply produces a CSV so users can also paste the O/P into Excel, Calc or whatever. :relaxed:

 

Anyone unhappy with the OVO developers position is free to use my Python script which wraps the normal Web interface to achieve the same.

A
Rank

I’m just looking at switching energy providers and access to this data is a prerequisite.

If this data exists then surely there is a right to be able to access it (and not a simplified form) under GDPR?
I’ve got a Chameleon Technology IHD6-CAD-PPMID smart meter which can send the real time data via wifi.
Bulb offer an integration with smarthings suggesting that the realtime data is accessible.
But I’m hoping to find another provider that are slightly cheaper. Hence why I looked at OVO.

 

ThePaulAdams
Rank
Userlevel 1

For those interested in something like this being easily accessible I’ve made a terrible walkthrough of how to do so. hope you don’t mind me posting this Tim.

https://github.com/ThePaulAdams/OvOEnergy

https://www.thepauladams.com/blog/2020/02/24/ovo-energy-integration-to-smart-home-dashboard/

https://github.com/ThePaulAdams/OvOEnergy/
uvarvu
Rank

@TerryE I tried to run your script but it’s erroring for me - something is wrong with the authentication.   Does the password field need to be in a certain format?   I left all of the quotes in place.

 

pi@raspberrypi:~/Documents $ python2 energy.py 
Date,00:00,00:30,01:00,01:30,02:00,02:30,03:00,03:30,04:00,04:30,05:00,05:30,06:00,06:30,07:00,07:30,08:00,08:30,09:00,09:30,10:00,10:30,11:00,11:30,12:00,12:30,13:00,13:30,14:00,14:30,15:00,15:30,16:00,16:30,17:00,17:30,18:00,18:30,19:00,19:30,20:00,20:30,21:00,21:30,22:00,22:30,23:00,23:30
{"message":"No refresh token cookie or authorization header sent","redirect":"https://my.ovoenergy.com"}
Traceback (most recent call last):
  File "energy.py", line 86, in <module>
    main()
  File "energy.py", line 78, in main
    ovo = OVOreadings(USERNAME, PASSWORD)
  File "energy.py", line 50, in __init__
    self.account = json.loads(resp.content)['accountIds'][0]
KeyError: 'accountIds'
pi@raspberrypi:~/Documents $

 

Transparent
Rank
Userlevel 7
Badge +2

Can I point out that the “new” Billing Software, which is the connection point to our Smart Meters, is still undergoing development and bug-eradication. Any script which tries to lift data from the online Usage pages will need to be continually updated to match the current Billing Software version.

To make it even more tricky, it’s becoming apparent that not all customers are running the same software version. See this discussion about the Zero-in option.

My current Online Usage interface has changed further since that discussion. Not only do I not have a Zero-in Option, but there is no longer the ability to choose Yearly or Monthly from the Usage entry on the let menu.

 

I’m afraid we’re chasing a moving target.

Save energy... recycle electrons!
T
Rank
Userlevel 3

… Any script which tries to lift data from the online Usage pages will need to be continually updated to match the current Billing Software version.

To make it even more tricky, it’s becoming apparent that not all customers are running the same software version. See this discussion about the Zero-in option.

 

Yes any script will need to be updated as the online interface changes, but I have only updated mine once in the last 3 years., and this latest one still works for me.  I don’t know about zero-in and am not interested as I have a near zero-energy new build and track all of my energy use through my own home automation system.

As to @uvarvu ‘s Q I use a password keeper / generator for my online passwords which generates strong passwords and these will often need quoting using the r’’’password’’’ syntax as below:

 PASSWORD = r'''D#adR'xGFj{Ks";=$'''

Also note that I have an electricity-only supply and the script may need tweaking for multiple supplies.

G
Rank
Userlevel 1

If anyone else is still looking for this, Home Assistant just added support for OVO in the latest release (0.114). Details on the integration can be found here: https://www.home-assistant.io/integrations/ovo_energy/

This uses the “ovoenergy” library which can be found here: https://pypi.org/project/ovoenergy/

The only problem is data is only available from OVO up to the previous day. I suspect there is some overnight batch process to format the data for the frontend.

Edit: Actually it seems OVO turned off real-time data years ago as it was “unreliable”:

 

Tim_OVO
Rank
Userlevel 7

Brilliant post here, @g-de - can you help me understand if the links above allow users to access raw data as if from an API?

 

As you can probably tell, I am not very computer literate. I am however decent at communities, so this has sparked my interest in that I feel it might help potentially everyone who has commented on this topic. Give us some more info, and I might need to re-assign the ‘best answer’ to your topic.  :) 

Carbon neutral - we need a community to get there!
Transparent
Rank
Userlevel 7
Badge +2

Well we could ask @Jequinlan to interpret for you @Tim_OVO :wink:

Save energy... recycle electrons!
G
Rank
Userlevel 1

can you help me understand if the links above allow users to access raw data as if from an API?

I can certainly give it a go.

 

Firstly, big thanks to Timmo and other contributors for developing the library. All thanks go to them and links to buy them a coffee for their effort can be found on PyPi or GitHub:

https://pypi.org/project/ovoenergy/
https://github.com/timmo001/ovoenergy

 

I assume the way the library was developed this was to reverse engineer the My OVO website. That is, browse the website as normal and and monitor the traffic to see what OVO URLs and APIs it uses. Assuming this is the case, one of these URLs (get account IDs) already seems to be different but the library is still working for the moment. The APIs being used are not officially supported outside of the website, so first, a warning that any changes to the My OVO website may break the library.

 

For anyone using Python, you should be able to include the ovoenergy library and look at https://github.com/timmo001/ovoenergy/blob/master/ovoenergy/cli.py for how to use it.

 

For replicating this in other languages, the process seems to be as follows:

  1. Login by sending a POST request to https://my.ovoenergy.com/api/v2/auth/login with the following JSON in the body: 
    {
        "username": "...",
        "password": "...",
        "rememberMe": true
    }

    From the response, save the cookies to use in all future requests.

  2. Next, send a GET request to https://smartpaym.ovoenergy.com/api/customer-and-account-ids . Be sure to save the items in the “accountIds” property (usually only one item).
  3. Finally, make a GET request to https://smartpaym.ovoenergy.com/api/energy-usage/daily/{accountId}?date={yyyy-MM} where {accountId} is the value from the previous request and {yyyy-MM} is the month in the format 2020-08.

You can also get monthly or half-hourly usage using the following URLs similar to step 3:

As mentioned above, if this is wanted for Home Assistant, the integration was added in version 0.114 (https://www.home-assistant.io/integrations/ovo_energy/).


Unfortunately the data only goes up to the end of the previous day as OVO no longer provide real-time data:

 

Jequinlan
Rank
Userlevel 6
Badge +1

This is absolutely brilliant all. I shall look and see if i can use this over next month or so to see if it can help model a TOU tariff price modler. Awesome!

John Quinlan
Jequinlan
Rank
Userlevel 6
Badge +1

So I have installed the HomeAssistant on a virtual Pi and am indeed able to get some OVO data! As I am not a programmer, actually getting the data from the API isn’t my expertise but if someone can help then I AM the right (type of) person to be taking that data and doing something useful with it. @g-de can you help?

 

Thanks.

John Quinlan
Transparent
Rank
Userlevel 7
Badge +2

And whilst we’re at it, let’s tag @NinjaGeek too. He might want to playing with this as well.

Save energy... recycle electrons!
Tim_OVO
Rank
Userlevel 7

Wowza this is great. @g-de I’m going to make your latest comment the ‘best answer’, which will mean it appears at the top of this topic below the question. Good to see @Jequinlan willing and able to support! 

 

I will be checking in with our devs though, to see if there’s any implications for customers using this. It might not be something we can stop, but it might not be something we want to promote. The fact that I’m unaware just shows how limited my knowledge is. 

 

Either way, a brilliant post sharing some great work that’s aimed at helping others to monitor their energy usage (and other use cases?). That’s what this community is allll about :)

Carbon neutral - we need a community to get there!
G
Rank
Userlevel 1

So I have installed the HomeAssistant on a virtual Pi and am indeed able to get some OVO data! As I am not a programmer, actually getting the data from the API isn’t my expertise but if someone can help then I AM the right (type of) person to be taking that data and doing something useful with it. @g-de can you help?

 

Thanks.

@Jequinlan Unfortunately I don’t think I’ll be much help on this. I’m not too familiar with Python, only enough to be able to read it and do basic stuff and my home assistant experience is all around setup and automation rather than data science. I suspect the data collection in home assistant may be too brief for this purpose (by default it only keeps 3 days I think).

That being said, there’s a data science section on the Home Assistant website which may be of use: https://data.home-assistant.io/

Alternatively, someone here can probably create a small python script using the library to output the data to CSV. Should make it fairly easy to import into something else then.

 

Wowza this is great. @g-de I’m going to make your latest comment the ‘best answer’, which will mean it appears at the top of this topic below the question.

Glad I could help!

 

I will be checking in with our devs though, to see if there’s any implications for customers using this. It might not be something we can stop, but it might not be something we want to promote.

I think the main issues would be the API changing unexpectedly and storage of the username and password. If OVO were to have provided some kind of developer API it would have allowed use of tokens and permissions to be configured for read-only access instead. Much more secure but quite a bit of dev and management effort for OVO.

It is unsupported and rather hacky so the choice will probably be to not promote it. Preventing it would be rather difficult. The open source community always seems to find a way around any restrictions :slight_smile:

Reply


@OVOEnergy latest: