Solved

Ihd 6 connectivity


Userlevel 1

I have connected my Chameleon IHD 6 wirelessly to my home network but can I download data from the IHD to my pc?

 

icon

Best answer by Blastoise186 6 October 2021, 22:32

View original

21 replies

Userlevel 7
Badge +5

Ooohhh! CAD-Only SMETS2 device huh? Hmm… Tempting…

Given how overpowered my home network is these days, I wonder if this could allow me to chuck one of those new gadgets into a corner where it’s in range of both Raichu and Mega Exploud, so that I can actually use CAD features in my flat...

Userlevel 6
Badge +1

@Peter Rue@Simon1D@Blastoise186@Jess_OVO: there is another way to get your energy usage.

Glowmarkt (glowmarkt.com) sell CAD devices that can be connected to your smart meter to fetch usage for you. Except that they don’t because demand has outstripped supply, and you can’t get devices from the shop. The smart meter data they provide is available to integrate with various products, through their web application and from an API.  The data from their CAD devices is real-time i.e. you see live usage as well as historical data. I don’t have any experience with their devices.

Glowmarkt also have a free service where you can sign up to access your smart meter data straight from the smart meter network into their web application.  Glowmarkt got accredited to access the smart meter network in order to provide this service.  The data from this service isn’t real-time but is within 30 minutes.  I’ve signed up to this service, seems to work really well. I don’t know how they make money out of this, but it’s a great service.

I wrote some code which accesses the Glowmarkt service (https://github.com/cybermaggedon/pyglowmarkt), and GitHub user danmed worked out how to run that in Home Assistant (https://github.com/danmed/Glow2MQTT).  This is probably only useful to you if you are already a Home Assistant user (or want to become one, it’s a great project :grinning: ).

 

You can join a waiting list for their IHD/CAD devices. They had a few available recently for people on the waiting list.

 

They will also shortly be releasing a CAD only SMETS2 device if you just want to access the data and not have another IHD. 

Userlevel 7
Badge +5

No worries, you may want to consider applying a license to your repository as well. It helps with allowing others to contribute and takes care of some legal stuff - otherwise you’ve got full copyright by default over the code and basically nobody else is ever allowed to touch it, not even to help contribute towards it. To do that, please head over to https://choosealicense.com/ and take a look at the options available. I use The Unlicense for stuff that I’m basically releasing into the Public Domain permanently and irrevocably, but you’re welcome to choose whatever licence you feel comfortable with - and especially one that fits with what you’re happy to allow.

Once you’ve done that, create a file in the root of your repository called LICENSE or LICENSE.md (it must be exactly one of those two or it won’t work). When you’ve done this, click Choose a License Template, pick the license you’ve chosen from the list, enter a few details and submit them. Once that’s done, commit the file and you’re all done!

Oh, and not setting a license can also come back to bite you later!

Userlevel 7
Badge +1

Cheers, @Blastoise186 - comments much appreciated!

The only recurring element would be an at most daily update but, nevertheless I'd like to be on my best behaviour there, too.

BW

Userlevel 7
Badge +5

I’m afraid coding stuff isn’t really one of my skills and pretty much the only things I’m capable of are pretty simply stuff - and I really do mean simple. Putting together even a few basic scripts for this particular repository was hard enough! Yes, I know GitHub isn’t technically meant to be used as a dumping ground for IT Admin type scripts, but I needed somewhere to put them! XD

With that being said however, it’s not ideal to be using Selenium or WebDrivers to scrape the HTTP/HTML version of MyOVO, since you’ll have a lot of potential junk to wade through and it’s quite load generating on the servers with no real benefit. I’d recommend against using that sort of thing for use cases that are entirely automated like this one, unless you’re testing stuff. Selenium and WebDrivers are after all, primarily intended for CI/CD use cases.

If you take a look at this PR from a few months ago, you should be able to get an idea as to how Home Assistant uses the API - and you should be able to find the community maintained integration from there, which will most likely help you to figure out how to improve your script. Ideally, you’ll want to minimise how much interaction you do with the HTTP/HTML side of things and lean more towards the raw API to help reduce server load. It might also give you some inspiration on how to rate limit things as well.

I’m not sure if there’s a preferred rate limit speed that OVO would consider reasonable for polling the API though. Perhaps Tim can ask around about that, since you’ll definitely want to play nice here - knowing what the reasonable limits are would definitely allow you to configure your script to stay within them.

As for the README.md file, that’s written in Markdown, basically a text file with fancy text formatting more than anything else - and anything in there cannot be executed as code. It’s not linked to any coding languages either and is present in the repository more just to explain what the repository is about. Likewise, the LICENSE file in some of my repositories allows GitHub to easily detect what license I’ve placed on them. Pretty cool huh? Any file named README.md will also be on the landing page of the repository as well, 

There’s also others too, like CONTRIBUTING.md which would set a contributions policy that GitHub would recognise, or SECURITY.md which would set a security policy, again automatically detected by GitHub. Any file in the repository that’s correctly named with one of these special names and located in the root, .github/ or .docs/ directories will be recognised as being special instructions somehow and GitHub will try to process it as such as long as it’s valid. There’s a few others too, but I’ll let you discover those for yourself. :)

Userlevel 7
Badge +5

That ReadMe file from @Simon1D is very much reduced from the version I read a few days ago.

As it’s just raw text I assume it isn’t constricted by Python :wink:

Userlevel 7

Nice one, @Simon1D!

 

...Depending on your technical skills you might also be interested in this unofficial guide to accessing your usage data via our API.

The thread @Jess_OVO links to is mostly 4 years old, although @Jequinlan did add a link there to his own executable, within the last year. (I don’t run windows, so that wasn’t an option for me.) I was able to get something more explicit (source code :-) to work earlier this year, and today (whoops - yesterday) I implicitly promised to provide up to date info.

In that spirit and in case it might help anyone else, I’ve posted my Python script here. I would welcome any comments, e.g. from @Blastoise186, regarding matters of security, and also on the demands it might place on Ovo’s servers (I’ve tried to make the script reasonably gentle - no worse than human interaction might be).

 

I’m keen to hear the opinion of @Blastoise186 on Simon’s Python work as well! :snake:

Userlevel 7
Badge +1

...Depending on your technical skills you might also be interested in this unofficial guide to accessing your usage data via our API.

The thread @Jess_OVO links to is mostly 4 years old, although @Jequinlan did add a link there to his own executable, within the last year. (I don’t run windows, so that wasn’t an option for me.) I was able to get something more explicit (source code :-) to work earlier this year, and today (whoops - yesterday) I implicitly promised to provide up to date info.

In that spirit and in case it might help anyone else, I’ve posted my Python script here. I would welcome any comments, e.g. from @Blastoise186, regarding matters of security, and also on the demands it might place on Ovo’s servers (I’ve tried to make the script reasonably gentle - no worse than human interaction might be).

Ignore the other files in that repository (for now, at least) they are all very much in active development but should settle down a bit over the coming weekend.


 

Userlevel 7
Badge +1

… The data from this service isn’t real-time but is within 30 minutes.  I’ve signed up to this service, seems to work really well. I don’t know how they make money out of this, but it’s a great service. ...

Interesting - thanks for the concise summary, @cybermaggedon.

I too signed up to let Hildebrand get at my data because I find the Bright app handy. As with all these things (gmail, facebook, etc.) there’s money to be made from knowing what everyone does, even at a statistical level. (The big players will claim publicly that they can’t identify individuals from patterns of usage. Politicians might accept those assurances, but it’s not true.)

These days (as I’m sure you know all too well) when a useful service is free, that means the service provider is happy to accept you (or your data) by way of payment.

(There used to be a joke, there’s no such thing as a free lunch. The joke version is, the only free lunch is our universe, in the quantum fluctuation/inflationary/big bang cosmology.)

Userlevel 7
Badge +1

PPS What I’ve done is using an old-school approach, reading all the data into memory and processing it as efficiently as I can, and this is how I’ll implement the estimation of meter readings too. It appears that Ovo’s systems use Event Streams, and there are some things that really are a little awkward (harder to do, perhaps impossible, really) when the data are flowing through your program: you can decide how much you want to refer to (a buffer size), but once the stream of data is no longer in your buffer, it’s gone forever. There will be an archive of course, kept somewhere, but processing things that way is NOT PART OF THE PLAN. And liable to be a lot slower. I suspect this must be why it took a week or more to fix whatever the bugs were when Ovo had a little technical difficulty the other week.

I’m confident that the algorithm for estimating meter readings that I will implement using arrays can also be implemented using event streams. But that’ll be a task for someone else, not me.

Userlevel 3

@Peter Rue, @Simon1D, @Blastoise186, @Jess_OVO: there is another way to get your energy usage.

Glowmarkt (glowmarkt.com) sell CAD devices that can be connected to your smart meter to fetch usage for you. Except that they don’t because demand has outstripped supply, and you can’t get devices from the shop. The smart meter data they provide is available to integrate with various products, through their web application and from an API.  The data from their CAD devices is real-time i.e. you see live usage as well as historical data. I don’t have any experience with their devices.

Glowmarkt also have a free service where you can sign up to access your smart meter data straight from the smart meter network into their web application.  Glowmarkt got accredited to access the smart meter network in order to provide this service.  The data from this service isn’t real-time but is within 30 minutes.  I’ve signed up to this service, seems to work really well. I don’t know how they make money out of this, but it’s a great service.

I wrote some code which accesses the Glowmarkt service (https://github.com/cybermaggedon/pyglowmarkt), and GitHub user danmed worked out how to run that in Home Assistant (https://github.com/danmed/Glow2MQTT).  This is probably only useful to you if you are already a Home Assistant user (or want to become one, it’s a great project :grinning: ).

Userlevel 7
Badge +1

PS. If anyone is wondering why my meters got stuck at the end of March, they didn’t. But Ovo changed the API in June (?) 2021 so that the “daily” files don’t include “meterReading” values any more. My data scraping utility had overwritten my files back to April before I realised and stopped it.

It’s not a problem really, because the recent files do include values for “usage” which were obtained (I have no doubt whatsoever) as differences between actual meter readings. So, I’ll have to do some reconstruction (“estimation”) starting from the only smart meter readings I do have, which are on the monthly statements. As someone said

“We don’t actually use the meter readings, so we stopped saving them.”

I would react to that, but my reaction would be impolite. Unprofessional. There are no polite words for retrieving the information needed (meter readings), processing it in a way that throws some information away (taking differences) and only using that incomplete version of what was retrieved.

I will focus on the estimation-in-the-presence-of-missing-data problem next. The fact that I’m having to estimate a few pieces of data that are missing not because of any smart meter communications problems, but because Ovo threw them away, should be embarrassing to someone.

Anyway. With these tools in place the outline design of the solution is, as some theoretical physicists are too prone to say “trivial”. Actually implementing that solution, and debugging it, isn’t quite so trivial because I’m only an amateur programmer (albeit one who understands algorithms and data analysis).

Cheers all.

Userlevel 7
Badge +1

I’ll continue by showing one more example...

Back in February was when our gas usage peaked. Zooming in, in the usual way:

and again:

and one last time:

At the peak (of more than a week of sustained 50 kph winds from the east - being located on the seafront has its pleasures, but a tall house also feels it, to the full) we got through 300 kWh of gas in one day, so heating the water (10 kWh) was lost in the noise.

Right. Back to coding. An early version is already on GitHub, for those who like such things, but it’ll all be there soon enough. (Well, only a tiny bit of my data will be there, for demonstration purposes - you can scrape your own data!) because that’s what free software is all about. Collaboration.

Userlevel 7
Badge +5

Oh, and I forgot to do this earlier. But I want to give credit to @cybermaggedon for sharing details with me regarding the internals of the Chameleon IHD6 and going through all the details. My answer above will be edited shortly to include that properly.

Userlevel 7
Badge +1

Having all the data available like this makes it possible to investigate, to test possible interpretations, to understand what’s going on.

For example, in August last year (that’s at 2020-08 in the plots), there is a little gap in the gas usage. It shows up in the HH data as well as the daily sums and even in the weekly sums, but not (well, arguably there’s just the slightest dip) in the monthly totals.

We can zoom in (I’ll go back to the un-grouped view of the data):

zooming in to show only some of the data

There really was a week or so in which almost no gas was consumed. It’s not a case of “missing data”, because there are tiny blips of usage every day. This is what a day looks like, before the gap:

Actually, that view runs from late one morning to midday the following day. It’s not a “midnight to midnight” view. (I admit - there’s a bit of showing off in this post. I hope @Transparent likes what he might see here...)

And here’s what a day looks like at the start of that low-usage period:

and the following day:

That was the day when we noticed:

  • the hot water isn’t hot!
  • the boiler’s not working!
  • put the immersion heater on!
  • where’s the number for our boiler maintenance/insurance policy?

Within a week, an engineer had visited, who found and replaced the faulty gas valve inside the old boiler. The replacement needed approval, because such things aren’t cheap - but as they’d serviced the boiler only the previous month, there was no argument, really, and that insurance policy paid for itself handsomely last year.

Things to note regarding my display of usage - it can be helpful to see both electricity and gas on the screen at once, and to synchronise the time axes (automatically ;-)) so they are showing related information.

I’ll continue by showing one more example...

Userlevel 7
Badge +1

To continue.

That default view shows smart meter reading above, usage below, electricity on the left and gas on the right. The labels say that, but I’m still working out what’s the best labelling of the x and y axes. At this stage, the key thing is that I’m confident that the data are correct. I’ve made no attempt to fill in the gaps in the smart meter data by estimation (yet), but I have “cleaned” the data a bit (specifially, I have replaced obviously stupid values, like the consumption, in 30 minutes of 2^24-1 litres of gas. If Ovo’s API says the half-hour value is over 50 m^3, I replace it with zero. To use the jargon, I’m pretty sure my gas supply is rated as U6 which, as far as I can tell, means a flow of not more than 6 m^3 per hour, which means the half-hour value can’t be more than 12. Once upon a time, maybe still, Ovo must have used that MAX value as a flag meaning “missing or bad data”. To be honest, that’s a simple-minded way to process data. Things like that come to bite you in the bum eventually. But never mind.

The other feature of my usage displaying thingy is that the data can be “grouped”. So the raw half-hour usage can be summed to produce daily values (so, groups of 48), weekly (groups of 48*7=336), ~ monthly (groups of 48*30) etc.

I’m still working on this, so that little detail is hard-coded and I have to edit and re-run but, once I’m clear what features I want in my “user interface”, I’ll change that. But the effect is as follows:

HH usage summed to produce daily totals

Immediately, the seasonal variation in gas usage becomes apparent.

Why is that? Because my old boiler either burned or it didn’t (“non-modulating”) and when it burned, the rate was nearly 40 kW of gas energy in. That’s nearly 20 kWh in a half-hour interval and that’s what it would do, most mornings, to top up the hot water in the cylinder. It would do that for around 40 minutes, iirc. I can no longer check this, because both boiler and cylinder were replaced last month.

I can take the HH data in bigger groups, lasting a week:

HH usage summed to produce weekly totals

or (more or less) monthly:

HH usage summed to produce 30 day totals

(no, they’re not actual months, they’re all 30 days long. For my present purposes, that’s fine.)

To be continued...

Userlevel 7
Badge +1

Depending on your technical skills you might also be interested in this unofficial guide to accessing your usage data via our API. Sure @Simon1D has some top tips in formulating this data in some bespoke usage graphs if that’s your thing too? :slight_smile:

Thanks for thinking of me, @Jess_OVO ;-)

Your comment prompts the thought that, in case anyone might be interested in using the API in its present form (which I’m hoping will remain as it is, in spite of one or two shortcomings, until the talk of making a more official/supported API available comes to something), it might be useful to prepare a summary of what (I have found) is necessary to make things work (at the moment).

As it happens, I’m currently putting together a demonstration of what is possible regarding the presentation of usage but, as a work in progress, I’ll just offer a snapshot or three…

all my smart meter data

This one shows the default view: Everything, meter readings and usage, since the smart meters were installed, with half-hour resolution.

(I’ve previously scraped all my smart meter data using the Ovo API and saved a local copy. It’s not polite to burden Ovo’s servers with repeated requests to view all my data - once is enough.)

This is the default interactive plotting widget that comes as part of the free (that is, open source and more...) matplotlib system. Everything I show is output from code I’ve written in Python using numerical data processing library, numpy.

The payoff when those details:

  • all the data
  • full resolution
  • interactive

are combined is, in my opinion, hard to overstate. It’s not easy to do justice to it in static snapshots, but I’ll try in the following…

[Edit: there is already a little detail that that graph shows up that I find absolutely fascinating, but it would be a disraction to discuss it here, so I’ll come back to it in a later post.]

Userlevel 7
Badge +5

No worries. :)

Unfortunately, I don’t think there’s ever going to be support for connecting any IHD to your PC, regardless of which energy supplier you’re with or the equipment you have. Smart Meters are considered to form part of critical national infrastructure and therefore are required to run on a completely closed ecosystem where only approved devices are allowed to connect to anything. Even the communications back to the supplier run on private links that don’t touch the public internet at any stage. The last thing you want is for someone to hack the system and cause a massive blackout!

The idea of allowing customers to use a smartphone or tablet to connect directly to a Smart Meter has been previously considered when the SMETS1 and SMETS2 specifications were being drafted, but it was ultimately rejected for security reasons - I think range and compatibility issues were also a factor. And I’d probably also argue that it’d have locked out anyone who doesn’t have a smartphone or tablet, so the IHD method is more universal.

It would have been kinda cool to have, but ultimately it probably wouldn’t have worked as well as some people might expect.

Userlevel 7

Welcome to the OVO online community, @Peter Rue.

 

Great to see our community volunteer, @Blastoise186  has already offered some tip-top advice in the best ways to access that really important smart meter usage data. Worth giving a mention here to our great guide to the usage pages of your online account or OVO app (download for Android or iOS).

 

Depending on your technical skills you might also be interested in this unofficial guide to accessing your usage data via our API. Sure @Simon1D has some top tips in formulating this data in some bespoke usage graphs if that’s your thing too? :slight_smile:

Userlevel 1

Thanks for the reply and explanation. I'm really quite pleased that I was not missing the obvious!

I do find it a little strange though. The data are there and being passed backwards and forwards but can't be interrogated. Perhaps next time the IHD is redesigned…

Meanwhile I will check out link that you gave. 

Thanks again

Peter

Userlevel 7
Badge +5

Hey there @Peter Rue ! :)

Unfortunately, I’m afraid this isn’t possible. The Wi-Fi features on the IHD6 are mainly for firmware updates and CAD (Consumer Access Device) features that may or may not be enabled by suppliers. It will not talk to anything on the local network at all, other than the Comms Hub above the electricity meter. I’ve previously monitored the traffic these devices generate using tcpdump in my router and they basically only talk for very short bursts as and when they need to and have never attempted to even discover other devices on my network.

While the IHD6 does also feature a Micro-USB Port for charging, this is literally for charging only - the data pins are physically disabled and disconnected on the board itself, meaning they lead to nowhere. While you might be able to get it to charge from a PC, I’m almost certain that nothing else will happen beyond the PC just delivering some power to the IHD to charge the battery.

Source: Thanks to @cybermaggedon who provided me with details about the internals of the Chameleon IHD6 and showed me pretty much everything inside one. Please do not dismantle your IHD however, as you WILL break it by doing so!

The best alternatives right now would either be to manually grab the details from MyOVO or ask the Support Team to give you a copy of your data. There is also an unofficial Home Assistant integration that might interest you - but please bear in mind it’s not supported by OVO and the OVO’s Support Team will not be able to help you with it so you’ll need to get help from the community if you need it.

 

Welcome to the OVO online community, @Peter Rue.

 

Great to see our community volunteer, @Blastoise186  has already offered some tip-top advice in the best ways to access that really important smart meter usage data. Worth giving a mention here to our great guide to the usage pages of your online account or OVO app (download for Android or iOS).

 

Depending on your technical skills you might also be interested in this unofficial guide to accessing your usage data via our API. Sure @Simon1D has some top tips in formulating this data in some bespoke usage graphs if that’s your thing too? :slight_smile:

 

Reply