z-wave network failure

We were out of town for several days out of cell coverage. Once we regained service, I received numerous alerts that our z-wave light switches, outlets, etc. were all offline. I tried the network check, re-scan (I don’t recall the exact name, the one behind the installer code), disconnecting the panel from AC and battery and got nothing. When accessing the lights from the panel all I could get was “acquiring device status.” And obviously nothing worked from the app or automated schedules. Also tried moving the panel around the house in case of some unknown interference.

Eventually I deleted the entire network thinking I would rebuild it, but now it can’t even detect a single device. Is this indicative of a hardware failure on the panel? Anything look out of place on the back end or other suggestions to try? Don’t recall what firmware I’m on so perhaps an update could refresh something. Had been working fine for the last ~2 years.

Saw a few threads in various places but no concrete solutions so if I missed one please point me that way.

Thanks.

Eventually I deleted the entire network thinking I would rebuild it, but now it can’t even detect a single device.

How did you remove the devices on the network? Did you do a “Reset Controller” only?

Something that is often missed with regard to Z-wave is that devices pair to the panel - not just the other way around.

To re-add devices you must individually use the “Remove Device” function on the panel first on each device.

General Z-wave installation tips can be found here.

Make sure to re-learn devices with the panel close by each one. Then when the panel is re-installed in its permanent spot, run a network rediscovery.

Don’t recall what firmware I’m on so perhaps an update could refresh something.

Keeping up to date with firmware is always a good idea, but the stability of existing compatible devices shouldn’t be affected.

Did you add any new devices recently?

How did you remove the devices on the network? Did you do a “Reset Controller” only?

Yep, pretty sure that’s all I actually did. I messed with this over several evenings with way too many interruptions. That said, should I be able to go into the panel and remove devices at this point or did I botch it? If I recall correctly I didn’t show any devices listed after the “reset.”

The network rediscovery was what I reference in my original post when I said “re-scan.” I’m not in front of my panel today.

Did you add any new devices recently?

Another GE light switch approximately 3-4 months ago with no issues.

That said, should I be able to go into the panel and remove devices at this point or did I botch it? If I recall correctly I didn’t show any devices listed after the “reset.”

Your panel will not show any devices if you reset the controller. This doesn’t impede you from using the remove device function.

“Remove Device” clears the network link in the Z-wave device you trigger, and can be used on devices which are currently part of the network or devices which are not part of the panel’s network.

From the Z-wave tips page linked above:

“When pairing a device, you should first remove it. This sounds rather counter-intuitive, but Zwave devices, unlike alarm sensors, are linked to a parent device. When a typical security sensor is installed, the panel is instructed to “listen” for its signal. The sensor could then be taken and learned into any number of other systems. Inclusion of a Zwave device pairs it to that system and it can no longer be added until it is first removed. Most of these devices are tested in a network during the manufacturing process. Using the remove device function of your panel will clear the previous network link from your Zwave device.”

Believe it or not, I did read the link when you originally posted. It just didn’t fully register with me that you could remove a device if it wasn’t visible or a part of your own network. Obviously I need to put a little more effort into this later on today when I have access to my panel. I’ll see about removing and re-adding and see what I can come up with.

Thanks, Jason.

Np, on some panels/z-wave controllers the “remove” function is labeled differently. On IQ Panel it is called “Clear Device” which is a bit more accurate.

Let us know if you see any issues with the devices after re-adding and running the rediscovery. It’s strange if all of them reported communication malfunctions at one time, but it could simply be interference of some kind. We would want the devices added to the network to troubleshoot though.

So – went through yesterday evening (started sometime around 5:30pm if you’re looking in the history) and removed everything and re-added successfully. Renamed devices on the panel as I added them. Everything was identified correctly on the add (manufacturer, type, etc.) and operated quickly and instantly from the panel. Re-mounted the panel, re-ran discovery, no issues and everything worked fine from the panel.

I gave a little time and looked in ADC and saw all the devices, only by type (light dimmer, light/appliance, node ID). Attempted to turn things and got nothing. About 30 minutes later I got my first alert of things going offline (I think 3 light switches). Then they came back online separately, then more went offline. I sent an All On command and 5 came back online, but never turned on, others remained offline. Then some of these 5 went back offline. Doesn’t seem to be a pattern of which ones are going offline. So as of right now, I have 5 light switches that are offline, and no commands from ADC seem to do anything on any switches online or offline.

A single failed device shouldn’t take down the whole network as that would defeat the point of the mesh network wouldn’t it? Is there an auto-re-discovery periodically to re-route in case of a failed node? All light switches are GE dimmers and standard, with one indoor and one outdoor outlet. I had an Aeon(sp?) repeater between the panel and next closest switch, though the whole network is small enough I probably never needed it. So for grins I deleted it and ran a re-discovery and one device came back online.

Potential hardware issue on the controller or interference? This setup has been working flawlessly for probably over 2 years. No new devices of any kind were added in our house that could be introducing any interference, especially since we were out of town when this began. Anything to try tonight? I should add that all other commands from ADC are processed instantly (garage up/down, arm/disarm, doorbell cam) so doesn’t seem to be a communication issue.

A single failed device shouldn’t take down the whole network as that would defeat the point of the mesh network wouldn’t it? Is there an auto-re-discovery periodically to re-route in case of a failed node? All light switches are GE dimmers and standard, with one indoor and one outdoor outlet.

A single device should not affect all devices, no.

Potential hardware issue on the controller or interference? This setup has been working flawlessly for probably over 2 years

Hardware failure is possible. Interference can be caused by devices on a similar band. One of the big ones we’ve seen are baby monitors, actually. Some wireless phones, etc. If the home has a shared wall it may even be a neighbor, depending on distances, but that would be exceedingly rare.

I should add that all other commands from ADC are processed instantly (garage up/down, arm/disarm, doorbell cam) so doesn’t seem to be a communication issue.

No, cellular signaling looks solid, but Z-wave communication shows a lot of failed messages, which you have of course observed.

A firmware update is a possible hail mary, but I couldn’t say I expect it to resolve anything, only that it might.

We requested updated status and three devices responded. Two still show malfunction.

I know it is likely not, but just to ask, is the panel located on or near a large metal object? (I’ve heard strange issues before only to find out the panel was inches from the fridge and microwave. Not a good spot)

What is the distance between the panel and the few nearest switches?

Hardware failure is possible. Interference can be caused by devices on a similar band. One of the big ones we’ve seen are baby monitors, actually. Some wireless phones, etc. If the home has a shared wall it may even be a neighbor, depending on distances, but that would be exceedingly rare.

No shared walls and the house has a yard and fence surrounding all sides so reasonable distance from neighbors. No other electronics of any kind have been introduced into the house in several months. Even then we’re talking 1 new zwave switch and the Skybell. Our baby camera is 2.4 Ghz and was actually with us out of town when all this began.

We requested updated status and three devices responded. Two still show malfunction.

I know it is likely not, but just to ask, is the panel located on or near a large metal object? (I’ve heard strange issues before only to find out the panel was inches from the fridge and microwave. Not a good spot)

I did get the notification of those 3 coming online, and then 7 minutes later a different one went offline so we’re back to 3 offline currently. The panel is mounted to drywall on a wood stud. It is directly above an outlet so there are a couple electrical lines behind it.

What is the distance between the panel and the few nearest switches?
The panel to repeater was maybe 12 ft through one standard hollow interior door (generally open). Then from repeater to nearest switch is maybe another 15ft open air. At that point, max distance between any other switch/outlet is ~ 8ft and spread out in a true mesh pattern, not in a point to point layout.

In thinking through the distances above, I realized that I did forget one switch in the garage to remove and re-add. I will do that this evening but don’t expect that to change the behavior.

I will also double check firmware. I’m fairly certain I have a firmware update cable somewhere. No idea where it would be so I’ll have to see if I can track it down.

In thinking through the distances above, I realized that I did forget one switch in the garage to remove and re-add. I will do that this evening but don’t expect that to change the behavior.

25 feet is a fair distance for the first node of a Z-wave network, but for switches it shouldn’t pose much of an issue. I wouldn’t expect to see what you are seeing.

Unless your couple closest devices are failing and the rest are routing through them, this is more likely a panel issue, especially if you cannot get any commands to go through.

I will also double check firmware. I’m fairly certain I have a firmware update cable somewhere. No idea where it would be so I’ll have to see if I can track it down.

It looks like this account has a Go!Bridge, so keep in mind OTA updates can be applied for free over broadband if you cannot find the cable or just want to go that route.

25 feet is a fair distance for the first node of a Z-wave network, but for switches it shouldn’t pose much of an issue. I wouldn’t expect to see what you are seeing.
I assumed that was stretching it which is why I added the repeater in my initial setup which essentially split that distance. I removed it only for testing purposes yesterday. One thing I may try is temporarily moving the main 2Gig panel to the middle of the kitchen and doing a rediscovery. At that point, the majority of devices would be directly connecting to the panel so could rule out some form of a node failure causing issues.
It looks like this account has a Go!Bridge, so keep in mind OTA updates can be applied for free over broadband if you cannot find the cable or just want to go that route.

Perfect. That must be how we did it previously. Fairly certain I’ve updated before (1.13 or 1.14) but didn’t recall plugging into it. Can I go ahead and authorize an update or does that panel need to be disarmed? Is 1.17 available? Didn’t see official release notes, but sounds like there may be some back z-wave enhancements so wouldn’t hurt. Are the TS1s done OTA as well?

1.17 is the most recent firmware via OTA.

The panel does not need to be disarmed to receive an OTA. During the OTA your panel will continue to function normally but will reboot at the very end of the update, then continue to function normally. After the reboot your panel will revert to the arming state it was in during the OTA. Please be aware that this process can take a few hours.

As long as the TS1’s are connected to the panel, they will update as well.

Keep in mind, powering down the panel or entering programming mode can cause damage to the system during the update process.

I have gone ahead and started the OTA update via broadband

Please be aware that this process can take a few hours.

You’re not kidding. Last I checked 3 or 4 hours later it still had an icon like it was communicating. I checked this morning and it’s at 1.17 (turns out previous was 1.12).

After I verified firmware was up to date, I deleted and re-added the aeon repeater and garage light switch I forgot, remounted the panel and did a re-discovery as I walked out the door. I single switch regained connectivity in this process. I sent an ADC ‘on’ command to the 2 remaining offline switches and they popped up and said the regained online connectivity.

So at this point, every z-wave device is reporting online, but I have no idea if they’re responding to remote commands. I will check this evening – hopefully a firmware re-flash and rebuild of the network cleaned out whatever gremlin developed.

EDIT - I remembered one of these lights is visible by a security camera and I can verify is responding quickly to ADC on/off commands so that’s promising. I’ll check everything more thoroughly tonight.

EDIT – I remembered one of these lights is visible by a security camera and I can verify is responding quickly to ADC on/off commands so that’s promising. I’ll check everything more thoroughly tonight.

That’s great news! No offline devices yet?

That’s great news! No offline devices yet?

Correct. 8 hours, no failures so far.