ESPixelSticks and Multicast

[DaveCorder]

Hi all,

I'm having a strange problem with my ESPixelSticks. Some are ESPixelPop Tinys, some are NodeMCU boards, but right now the issue is affecting all of them.

I had almost everything working fine with multicast last night, except for one stick that refused to respond to multicast data (i put it in unicast and it was fine). Now, in trying to troubleshoot that issue today, I've managed to break things such that none of the sticks are responding to multicast data.

I've tested from both my desktop workstation and my show controller, both of which are on the same VLAN as the pixel sticks. Both seem to send data just fine - I can see the multicast traffic going out in Wireshark, and I can see it passing through my Ubiquiti UAP-AC-PRO access point (with tcpdump on the AP itself) (edit: using Xlights test mode on workstation, xschedule and da_e131 on show controller). But the "View Stream" testing mode doesn't show anything.

Network setup:
IBM BladeSwitch G8000 (core network switch)
D-Link DGS-1210-10P (PoE switch for Unifi APs, connected to the G8000)
Netgear GS728TXS (switch for homelab rack and desktop workstation)
Ubiquiti Unifi UAP-AC-Pro v2 (dedicated network 'show' for lights, SSID broadcast turned off).
pfSense (network router)

I have IPTV-based cable service, so I have multicast snooping enabled on the IBM and D-Link switches (but not the Netgear, presently).

In attempting to fix the one strand that wasn't working right last night, I disabled (I think) IGMP snooping on the UniFi AP and on the D-Link. This seemed to break multicast for all the sticks, so I reverted the changes, but multicast is still broken.

I'm normally pretty good at this computer and networking stuff, but right now I'm stumped (maybe I just need a break...). Anyone have any ideas on what I can look at to further troubleshoot this issue?

[stampedeboss]

After reenabling IGMP, did you reboot the AP and D-Link?

[DaveCorder]

After reenabling IGMP, did you reboot the AP and D-Link?

I did reboot the AP, but not the D-Link. I did that just now (along with another AP reboot, since it's powered off the D-Link). No change.

[jchuchla]

The UAP-AC-PRO units as well as most of the Unifi line have a long history of complaints about poor handling of multicast. That being said, I have them here, and they seem to be fairly reliable with more recent firmware. I don't use them for the show though. I use an NSLM2 instead.

If you have managed switches, you must have IGMP snooping on for multicast to pass. You also must have one IGMP querier on the VLAN.

You absolutely need to disable "multicast enhancement" on ubiquiti products when you're using e1.31

You should download sACNview onto a laptop and use that for testing. sACNview is the standardized test suite for e1.31 sACN traffic. It will generate data as well as listen for it. It does everything strictly to spec which is a good thorough test for your network and the sACN equipment. Connecting the laptop to various points in the network (both wired and wireless) and using sACN view as either a listener, or a sender, you should be able to figure out which piece isn't behaving.

Another thing you should be checking is the multicast registrations in the switches. (aka multicast forwarding database-MFDB, or Multicast MAC table) Make sure you see the destination mac addresses of the appropriate universes registered to the appropriate switch interfaces.

[DaveCorder]

Thanks for the tups jchuchla. All the switches are managed and IGMP snooping is enabled (I think...I'll triple-check). I haven't found proper instructions on disabling multicast enhancements on the UAP-AC-PRO. I think I did that successfully, which lead to the initial breakage today, and backing that change out hasn't restored functionality.

I'll play around with it some more...it is looking to be an AP issue. I just discovered that the wifi card in my workstation supports monitor mode, and so does Wireshark (it's a mac, and this combo didn't work when I last looked into it a few years ago). I captured some traffic but didn't see any of the multicast data being broadcast, even though I see it going through the ath0 interface on the AP (yay for ssh access and tcpdump installed...).

[jchuchla]

"multicast enhancement" basically converts multicast traffic to multiple unicast traffic at the AP for only the radio segment of the network. It is a benefit when data accuracy is required. But is a big disadvantage where data timing is important, as is the case with sACN. To my knowledge, it has nothing to do with snooping other than snooping is required for it to function. It needs to be aware of the multicast subscriptions so it can convert them.

Wireshark isn't the best troubleshooting tool for troubleshooting sACN. You need to make sure IGMP is working first. You can use wireshark for that. If the IGMP subscriptions aren't happening, the multicast traffic shouldn't be flowing, and you consequently won't see it in wireshark. You can use wireshark on the wifi segment to watch for the IGMP group membership requests and leave requests from the ESPixelSticks as well as the membership queries from the querier. Speaking of which, you didn't mention which one of your devices is the querierer. Note that many switches don't have the querier functionality. Only one of mine does.

I use wireshark on my MacBook pro all the time. It has no problems reading the whole wifi landscape. The wired interface works just as well. And it's an older 2011 model too.

Are you using the 3.0 firmware on your ESPixelSticks? There's some significant fixes over the 2.0 firmware with respect to IGMP handling. The previous versions would only join the group for the first universe, and only did it once (I think) the newer ones properly respond to membership queries and do their periodic group joins for all universes they care about.

[DaveCorder]

I did find one of my switches did not persist the "enable IGMP snooping" setting when I rebooted it, so I fixed that.

Now I have the wifi on my desktop connected to the show network and sACNview is showing all the multicast traffic for all the universes I've configured, so it looks like multicast and IGMP are now working correctly.

All of my ESPixelSticks are running the new 3.0 release. They all work fine in test mode or unicast mode. When configured for multicast, the total packets counts on their status screens go up when Xlights is testing or outputting to lights. By all accounts things are looking right to me....the lights just aren't lighting up (nothing physical has changed since things were working last night). I've soft-rebooted and hard-rebooted the ESPixelSticks several times over the course of the day, to no effect. I have no idea what kind of gremlins are in my network right now, or how to get them out.

(I was just making a general observation about Wireshark on OS X -- it had just been a while since I last used it and I was pleasantly surprised to see major improvements since then.)

[DaveCorder]

I'm still having issues with some of my ESPixelSticks not processing multicast properly. Some, but not all. Everything's running 3.0, everything's got a static IP address, everything's connected to the same access point. I'm just utterly baffled and frustrated right now.

[jchuchla]

ok, back to basics. Have you verified the following:
Access point is only hosting one SSID. The one for the show.
Multicast Enhancement is off
Only the 2.4G radio is used for the show network.
Only ESPixelSticks are associated with the access point. No smartphones, no tablets, no laptops.
You don't have any connections open to the web interfaces of the ESPixelSticks while troubleshooting lag.

[DaveCorder]

ok, back to basics. Have you verified the following:
Access point is only hosting one SSID. The one for the show.

My SSID is 'show' and it's the only SSID on 2.4 GHz/Channel 11 on that AP (my other UAC-AP-PRO has a different set of networks on 2.4 GHz/Channel 6, and both have the same set of "production" SSIDs on 5 GHz).

Multicast Enhancement is off

I've not found clear instructions for disabling this specific functionality on the UAP-ACP-PRO hardware (https://help.ubnt.com/hc/en-us/artic...le-Explanation seems to suggest "multicast enhancement" is the same as "IGMP snooping", which I'm suspicious of). In any case, what I've done is set, in the sites/default/config.properties file, config.igmpsnoop_enabled.show=false

This results in the following configuration on the AP:

BZ.v3.9.3# grep mcast /tmp/system.cfg
wireless.1.mcast.enhance=0

If there's some other setting on the APs or in the Unifi controller that I need to modify, please let me know.

Only the 2.4G radio is used for the show network.

Yep, the show SSID is configured only on the 2.4 GHz radio on a single AP.

Only ESPixelSticks are associated with the access point. No smartphones, no tablets, no laptops.

Confirmed.

You don't have any connections open to the web interfaces of the ESPixelSticks while troubleshooting lag.

I have had some connections open to some of the ESPS web UIs, but this does not seem to correlate with which ones are not properly responding to multicast data.

Tonight I dug an old Motorol WR850Gv2 out of the garage (same hardware as the venerable Linksys WRT54G). I already had the Tomato firmware loaded on it, so I configured it as an AP-only with the same SSID as I had on the Ubiquiti AP, activated it, and deactivated the Ubiquiti. This has resolved my multicast issues, though there is now some noticeable lag (particular on the GECE strings on my roof), which I will need to investigate further. I appreciate your assistance.