Lessons learnt DMX over IP.. Part(1) there is standards / there is no standards

[mrpackethead]

Somebody asked me a question the other day that prompted me to think about some of the issues that i've had in getting DMX over IP to run. While the lessons learned come from larger commercial installations, i think they are still well worth considering in any display. You can "get away" with things when they are small, but those decisions might come back to haunt you at a later date. I'm thinking I'm going to write a few things about what i've learned over the next few weeks...


Theres two main protocols that are used for transporting DMX over an IP network; Art-net ( and that comes in three versions, v1, v2, and most recently v3 ) and Streaming ACN otherwise known as E1.31. At a most basic level both of them take a "universe" of channels ( thats a block of 512 channels ) wrap them up in a data packet and transmit them onto a IP network. The specifications for the 'standards' are published.. The art-net standard is free, the e1.31 standard you need to pay for.

So the first thing i've learned is that while there is standards, theres not that many devices / systems that really adhere to the 'standards' to the letter and or fully. This isn't just in DIY style hardware and software, this also in commercial systems as well ( and some of them cost > $100k !! ). I must add that the controllers that i've developed and sell also are not 100% compliant with the standards, but we are working on it.

The lack of adhering strictly to standards is done for a bunch of reasons. Sometimes its clearly just bad design, but more often its a compromise that has been made to make something work and it was a deliberate choice.

Lesson #1. Don't assume because your media server ( vixen, lsp, Grandma, madrix or whatever ) which says it runs a 'standard' will interconnect and run flawlessly on a network with your end devices and controllers, when they also say they will use a 'standard'. Either test it all out yourself, or take the easy path and learn from someone else's experience. This testing needs to be at the scale you intend to run it at.

My lesson 1a is to try to design into the stuff i'm doing 100% compliance to the standards. Though thats really tough.

I've love to hear your stores about when stuff hasn't worked ( or worked ) the way you expected.. I hope that we can all learn some good stuff.

[CaptKirk]

"There are lies, damned lies, and then there are 'standards'!"

[dirknerkle]

I know precious little about DMX in general and even less about Art-Net or E1.31 so I'm not one to spout off here too much. But I know that designing a product requires a bit of a balance between performance and cost.

In empirical situations, 100% compliance with a published standard is most certainly an admirable goal, but in today's world of practical economics, it's quite a temptation to shave a bit of cost here and there with a part that's perhaps not the best quality, or one that has a narrower operating temperature window, or sometimes we find that leaving a part out altogether (even though it's suggested that it be there) seems to make little or no difference in the product's performance.

For example, perhaps the addition of a decoupling cap doesn't seem to make the chip work better or worse. The technician would say "Put it in there, it'll make it more stable" while the accounting department says "if it doesn't seem to hamper performance, take it out and save the money on both the board and the part" while the marketing guy says "the customer won't notice -- they'll replace this stuff in a year anyway with the next generation model."

Cases in point: do your lights blink better with 18 gauge than the 16 gauge extension cords? Does your controller work better with diagnostic LEDs or without them? Do you use the taller 5mm terminal blocks on your controllers or the shorter ones, and do you notice a performance difference between the two? Do the strings of mini incan lights you buy at $1.99 work better than the ones you buy at $1.79?

My point is that while there are standards in everything we do, we likely tend to modify our application of those standards by practical means, lest we be guilty of overengineering.

In my case, I had some Ren-W's that I used one winter that used the MAX232N chip, of which I had bought several dozen at dirt-cheap prices. They worked fine until the temperature dropped below zero, which was quite a bit below the lowest operating temperature for the N part. Switching to MAX232IN chips (good down to -40C/-40F) was an instant solution. Lesson learned: always check the operating temperature for designing gear that may be used in extreme conditions.

[jstjohnz]

A couple of comments. Sometimes portions of the standard may not make sense for a particular application. An example that comes to mind is arbitrating bwtween 2 or more simultaneous senders. The default is supposed to be "HTP" or Highest Takes Precedence. In other words if 2 senders are sending different dim values to the same channel, you go with the highest value. While that may make sense for individual single-channel light fixtures. it doesn't for pixels. If one sender is trying to light a pixel RED while another is trying to light the same pixel GREEN, and you use HTP you wind up with a YELLOW pixel.

There are other examples, for example in multicast E1.31, the universe numper appeares twice in every packet, in the header and in the packet itself. The standard says that if the 2 values differ, use the one in the packet. Why? If the 2 values differ, something's screwed up, and whichever way you go you've got a 50% chance of being right.

Also ETC has added their 'CC' extensions to the SACN standard that allow a sender to take control of a portion of a universe. Well, there's a lot of overhead involved in implementing that feature because there's no theoretical limit to the number of possible simultaneous senders.

Anyway, the good news is, for the most part, if you're dealing with a single sender, which is the case for most of the installations we deal with here, everything generally talks together pretty well.

[Zeph]

The hard part in a well established ecosystem of connecting products, is being interoperable with other devices which bend or break the standards. This comes up in nearly every sphere of technology I've dealt with in any depth. Sometimes a large part of the complexity of a long-established piece of code is all the workarounds they put into it over the years to compensate for the quirks they've run into (and pray they documented them!).

The lighting world appears to have loose standards, for better or worse. Formal certification, or even attending plugfests, isn't generally required from what I can tell. So the kind of sensible voluntary attempts to adhere to standards as often as possible described above seems like a good balance. Some features just aren't needed so they get simplified; but it's good to avoid pointless deviations and to have a really good reason for departing from the norms.

[mrpackethead]

The lighting world appears to have loose standards, for better or worse.

And that really summarizes why i've made Lesson #1, "don't assume its going work". I' was really hoping to hear some storys about when it didn't work ( due to standards 'complacence' issues ) and how you worked around it.

[chilloutdocdoc]

And that really summarizes why i've made Lesson #1, "don't assume its going work". I' was really hoping to hear some storys about when it didn't work ( due to standards 'complacence' issues ) and how you worked around it.

This has become my new motto for work. Can't explain many of the details, but it involves working with embedded systems developed by a fairly small company in which their documentation doesn't mean diddly. Just because the paper says something, doesn't mean it always works that way. Also, they say they've implemented libraries in compliance with "standards" but, they don't. It often requires patching their libraries (as I have access to source) and letting them know that they've made mistakes, hoping that they'll fix it in their next firmware release (generally doesn't happen)

I'll toss a couple of questions out, some of them might seem a bit unintelligent as I'm not a networking guru, some might just be plain stupid. (Hey tech support asks if the device is actually plugged in, so can I)

0. Was everything plugged in properly?

Onto more important issues
1. Was the media server output guaranteed to be accurate (at software and network output level)?
2. Can we assume that the network hardware was properly handling the multi-cast setup (both server and switches) but NOT assume about the controller?
3. In saying that the network wasn't getting congested, does this mean only the network itself (copper, switches, etc) but NOT assume about the network interface on the pixel controller eg. Could there be a problem with the pixel controller instead of the network hardware?
4. Was the controller properly handling the stream of packets that it was receiving (looking for the right addresses etc)?
5. Was the controller able to handle the volume of the packets necessary for this network?
6. Was the controller trying to respond to more than one input universe?
7. Is it even within spec to receive 8 universes in one controller (did they really expect people to be doing it)?
8. Was there a different lag between different controllers?

Even More Important: What's the prize for the winner? A few Stella Strings? (one can hope)

Overall trying to logically step from the media server, through the network to the boards. Systematic solving is often more fruitful than poking at random items for larger issues.

[mrpackethead]

0. Was everything plugged in properly?

Yes, everything was plugged in fine. All the cables were high quality Belden Datatuff Cat 5. They also had been scanned and passed.

1. Was the media server output guaranteed to be accurate (at software and network output level)?

Yes, when wireshark was used to have a look, it showed at all and every packet was leaving the media server as expected.


2. Can we assume that the network hardware was properly handling the multi-cast setup (both server and switches) but NOT assume about the controller?

The switches were doing what you would expect by default with multicast traffic. The server ( see above ) was sending properly formed packets. The controllers were receiving multicast traffic, but when there was a lot of it, things started going problematic.


3. In saying that the network wasn't getting congested, does this mean only the network itself (copper, switches, etc) but NOT assume about the network interface on the pixel controller eg. Could there be a problem with the pixel controller instead of the network hardware?

Yes, none of the ports showed any congestion, dropped packets, or errors.


4. Was the controller properly handling the stream of packets that it was receiving (looking for the right addresses etc)?

No, when there was lots of traffic on the network it went bad. When there was only a small amount it was all good.



5. Was the controller able to handle the volume of the packets necessary for this network?

When the controller was isolated and only the universes that it needed where transmitted, it ran ok ( all 8 universes )

6. Was the controller trying to respond to more than one input universe?

See (5)


7. Is it even within spec to receive 8 universes in one controller (did they really expect people to be doing it)?

I'm not actually sure, but it works.

8. Was there a different lag between different controllers?

Yes different controllers reacted differently but it was quite random and adhoc, there appeared no real pattern, other than the more universes transmitted by the media server the worse things got.

Even More Important: What's the prize for the winner? A few Stella Strings? (one can hope)

The joy of knowing that you are the winner.

Overall trying to logically step from the media server, through the network to the boards. Systematic solving is often more fruitful than poking at random items for larger issues.

Good plan

[chilloutdocdoc]

0. Was everything plugged in properly?

2. Can we assume that the network hardware was properly handling the multi-cast setup (both server and switches) but NOT assume about the controller?

The switches were doing what you would expect by default with multicast traffic. The server ( see above ) was sending properly formed packets. The controllers were receiving multicast traffic, but when there was a lot of it, things started going problematic.

They're doing what you would expect by default. That leads me to believe that the default isn't the proper setup. Was there some configuration that was missed or a different type of switch needed? I've read a few things about multicast has to be supported by the hardware.

6. Was the controller trying to respond to more than one input universe?

See (5)

I didn't explain this well, say controller was supposed to receive universes 1-8, was it reacting to only 1-8 or was it also reacting to other universes (eg 9, 50, 102, etc)

Even More Important: What's the prize for the winner? A few Stella Strings? (one can hope)

The joy of knowing that you are the winner.

The joy will come from being the new owner of a few stellascapes products courtesy of somebody else's budget, how kind.

Other ?'s

Were the controllers properly handling the multicast data?
Is the E1.31 Spec in line with the Multicast spec?
Was your noted non conformance with standards an issue?

[Henedce]

In my line of work I have multiple sayings about assumptions or to assume something.
to ASSUME will make an ASS out of U and ME
Assumption is the Mother of all F$#K ups
I drum this in to all new employees. I am yet to see a major screw up that does not contain the words " I ASSUMED"