The file format at the end of the universe
OK, about universes.
TL;DR summary: Let's keep the concept of universes out of a this file format, and treat it as part of configuration instead. Go for a clean separation based on function. Including universes in the file format feels like leaky abstraction. And some general concepts of not including the concept of universes anywhere in our designs they are not needed.
The full version.
Are universes a truly useful concept, or a fossilized relic of earlier times? It seems to me that with pixels, they get in the way much more often than they assist us. We keep having to do odd arithmetic (512 channels can support a max of 170 pixels with two left as a gap before the next universe, using 50 pixel strings the most one can get is 150 in a universe, then 62 unused channels). People wind up juggling string lengths just to fit into arbitrary containers (42 pixels allows 4 strings with 168 pixels to fit into one universe, wasting only 2 pixels and 2 loose channels). Each firmware deals with the gaps of lost channels differently (leave them as empty memory cells, or use contiguous memory for the "active" channels of each universe?). Some can bridge between universes, some cannot.
How do we handle a 240 pixel continuous strip (720 channels)? Do we put 170 pixels in one universe, and 70 in another, or 120 in each? If we have two such strips, do we split them into four universes (170,70,170,70 or 120,120,120,120...) or three (170,170,70+70 or 170,70+100,140). Or we could treat them as 1440 contiguous channels broken into three universes of (512,512,416) ignoring pixel boundaries.
My thought at the moment (and I'm open to changing this based on input from others), is that we should move away from universes as a core organizing principle. That doesn't mean we don't support them at all - we are forced to make use of them in DMX and e1.31 (including questions like the above), but what if we considered that just a particular encapsulation protocol we sometimes need to use for carrying some of our channels, rather than a basic concept of organizing channels? Deal with the concept of universes only where they are required, and localize the knowledge of them on a "need to know" basis within the software chain of custody of conveying channels to lights.
Here is another encapsulation example to help clarify. If we run some of our pixels using nRF24L01+ transceivers, then we must encapsulate pixels into 1-32 byte over-the-air packets which can control at most 10 pixels, but we don't have to organize the rest of our system with those constraints in mind. Before that encapsulation, pixels might be grouped into universes for transport, or might not. After that encapsulation, pixel strings need not be in 10 pixel increments. In the sequence data (eg: this file format), there's no need for 30 channel RF packets, nor 512 channel universes, to show up anywhere before or after that encapsulation.
There are other encapsulations, like HyperDMX with 2048 channels. There may be others in the future. We can adapt to these best if we consider them as choices of encapsulations rather than as fundamental or pervasive units of conceptual organization of channels.
This file format can be used by an Arduino or RPi reading an SD card and directly controlling a 240 pixel strip (or several strips of whatever length). No universes are relevant. It could also be sent out to Renard protocol controllers. Or used to control some servos. Or sent out as DMX or e1.31. Or sent out over the air on nRF24L01+ transceivers. The only case where universes comes in is if some or all of the data is sent out as DMX or e1.31. Different destinations may have different encapsulations, and I see no reason to use 512 channels universes anywhere they are not inherently required.
So I don't (currently) see a reason to include the concept of universes in the file format. The concept (and numbering) of universes might be involved in configuring some software that reads the files, but that's a more complex thing than we can or should embed in the file format.
(Example config for a complex config: send channels 1-300 to universe 2 on dmx port 2, channels 301-364 to renard port 1 at 115200, channels 365-514 over the air via nRF24L01+. The concept of universes comes in only when and if one is sending some of the data on to some further encapsulation which requires universes - it just gets in the way for everything else.)
Note that if you changed your Renard PICs in the above to use RenardDMX, you could change this configuration, but there would be no need to change the binary sequence file.
And note that having the length and number of universes embedded in the file is not sufficient to fully configure even the DMX/e1.31 outputs; you need additional configuration metadata anyway to tell the software reading the file what else it needs to know about distributing the channels - so ALL the universe-related configuration can be in that configuration metadata which is not embedded in the binary file format.
I'm suggesting a clean separation between channel data (in this file format) and configuration data (handled in inherently diverse other ways). The concept of universes would (when and if needed) be in the configuration metadata, not in this file format. The file format tells you what each channel (or RGB cluster) is supposed to show, but doesn't get into details about configuring the various hardware devices and software protocols used between reading the file and controlling the lights.
That way we also don't have to keep re-inventing versions of the file format for new encapsulations.
mrpackethead wrote:I thought i'd drop my 5c in about 16 bits. The Pixels we sell commercially, have 16 bits of control. However, as has been said, there is no need to be able to use *all* of these 16 bits. In fact, i'd argue that 7 bits are enough to represent enough states, so you can't distinguish between them. The 16 bit capablility gives us the ability to buidl extremely smooth responses right down to the very very low levels. in fact, at level 1, many people say they can only just see it turned on. ( some say they can't ). I can see a difference between 1 and 2, but beyond that, i cant' pick changes..So, our controllers convert standard 8 bit DMX values into 'corrected' 16 bit values.. I'd suggest that it is the end controller that is the right place to put this capability surely?
Agreed. That's a succinct way of saying what I have also suggested, albeit more verbosely 🙂
I went a little further to suggest that we consider the byte values in this file format to be (by default) gamma encoded (with a flag for linear) because I think that's what we "intend" when we create the sequences. If the end controller can gamma correct 256 levels, it should do so (unless the linear flag is set). At worst, dumb controllers will do what they do now, treating effectively gamma encoded values as if they were linear. But since this is a new file format and new software needs to be created to read it, there's the opportunity to do it right.
I did not suggest including curve or lookup tables, because the facility for this is going to vary too much. And I suggested that we pick a good standard gamma factor rather than expecting every file reader to do the computations for it. That way, even a PIC or AVR uC can use a fixed lookup table stored in Flash memory. I think gamma correction will be sufficient and simple, and that custom dimming curves will be overkill.
A smart reader of the file could just send the 8 bit values directly to a smart controller and let the latter do the gamma correction.
However it might do its own gamma correction if sending to a gamma-unaware dumb controller or directly to a linear pixel. Only the reader knows which of its outputs are which. But in any case, the adjustments are preferentially made after reading the file format (with 8 bit gamma encoded values possibly translating to more resolution at the pixel), rather than before writing to the file format (with more resolution being stuffed into 8 bit linear).
This would mean that, for example, if HLS was writing a sequence out to this file format there would be two options. Ignore the dimming curves (for output to this format) and write the straight values to the file (implicitly gamma encoded), or use the dimming curves but set the linear flag in the file so gamma (-ish) correction won't be done twice. The former is preferred.
Re: The file format at the end of the universe
Zeph wrote:OK, about universes.TL;DR summary: Let's keep the concept of universes out of a this file format, and treat it as part of configuration instead. Go for a clean separation based on function. Including universes in the file format feels like leaky abstraction. And some general concepts of not including the concept of universes anywhere in our designs they are not needed.
The full version.
Are universes a truly useful concept, or a fossilized relic of earlier times? It seems to me that with pixels, they get in the way much more often than they assist us. We keep having to do odd arithmetic (512 channels can support a max of 170 pixels with two left as a gap before the next universe, using 50 pixel strings the most one can get is 150 in a universe, then 62 unused channels). People wind up juggling string lengths just to fit into arbitrary containers (42 pixels allows 4 strings with 168 pixels to fit into one universe, wasting only 2 pixels and 2 loose channels). Each firmware deals with the gaps of lost channels differently (leave them as empty memory cells, or use contiguous memory for the "active" channels of each universe?). Some can bridge between universes, some cannot.
How do we handle a 240 pixel continuous strip (720 channels)? Do we put 170 pixels in one universe, and 70 in another, or 120 in each? If we have two such strips, do we split them into four universes (170,70,170,70 or 120,120,120,120...) or three (170,170,70+70 or 170,70+100,140). Or we could treat them as 1440 contiguous channels broken into three universes of (512,512,416) ignoring pixel boundaries.
My thought at the moment (and I'm open to changing this based on input from others), is that we should move away from universes as a core organizing principle. That doesn't mean we don't support them at all - we are forced to make use of them in DMX and e1.31 (including questions like the above), but what if we considered that just a particular encapsulation protocol we sometimes need to use for carrying some of our channels, rather than a basic concept of organizing channels? Deal with the concept of universes only where they are required, and localize the knowledge of them on a "need to know" basis within the software chain of custody of conveying channels to lights.
Here is another encapsulation example to help clarify. If we run some of our pixels using nRF24L01+ transceivers, then we must encapsulate pixels into 1-32 byte over-the-air packets which can control at most 10 pixels, but we don't have to organize the rest of our system with those constraints in mind. Before that encapsulation, pixels might be grouped into universes for transport, or might not. After that encapsulation, pixel strings need not be in 10 pixel increments. In the sequence data (eg: this file format), there's no need for 30 channel RF packets, nor 512 channel universes, to show up anywhere before or after that encapsulation.
There are other encapsulations, like HyperDMX with 2048 channels. There may be others in the future. We can adapt to these best if we consider them as choices of encapsulations rather than as fundamental or pervasive units of conceptual organization of channels.
This file format can be used by an Arduino or RPi reading an SD card and directly controlling a 240 pixel strip (or several strips of whatever length). No universes are relevant. It could also be sent out to Renard protocol controllers. Or used to control some servos. Or sent out as DMX or e1.31. Or sent out over the air on nRF24L01+ transceivers. The only case where universes comes in is if some or all of the data is sent out as DMX or e1.31. Different destinations may have different encapsulations, and I see no reason to use 512 channels universes anywhere they are not inherently required.
So I don't (currently) see a reason to include the concept of universes in the file format. The concept (and numbering) of universes might be involved in configuring some software that reads the files, but that's a more complex thing than we can or should embed in the file format.
(Example config for a complex config: send channels 1-300 to universe 2 on dmx port 2, channels 301-364 to renard port 1 at 115200, channels 365-514 over the air via nRF24L01+. The concept of universes comes in only when and if one is sending some of the data on to some further encapsulation which requires universes - it just gets in the way for everything else.)
Note that if you changed your Renard PICs in the above to use RenardDMX, you could change this configuration, but there would be no need to change the binary sequence file.
And note that having the length and number of universes embedded in the file is not sufficient to fully configure even the DMX/e1.31 outputs; you need additional configuration metadata anyway to tell the software reading the file what else it needs to know about distributing the channels - so ALL the universe-related configuration can be in that configuration metadata which is not embedded in the binary file format.
I'm suggesting a clean separation between channel data (in this file format) and configuration data (handled in inherently diverse other ways). The concept of universes would (when and if needed) be in the configuration metadata, not in this file format. The file format tells you what each channel (or RGB cluster) is supposed to show, but doesn't get into details about configuring the various hardware devices and software protocols used between reading the file and controlling the lights.
That way we also don't have to keep re-inventing versions of the file format for new encapsulations.
I don't think you should dismiss universes.
Remember - this is not a controller - it is a Player. It sits between the sequencing software that normally talks to a controller using universes and the actual controller that receives universes since the premise is based on E131..
By abstracting away from universes - you are making placing an undue burden on the Player since it has to convert to universes to do E131. Since the sequencer already does universe/E131 packet processing - what is gained by abstracting the input only to make it configurable in the player.
I thought the Player was going to be a nice tool to use instead of a Play Computer - just a drop in replacement.
My 2 cents.
Joe
Note for those skimming the threads. The discussion of rationales and design decisions can get very involved, but the file format will be very simple to generate and to read.
A lot of the discussion is about what to leave out, and about WHY to include or exclude things. In the end, the WHAT or HOW of using the file format will be simple, and most people will never need to read the rationales.
So don't get discouraged if this thread seems like a lot of analysis to wade through.
Joe, I suppose that it may depend on whether one sees a need for a file format which is basically a transcription of e1.31 into file format, or a format for more general channel data which can be used by any uC driving any kind of output that uses channels. I think the latter can subsume the former but not vice versa, while staying simple.
One of my core interests in this is using an Arduino to drive pixels, with nary a universe in sight. But it seems like the exact same file format can be well suited to drive e1.31 outputs from a RPi, so why have two new formats for such closely related niches?
I do think that this file format can be used to replace the sequencing computer when playing a sequence, either way.
As you say, this is for a Player, not a Controller. I see the concept of universes as relevant to some but not all Controllers, and a Player format should support both universe-oriented and non-universe oriented Controllers. (Or go to Renard or Pixels directly for that matter). Put another way, the universe concept is for me encapsulated within the Controller abstraction, while the Player abstraction is universe agnostic. Introducing the universe concept into the Player feels like a leaky abstraction; i don't see any reason the abstracted Player should need to care about universes - except to the degree that a given implementation is configured to output some channels to a Controller which uses universes. Likewise the abstracted Player need not understand nRF24L01+ protocols, unless a specific implementation is configured to use them.
And that's just how I'm currently viewing it from 50,000 feet, not THE TRUTH. I'm looking to collaborate and synergize.
So I want to better understand what you see gained by describing universes in the file format. Maybe we could specify some optional "universe configuration" metadata in the header, for those use cases where it would be useful, without compromising the generality. This metadata could be ignored when irrelevant. Maybe there's a simple fix that meets your needs for an e1.31 oriented format - but let's do it right if so.
Is a single size and count of universes ALL the configuration that a DMX oriented Player implementation would need? Does it need the ID number of each universe, or a per-universe channel size? What if the Player was going to send some universes out via DMX and some via e1.31? Or via ArtNet? Does the file need to somehow describe which output each universe goes to? (Even within a flexible DMX oriented system with no Renard or direct pixel control).
As i see it, if you have to also configure the Player with "which universe goes to which output" (e1.31 address, serial port, whatever), then you could at the same time configure the Player locally to know that a given range of channels from the file should go to that output, with more flexibility than assigning universes (and without embedding that info into the file format). If the Player knows that a controller for 100 pixels is on serial port 2 to be communicate with via DMX, what should it do if the file format claims there are only 150 channels for the universe assigned to that port by the binary file? It seems that in general the Player implementation and configuration should know its own hardware and what controllers or devices it is sending the data too; having that configuration knowledge also in the file (or splitting the Player configuration between locally saved config and config information read from the file) seems redundant and conflict prone.
But I'm very aware that I may be missing some payoff, and seeing only part of the picture.
