In video they have I-frames that are complete frames. Then they just send the changes for subsequent frames, which could be simple addressing. This would fit in with DMX that only requires a full output once a second to keep the lights alive. You could keep the full frame in memory and then only have the changes in the file. This would help reduce the file size.
I also like the idea of a scripting language for the output video. Something simple that does not require a lot of processing, but tells the PI how to fill the output universes. I have a sample for what I did last year, but there must be a standard for this type of thing.
It's clear that there will be multiple formats needed for multiple purposes.
There is a niche for a simple binary format for input to a sequence player, such as proposed here. Not just for the RPi with FPP, but also for the Arduino or PIC or Propellor or Teensy or MSP430 - they can all share a format like this one. This class of format is compact, easy to program, can be interpreted with small program and data sizes, and is very fast. Of course, larger or more powerful architectures can handle more channels and smaller devices can have more limited channel & speed limitations, but both can use the same simple binary format. The format should be designed to serve these closely related purposes (small device players). Obviously, larger systems like xLights can also play these format, and Vixen could possibly be adapted to read them in to edit or play them (with some limitations).
Storing information for full featured editors is a different niche than simple players. You want more info that allow a sequence to be reopened in an editor with full context between sessions. The Vixen 2.x format is an example of that, but every sequence editor has some format with that extra information. That's another niche.
Base-64 encoding in Vixen 2.x is going to take 33% more space, plus some overhead at the start and end. That's fine as an interchange between large programs, but it definitely slows down a micro to decode and reshape it back to binary, and adds unneeded software complication for a player.
One could also define a more complex format for video, with something like I frames and deltas. What does that buy you, tho? Mainly file size compression, of a variable degree. In the case of a player like FPP, (1) affordable flash can easily handle the size of the a simpler uncompressed format, (2) the flash can be read fast enough to handle uncompressed data, (3) the player needs to be able to handle the full uncompressed bandwidth anyway for the worst case, (4) it's less processing to copy the binary information around. I'm not sure the advantages of such compression are very useful for the FPP niche. Perhaps it would be useful for an Arduino class player which was limited in flash read speed, under the right set of circumstances, but mostly I think they can read flash more than fast enough for the number of pixels each should control.
A 16 bit pixel color depth is a possible enhancement. It doesn't per se buy you audio tho, unless 10-50 samples per second audio is appealing 🙂 Otherwise you need to embed many audio samples per sequence step, possibly a non-integer number of them. It would be possible to define a format which allows embedding interleaved audio, but that would likely be unrelated to whether the pixels have 8 or 16 bit depth.
===
When this niche came up in the Arduino discussions, I suggested using uncompressed 8 bit BMP format (which is lossless and simple). This could be done as monochrome (individual channels as 8 bit raster elements) or as RGB (3 byte raster elements). Each row would be one sequence step. For RGB data, each image pixel represents one display pixel at one given sequence step (vertical row). The BMP header would indicate the monochrome vs RGB format, the width (= number of channels or RGB pixels) and the height (number of sequence steps). The first bytes of the first row would contain some extra metadata, like playback speed. The micro could find things like mono vs RGB, and the width/height at fixed offsets into the fixed length header, and the other metadata in the first row, also at a fixed offset. Data would be packed binary. This would be an entirely legal BMP file, with the only quirk being that the first row contains metadata masquerading as raster pixels.
Some side effects - each sequence would become an image, which should be fun to look at. A sequence can be compressed as any lossless image format and recovered to BMP binary (say for storing on disk). You could even edit images in an image editor, and then play them back as a sequence.
The main disadvantage in the current context is that it might not be able to store enormous channel widths; I'd have to look at the spec to see the limits. It would probably handle way more than an Arduino needs to deal with, but the FPP has ambitious goals.
I've been thinking that one of the first projects I wanted to do to enhance xLights (when I get some slack time) is to produce an export and import for this augmented BMP format. I still want to do that whether or not FPP defines a new binary format for the same niche, because I think it will be fun and interesting. I want to see my sequences as images, and I want to play some images as sequences. Imagine playing a real image as a sequence going to a strip attached to a wand - instant POV (the player could ignore/not require the speed metadata in the first row).
If 16 bit resolution is desired, we could look at whether there is an uncompressed PNG format which is easy for a small micro to decode. But that brings up a whole different aspect.
Have a talk with Joe on the HLS format. It's simple, Human readable and easy to use. I was looking at doing something with it myself.
The other format to look at, and maybe the best way to go, would be xLights. big plus is the you could use it as the conveter so no creating your own program. Only issue there is that I don't think he's done video yet.
Personally I'd rather not see yet another format but use a current one.
OK, about 16 bit color depth in pixels (or 10-12 bits packed into 16 bits of file data).
There's no real need for 4000 distinct brightness levels in our pixels - 256 brightness levels is way more than the eye can distinguish in pixels, IF DISTRIBUTED WELL. If we use linear brightness interpretation, there are some discernable brightness gaps between values (0,1,2,3) at the low end, but nobody can see the difference between 100 and 101 (much less 254 and 255). Having 15 (or even 3) additional levels between 2 and 3 could be useful, but adding 15 or 3 levels between 100 and 101 is a waste.
The answer to this is to operate on values which approximate perceptual brightness levels (where the perceived difference between 10 and 20 is about the same as between 110 and 120). That is, roughly exponential or logarithmic functions (depending on which end of the telescope you peer into). Gamma curves are a traditional way to handle this.
The question is where the gamma curves should be interpreted. One approach is to have the sequence editor apply some remapping before sending values to pixels or to an output file for exact playback. This is attractive because the programmers of the editors can do it without regard to the variety of hardware it will play back on - so long as it's 8 bits. If we define a 16 bit per color file format, this would give the sequence editor more room for the finer granularity it needs at the dim end of the scale - but the editor will need to know if the destination uses 8 bit, 10 bit, 12 bit or other resolution before saving the file, and there might be mixed pixel types involved with some pixels using 12 bit resolution (TLS3001) and some using 8 bit. That makes for a messy configuration and file format.
So there is some attraction to implementing the gamma correction at the player/controller end, where it knows what kind of pixels it's dealing with. In that case, 8 bits is plenty of resolution for the file format. Suppose you have an Arduino connected to TLS3001 pixels; it would expand the 8 bit values to 12 bit pixel values using a lookup table based on gamma (or whatever). Another player/controller node might be sending output to 8 bit pixels (or 7 bit pixels like the 8806 or 6 bit pixels like the 6803 or strange combos like the GECE). In some cases the player/controller may add additional levels to what the hardware provides (eg: sending values 2,2,2,3 to simulate a value of 2.25 and getting 10 bits effective resolution form 8 bit pixels, etc). The thing to note is that the knowledge of how best to apply the gamma curve resides at the player/controller end, depending on what kind of pixels it has been configured to support.
The same sequence file can be played on different pixel types, each being used to do its best to translate the 8 bit values into exponential brightness levels based on what the particular pixel hardware provides and what the controller can simulate.
Implications for the file format:
1) Have a value in the metadata which indicates whether the values are logarithmic or linear; make logarithmic the default and encourage people to use that.
2) The player/controller reading the file format should apply its own gamma correction for the pixels it has been configured to handle (unless the linear flag is set)
3) A 16 bit pixel color resolution is not highly needed in the file format, 256 logarithmic levels (gamma corrected) is more than the eye can perceive in this context
4) Fallback - even if you play a semantically logarithmic value as linear, it still works (in essence that's usually what we do now in most cases)
About the term "player/controller". There may be variation about where within the player/controller/pixel chain the gamma correction (logarithmic to linear conversion) gets applied. In the case of an Arduino reading an SC card and controlling a pixel string, the same device will be both playing and controlling, and would do the gamma correction. When there is one device as player and a different one as controller, either could do the gamma correction (but not both!), I can picture both cases. There can even be pixels which handle this directly so neither player nor controller device need to worry about it. Sorting out which piece of a player/controller/pixel architecture does the gamma correction is beyond our scope in this message - the point here is just that gamma correction, if needed, is preferentially done AFTER reading from the file format (by some component of the player/controller/pixel devices), rather than BEFORE writing to the file format (by the Sequence editor or generator).
An 8 bit linear format would also be supported as an alternative, meaning that the player/controller should NOT do gamma correction but should instead send out exactly the values it reads from the file (perhaps with more bits appended if the pixel needs them). That could be good for servo control or other uses for which gamma correction is inappropriate.
There is one key piece not addressed here: what gamma factor to use. I would suggest that to keep it simple, we define a standard gamma for the file format, which can then be implemented even in small controllers using a single fixed table - rather than providing a gamma value in the file which then has to be dynamically interpreted by the player/controller. Given the nature of our context, this does not need to be really precise. If we used a 2.2 gamma encoding but a pixel string did its own correction assuming a 2.0 gamma, nobody is going to tell the difference in practice. So I don't see the need to encode a different gamma for every file; one good choice should be close enough for everything.
Defining a new file format for playback of sequences is a good opportunity to "do this right" before the first implementation is completed. If it's in the spec from day one, we won't have to worry about lots of software that was written to handle the format before gamma correction was added.
I take the point about not creating a new format if there exist good ones already with the needed functionality.
The problem with using a current XML format is that it's a hack to pseudo parse it on a small micro, and it's slower and more complex to play back when encoded as text.
The big picture is this: while editing a sequence, use whatever format your software supports for editable sequences. If you want to export it to a Flash based file ready for playback, convert it to a simple efficient playback format on the SD card or thumb drive or whatever. The binary format is for a different functional niche than an editable sequence, so the optimal format can also logically differ. It's not "another format for the same purpose" but "a new format for a different purpose". It's a bit like source vs object code; there are reasons they can differ without being redundant.
It's going to be translated into binary byte sequences anyway before being sent to pixels or DMX or whatever. The question is just where that happens. There are some advantages to doing that conversion on the way into a play-only format on flash memory rather than on the way out of that format.
Case 1:
Edit in XYZ editor, saving your sequence in its native format
Export completed sequences for the player into (eg) Vixen 2.x XML text format to be written to flash memory
Microcontroller/RPi reads from Flash and translates Vixen 2.x XML text format to binary byte sequences
Microcontroller/RPi sends binary byte sequences to pixels
Csse 2:
Edit in XYZ editor, saving your sequence in its native format
Export completed sequences for the player into a simple binary byte sequence format to be written to flash memory
Microcontroller/RPi reads the simple binary format from Flash and sends it to pixels
Translating to Vixen 2.x XML text format as an intermediate Flash-resident format, between the sequence editor's native format and the binary byte sequences the microcontroller sends to pixels, seems like just extra work. A format which is well adapted for the smaller computer reading it seems to have value to complement rather than replace the editor's native format.
But that's just my take.
