Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 26

Thread: How you tell if your system is healthy?

  1. #11
    Join Date
    Feb 2008
    Location
    Marietta, GA
    Posts
    539
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    LOL. I have similar diagnostics as well! One of our neighbors who drives by "frequently" will all call and tell my Snoopy has fallen or the "Ho"s are flashing etc.....
    Making it bright since 1989...



  2. #12
    Join Date
    May 2016
    Posts
    193
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    I got lucky this year. I had my only failure on the last day I had lights up. My primary diagnostic was that I’d see the display running each night when I came home. I’d pay real money if there was a doohickey to tell me when a pixel would fail and where it would fail.

  3. #13
    Join Date
    Jan 2016
    Location
    Ashburn, VA
    Posts
    345
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    This is a DIY hobby. It is not an engineered repeatable industry. Even so, most of the diagnostics are "I or my family or my neighbors look, listen, and report back". I understand that's good enough for most people, and it's enough for me most of the time.

    Does any look at the diagnostics on their hardware?
    ESPixelStick
    Up time
    Total packets
    Sequence errors
    Packet errors
    E682
    Up time
    Packets received
    Sequence errors
    Invalid packets
    E1.31 bridge
    Packets received
    DMX channel count
    Sequence errors

    I can make good guesses but can anybody definitively tell me what they mean?
    Is there anything useful we can learn from them?
    Does FPP, xLights or Vixen report the number of packets sent that might be used to compare against the hardware diagnostics?

    The reason I went down this path is that I am experimenting with waterproof and non-waterproof connectors and trying to get metrics on their performance in various weather conditions.

    I like the ideas of a "metrics pixel" at the end of strings. It would require a lot of supporting infrastructure but it is a good thought.
    SNMP is not the direction I was thinking.

  4. #14
    Join Date
    Jan 2017
    Posts
    291
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    So what would we want "metrics pixel" to look at?

    Voltage on the power line
    Voltage on the data-line?
    Quality of the data-signal, as measured by ? (shape of the incoming signal?)

    Then how simply can these be measured and reported? I would imagine that there might exist some very simple tricks that could be implemented.

  5. Likes Jerry-Rigs liked this post
  6. #15
    Join Date
    Nov 2011
    Location
    Chicago - Southwest Suburbs
    Posts
    7,569
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    most of the information metrics is there for the purpose of live troubleshooting. especially the packets received stuff. You'd need to extend on those concepts to get better system health data.

    packets received and sequence errors can be used together to determine the health of your network. But it's important to note that one sequence error doesn't necessarily mean that one packet was lost. a sequence error means that two packets received consecutively are not numbered in sequence. 1,2,9,10 would be one sequence error even though 6 packets are lost. Sometimes, a packet sequence restarts at 1 when a new show starts, or even per sequence with some software. This is normal, and you can expect occasional sequence errors even when no data was actually lost. The controller has no way of knowing that the jump from some number back to one is just a reset, or an actual gap in data.

    uptime can be a good indicator for power failures, or for hardware reboots. But you may have to aggregate the data from multiple devices to get useful metrics. Was it just one that rebooted, or many, etc...

    Keep in mind that if you're polling wifi controllers for metrics, those communications will be unicast and that can actually cause lag in your multicast wifi network.

    For the purpose of getting metrics of data crossing the connectors, you'd need to measure and compare data on both sides of the connections. I don't think you're going to find a ready to go product that will record the pixel data on the other end. That's something you'd have to build. I'm also not aware of any of the controllers providing metrics on the data it's outputting. it tells you what it's received, but not what it's sent out to the pixels. So you'd need your recording/comparison devices on both sides of the connections. One to know what it should be, and one to know what actually crossed the connection.

  7. Likes Jerry-Rigs liked this post
  8. #16
    Join Date
    Dec 2011
    Location
    UK S80 postcode
    Posts
    1,370
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    Hi, following - an interesting topic.
    Last edited by Barnabybear; 01-14-2019 at 08:44 PM.

  9. #17
    Join Date
    May 2016
    Posts
    193
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    I had more trouble from the GFCI breaker tripping than anything else this year. Next year I’m going relocate power supplies so that no plugs are exposed to any moisture—just run heavy gauge wire to distro board at the tree.

  10. #18
    Join Date
    Jan 2016
    Location
    Ashburn, VA
    Posts
    345
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    I like to quip that nothing is impossible, just really really really hard ... and expensive.
    I'm just brainstorming here. Please poke holes or make suggestions, but also please be gentle.

    Please help me understand.
    I think the ws2811 behavior is to display the last data received until new data is received. Is that correct?
    Do show players (FPP/xSchedule/Vixen) send an entire frame to the entire show on every refresh or if there is a steady state on a universe (or string or output), does that universe not get an update?
    I think if any pixel on a universe gets an update, all pixels on that universe get an update even if the updated value is the same as the old value, correct? Universe in this context may not be the correct unit. I may mean controller output.

    First train of though:
    If a show player could report how many packets it sent to a controller, could that value be compared with the packets received as reported by a controller?
    The values would need timestamps to be able to be correlated.
    I understand that just the act of querying a device can change the traffic pattern and skew results.

    Second train:
    A controller knows if it has sequence errors. But sequence errors by themselves don't mean a lot as pointed out by JChuchla.
    What if a controller knows (or discovers) the refresh rate. It could know when the next packet was expected and record/report the lag. Couple that with packet sequence numbers to determine if there was truly a dropped packet.
    I do not know the controller hardware internals. Do controllers have accurate enough timers to discover the refresh rate and detect lags? Would the overhead kill them?
    I wonder if sporadic's ESPixelStick hardware and code base would be up to that challenge?

    3rd train:
    I like the idea of a Metrics Pixel (MP) at the end of a string (controller output?).
    What small processor would have:
    a programmically accessible (refresh time)/2 or higher precision timer
    a fast/large enough SD card for data logging
    a wifi connection for command and control (maybe a web page)
    Does that sound like an ESP, Pi, or BBB? I keep wondering if sporadic's ESPixelStick could handle it? What about porting to a beefier ESP?
    Vampire power? Power inject? Battery?
    Maybe the layout could be configured to include a single pixel custom prop at the end of every controller output to represent MP's. That way the show player would not need mods. The custom prop may need a continuously changing pattern to force updates. Would that induce problems?

    4th train:
    If all I am really interested in is how well my non-waterproof connector (Anderson PowerPole, but don't hi-jack the thread) performs with pixel data when exposed to the elements, I suppose I could put a O-scope beyond the connector and look at the wave form while torturing the connector with rain water, ground runoff, salt spray, ice, snow, etc. That would require me to have and know how to use a scope, neither of which are true.

  11. #19
    Join Date
    Nov 2011
    Location
    Chicago - Southwest Suburbs
    Posts
    7,569
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    If all you're concerned about is data transfer across the connector, then your 4th train of thought is the most appropriate. It could be a dual channel scope. One channel on each side of the connector. You'll be looking to compare the waveforms from each channel. Knowing how to interpret the differences requires a bit of knowledge, but recognizing a that there's a difference is easy. You can also use a data logger to accomplish similar.

    But really, aside from changes in environmentals, the differences across the connector will remain rather constant. There's not much need to monitor it over time. Spot checks would be sufficient.

    All of the current pixel controllers will refresh the entire string of pixels on every update.

    Trying to read the data at the end of a pixel line against data sent from a controller is going to require intimate knowledge on how the controller's firmware works. Some are synchronous, some are asynchronous. For example, the last time I looked at the 681 firmware was baco around version 2. But I bet it's still similar. The processes that collected sACN data from the controller and the process that drives the pixels are completely decoupled. The listener part listens to the sACN data coming in, and stuffs it into a buffer. The pixel drive part just loops thru the buffer constantly refreshing the pixel string. So you have no correlation (except for perfect world math) between data frames received vs pixel frames sent. This is an asynchronous design. Each controller make and model may do this differently. it could be the opposite where the controller only refreshes the string once per received packet. The way to know which way is to send the controller some data and then stop. Then within a minute or less, unplug the pixels and plug them back in. If they come back to where they left off, it's constantly refreshing. If they don't come on, or come on to something other than before, then it's probably only one refresh per data packet or a synchronous design.

    For any reliable results, you want your monitoring system to be out of band from the system you're monitoring. IN other words, don't use the same network to carry diagnostic data as you're using for the data you're monitoring. They really should be isolated systems, decoupled as much as possible.

  12. #20
    Join Date
    Aug 2012
    Location
    Charleston, SC
    Posts
    1,085
    Post Thanks / Like

    Default Re: How you tell if your system is healthy?

    At one time I was entertaining the idea of an onboard current monitor that you could correlate with show data to determine if a string goes out. Power injection kills it though.

    Sent from my Moto G (5) Plus using Tapatalk

Page 2 of 3 FirstFirst 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •