This is the software that runs the Rudolph de Harak digital clock on the side of 200 Water Street in New York.
The clock was orginally installed in 1971. Each digit contained a set of florecent bulbs, some with colored gels. There was a very large relay-based controller that occupied the space that is now Starbucks.
Unfortunately this setup proved to be very dificult to maintain so in 2015 Rockrose (the owner) upgraded the clock to use modern LED fixures controlled by a tiny digital computer.
This repo contains the master controller software. It runs on a Raspberry Pi using the Raspberrian Linux distro. It controls the 72 independent digit panels using Open Pixel Server formated UDP packets.
The controller is accessable via ssh and also publishes a web page where you can run demo displays and check some diagnostics.
Each digit panel contains a BeagleBone Black computer running the LEDscape package to drive the attached LEDs.
The master controller uses DNSMASQ to hand out IP addresses and maps those addresses to digits by MAC address using DNS entries in the dhcp-hosts
file.
Each panel gets a DNS name that is mapped to its location on the clock. The top row of digits are H01
- H12
, and the bottom 5 rows are M00
- M59
.
The clock runs continuously using the local linux time. The local time is disciplined using NTP whenever there is a connection to the internet. The timezone is set inside the clockmode.sh
script, as are the colors for hours, minutes, seconds, and background.
In the case when minutes and seconds are on the same digit, seconds win.
The background is currently set to turn on at 8PM and off at 7AM. This saves a bit of power and wear as the dim background light is not visible during daylight. This is set at the top of the clockmode.sh
script.
All of the wiring and controls are in the closet behind the front desk at 200 Water Street.
The breaker panel controls power to the clock. Each breaker controls several columns, which are labeled in sharpie.
These are AFCI breakers, so they can half trip anytime there is a ground current detected. In this case, they need to be switched off and then on again to reset. If an AFCI breaker trips repeatedly, it like means that either there is water getting into the panels or outside conduit, or that the breaker needs to be replaced.
There is a master relay to the left of the panel that turns off all circuits. There is a control switch for the master breaker outside on the bottom right of the clock itself...
THIS SWITCH IS WATER DAMAGED AND SWITCHES UNPREDICABLY. It is therefore important to keep the "relay" breaker turned off. This leaves the relay in the ON
postion and disables the non-functional outdoor switch.
A single CAT5 network cable comes in from each digit box to the patch panel in the control room. These cables run in a bundle over theroof of the adjacent resturant.
There is a patch cable that connects each digit to one of the two switches. The Raspberry Pi master controller is connected to one of the switches. The switches are configured to run at 10Mbs/half.
The raspberrypi also uses Wifi to connect out to the internet through an access point under the front desk in the lobby.
This internet connection is used to...
- Keep the clock synced to the correct time and adjust for DST
- Provide the web-based demo control page using a Dataplicity wormhole. The page is served by a local Apache running on the pi.
- Allow remote SSH connections via both Dataplicity and R3motiT.
Listed expected order of likelihood
Likely a blown breaker, probably after rain. Find the corresponding breaker on the panel and run the breaker reset procedure below.
First try reseting the whole column using the Breaker Reset procedure below. If that does not work, then the panel is likely damaged and needs to be replaced with a spare. Sorry.
THis indicates that the digit panels are not getting updated packets from the controller. This could be due to...
- a power failure
- the controller is unplugged or not running
- a network connectivity problem between the controller and the rest of the clock.
Try running the startup procedure below. If you are not able to access the controller to run the procedure, check to see if it has power and is connected to the network.
This pattern indicates that the digit hasis not getting any updated packets (OPC) from the controller. This can either indicate that the controller is not sending packets becuase no effect is running, or that the panel has lost its network connection to the controller.
Check the entire network path from the swicth to the digit.
- The patch cable from the switch to the patch panel
- The punchdown on the back of the patch panel
- The cable running up into the ceiling and up over the roof (these cables are exposed to the elements!)
- The connector inside the digit box
- The plug going into the board on the digit panel
Either...
- There is still a wet short in the circuit. You might have to wait until a dry day.
- Dead breaker. Replace the breaker.
The controller lost its connection to the internet. Try rebooting the Timer Warner router in the lobby front desk. If the internet service has changed (front desk gets a different Wifi network), then you will need to TELNET onto the controller and update the network config in the file /etc/wpa_supplicant/wpa_supplicant.conf
.
There is a chon
job that runs midnight.sh
every night at midnight and restarts the clockmode.sh
script every night...
00 00 * * * /home/pi/De-Harak-Clock-Controller/midnight.sh
Use crontab -e
if you ever need to edit or re-add this line.
There is a cron
job that runs checkdigitsl.sh
every minute.
It writes to log files in /home/pi/logs/
with the date as the name.
Each line starts with the 24h time followed by one char per digit. Hours first, then minutes.
Char | Meaning |
---|---|
. | Good |
P | No response to ping, but name in DNS so digit was alive recently |
H | No hostname in DNS, so digit has not DHCPed in a while and entry expired |
For example:
pi@raspberrypi:~/clock-controller $ cat ~/logs/20180112.log
1607......P......P..............P........................................H..
1608......P......P..............P........................................H..
1609......P......P..............P........................................H..
1610......P......P..............P........................................H..
1611......P......P..............P........................................H..
Here is what the clock looked like whne those log lines were written...
pi@raspberrypi:~/clock-controller $ ./checkdigits.sh
H means host not found in DNS
P means failed to ping
h01 h02 h03 h04 h05 h06 Ph07P h08 h09 h10 h11 h12
m00 Pm01P m02 m03 m04 m05 m06 m07 m08 m09 m10 m11
m12 m13 m14 m15 Pm16P m17 m18 m19 m20 m21 m22 m23
m24 m25 m26 m27 m28 m29 m30 m31 m32 m33 m34 m35
m36 m37 m38 m39 m40 m41 m42 m43 m44 m45 m46 m47
m48 m49 m50 m51 m52 m53 m54 m55 m56 Hm57H m58 m59
This log format is helpful to see what happened if several digits have problems at once. You can go back and see if, say, a whole column went out at the same moment - in which case it was probably a breaker problem.
Easiest way to install is to use the release in this repo to flash a new SD card. Then cd
into the repo and do a git pull
to make sure everything is up to date.
Use tzselect
to set the right timezone if necessary.
Clone this repo into /home/pi/clock-controller
Run install.sh
Use sudo raspi-config
to disable graphical desktop on boot (boot into CLI).
The clock should start automatically on boot when the pipereader.service
starts.
You can also manally send a command to the pipereader to switch to clockmode like this...
echo "clockmode" >/tmp/clockcommands
The pipereader is a little script that reads lines form the clockcommands
FIFO and then executes scripts in the clock software install directory. The FIFO makes it
so that anyone can send commands without path or permisions issues (like the cgi script for the webinterface), and also serializes the commands so they don't
step on each other.
Then go look at the clock. You should see most digits looking like a clock, but a few with rainbow patterns. These digits did not successfully connect to the network on boot (see here for info). To fix these digits, they must be repowered. Find a column on the clock that has one or more unconnected digits and then find the circuit breaker that controls that column. Run the breaker reset proceedure below.
- Switch the breaker off (all the way to the right) until it clicks.
- Count to 3-Mississippi. You want a full 3 seconds here so don't rush.
- Switch the breaker back on (to the left) until it clicks. If it does not click satisfyingly, then see troubleshooting above.
- Give the column a minute to boot
- Go out to the clock and check the digits in the reset column(s). If any are still showing the rainbow pattern, then repeat starting at step #1 until all digits are showing the clock pattern.
There are lots of demos and test patterns available.
Each has a command that the pipereader should know. Look in pipereader.sh
for a list.
For example, to start the sinbowwave pattern (a sinbow is a rainbow by with sine shaped color curves), you'd enter...
echo "sinbowwave" >/tmp/clockcommands
Always remember to restart clock mode when you are done! (Or chon should do it automatically at midnight)
All panels are identical, so can easily be swapped out with spares in case of problems.
Once a digit has been swapped, you need to update the DHCP/DNS entry for the MAC address of the new panel to map to the correct location on the clock. To do this, on the Pi command line run...
digitpicker.sh
Scroll though and find the old panel that was located in the replaced position and change the name to something like deadH20
. Next look for an entry with a name like beaglebone
or *
rather than a good name. This is likely the new panel. Hit enter and the digit should blink on the clock. Next enter the correct DNS name for this position (i.e. H20
) and press enter. Then save the file and restart the clockmode program so the changes take effect.
There is a hidden wifi access point in the digit 57 box to give direct access to the internal network from in front of the clock. This can be helpful in cases where you are trying to find a digit and the wifi in the lobby is not working. SSID and key are written on paper inside the binder in the clock equipment room.
There are several scripts showing examples of how to update the display for modes other than clock. All use OPC over UDP to set the pixels in each digit. ticktest.sh
is a good place to start. It lights up each digit in sequence with red, green, blue, and white. Note that you can use any RGB color. To run ticktest
, enter...
echo "sinbowwave" >/tmp/clockcommands
For more complicated displays, C is likely faster than shell scripts. The beginnings of some C drivers are in maptest.c
.
You can use the checkdigits.sh
script to get a quick overview of which digits are working...
pi@raspberrypi:~/De-Harak-Clock-Controller $ ./checkdigits.sh
H means host not found in DNS
P means fialed to ping
h01 Hh02H h03 h04 Hh05H h06 Hh07H Hh08H h09 h10 Hh11H h12
m00 Hm01H m02 m03 Hm04H m05 Hm06H Hm07H m08 m09 Hm10H m11
m12 Hm13H m14 m15 Hm16H m17 Hm18H Hm19H m20 m21 Hm22H m23
m24 Hm25H m26 m27 Hm28H m29 Hm30H Hm31H m32 m33 Hm34H m35
m36 Hm37H m38 m39 Hm40H m41 Hm42H Hm43H m44 m45 Hm46H m47
m48 Hm49H m50 m51 Hm52H m53 Hm54H Hm55H m56 m57 Hm58H m59
pi@raspberrypi:~/De-Harak-Clock-Controller $
In this view, for example, digits h01
and m17
are working fine but h11
and m10
do not have DNS entries (yikes!!!).
Looking at the above pattern of completely dead columns, it seems likely that a couple of blown breakers is the cause. In this case, use the 3-Mississippi procedure above on each breaker to to reset it.
This simple utility sleeps until the next second on the clock. If it is 10:55:22.75 and you run ./nextsecond
, then it will sleep for about 0.25 seconds.
The clock uses this to always start an update at the top of a second.
You can test that it works by running...
./nextsecond; date "+%N"
It will print the number of nanoseconds since the last round second> I should always be < 010000000, which is 10,000,000ns, which is 10ms.
From the controller, you can ssh
onto the digits like this...
ssh -l root m20
...where m20
is the digit you want. No password needed.
Instructions to build new digit panels... https://www.instructables.com/id/Massive-Neopixel-WS2812B-Display-Panels/
Then flash the LEDscape image on to the Beaglebone by putting an SD card with this image in it... https://github.com/bigjosh/LEDscape/releases/tag/1.1 ...and holding the USR button (near SD slot) at power up until you see cylon LED pattern.
At power up, the board should go into redbeat
pattern where it flashes red 1 second per minute.
To check the LED layout is correct, send something like a bullseye pattern from a controller.
sudo systemctl disable /root/DigitPanelDemo/ledsd.service
sudo killall leds
Steps to make a BBB into a controller if you don't want to use the premade image above, here are steps to create it..
- Start with installing Debian 7.11 2015-06-15 4GB SD LXDE. http://beagleboard.org/getting-started
- Install
Ledscape
https://github.com/bigjosh/LEDscape#installation - Install
bbbphyfix
https://github.com/bigjosh/bbbphyfix#install - Install
devmemkb
https://github.com/bigjosh/devmemkm#installation - Remove
wicd
...apt-get remove wicd*
nano /etc/network/interfaces
- Uncomment these two lines...
auto eth0
iface eth0 inet dhcp
- Install
ledscape-config.json
with...wget --directory-prefix=/etc/ https://raw.githubusercontent.com/bigjosh/De-Harak-Clock-Controller/master/bbb/ledscape-config.json
sync
Note that we remove wicd
since it uses lots of CPU when idle and this increases the chances of the ARM having memory contention when the PRUs go to access the GPIO pins, which causes white flashes.
- Put in 4Gb SD card
sudo /opt/scripts/tools/eMMC/beaglebone-black-make-microSD-flasher-from-eMMC.sh
nano /boot/uEnv.txt
- Uncomment the flasher line
cmdline=init=/opt/scripts/tools/eMMC/init-eMMC-flasher-v3.sh
ssh root@h01 -oStrictHostKeyChecking=no sudo systemctl disable /root/DigitPanelDemo/ledsd.service
ssh root@h01 -oStrictHostKeyChecking=no sudo killall leds
A chon takes a log snapshot of all working digits every minute using checkdigitsl.sh
. They are in /home/pi/logs/
.