This last week has been a hectic one with the usual repair & maintenance taking place across the various island locations where we stream live from 24x7x365 and have done for many years. Much of the planned work on this occasion was to replace equipment that was damaged back in June - the same incident that may have been obvious for shoppers of a large supermarket on the south side of Lerwick.
To be clear…over the years these remote and unmanned island locations have been engineered (many over-engineered) to anticpate electrical incidents or communications failures. Most of those incidents are related to unforeseen weather related events or the occasional fibre optic cable break north and/or south of Shetland. Due to the over-engineering then most of these failures can be masked from those enjoying the live streams.
In recent times the number of power incidents that are not directly related to weather or that cover a much wider area have increased with a direct impact on the cost and frequency of repairs.
We do employ surge protection and Uninterrupted Power Supplies (UPS) in many locations but space is limited in most rural locations so we take calculated risks and accept that power supplies will likely fail…expensive fuses but usually (not always) they will protect the expensive kit.
On Tuesday 29th September it would appear that SSEN Distribution were also doing maintenance and/or system testing. Who knew…
I was also preparing a major update across our SD-WAN (Software-Defined Wide Area Network) - a virtualised technology that uses software to intelligently manage, control, and optimise how data moves across a wide-area network connecting separate webcam locations and our cloud infrastructure. Ironically, this was to allow for additional resilience and to allow a new site in Scalloway to come online later this year. What could possibly go wrong…
To manage this configuration I was using a PC in Lerwick that was not on a protected mains supply. This work commenced shortly before 2pm and changes were being made acoss many remote locations in parallel. Around 7-10 minutes into this work then all power was lost and it was also lost across many of our rural locations that were taking instructions from Lerwick. Needless to say I had some heartburn…
It quickly became clear I had lost contact with the majority of our remote locations and the ones I could get into via other routes were now wrongly configured, brain dead or incorrectly routing traffic on the road to nowhere.
The NorthLink ferry south was my home on Tuesday evening but the variable 4G or StarLink WiFi just caused more frustration. An onward journey south on an LNER Azuma was a great demonstration of just how poor the mobile coverage really is.
As of 9pm on Wednesday 30th September a plan has been made to try to recover some of those locations but it will take time to implement and to avoid making the current situation worse. Some visits to remote locations will be needed to replace items. This is simply not sustainable financially.
What would be really useful is yet another power cut - a nice clean OFF and then ON. I’m sure I won’t have too long to wait.
