Pskreporter is a a website which is used by the vast majority of radio amateurs worldwide for tracking, and understanding, how their signals are propagating around the world. It’s an amazing example of how how low-end, relatively simple hardware and software can be used to support enterprise level amounts of traffic.

At any given moment pskreporter can be receiving up to 500 reports per second coming in from thosands of amateurs worldwide, and distributing maps in almost equally large numbers.

In enterprise settings an entire team of of devops engineers would be keeping such a system running, but pskreporter has been run by just one (heroic) man, (Phillip Gladstone) for well over a decade on a single instance of a MySQL server. It’s a classic case of the lack of need for premature optimisation. :-). Admittedly, with the new resurgence of ham radio, and digital ham communications, it’s finally beginning to show it’s age and and is subject to outages and and quite some serious slowdowns on occasion.

Indeed, because of the slowness of the site, (and the impatience of hams), the load is frequently exacerbated by users constantly refreshing the page instead of waiting for the 5 minute auto update.

Last summer asked Philip if I might get a raw feed of reports from pskreporter with the goal of testing a new architecture where the reports were fed into an MQTT broker.

MQTT is a lightweight protocol, originally designed for the Internet of Things, to allow large numbers of very small systems to report status into industrial control systems. Over the last couple of years, it’s become the goto system for all kinds of home automation and other projects.

The MQTT feed would enable other systems to subscribe it and create real-time, streaming applications while being totally decoupled and creating no extra load on on the main system.

To my pleasant surprise not only did Philip agree to provide such a feed, but he also put me in contact with Tom Fanning, who is a radio amateur based out of Reading in the UK and who had actually put together such a broker, and had been running it for several years, but just for personal use. Tom and I got together and very shortly I had access to the broker. I then started work on the first application for public use based on the feed provided.

The resulting web application is called Spotty. Spotty produces maps with spots displayed in real-time as they come in. And there is no update button! And Spotty is totally decoupled from the main pskreporter site, and thus generates no extra load. Because the MQTT broker messages are carefully abbreviated JSON messages, the amount of bandwidth required is kept to a minimum.

Currently, Spotty produces maps which will track signals sent and received by just one ham, or by all hams in a fairly wide geographic area. So I usually track my callsign, G8LKD, and compare it with signals from the area defined as IO91. This covers most of central and southern England.

This is therefore an example of the new(ish) technologies from the Internet of Things meeting well tried and proven tech which has been running for many years.

Pleasantly, other programming hams are now connecting into the broker and creating other applications with different functionality. This is pretty easy as the broker exposes a simple to use API. To date, it has required little to no support.

Tom is now produced a a sampled version of the worldwide traffic giving us just 1% of the traffic of the full feed. The reason for this is that taking a 500 message per second feed is sizeable task for any system and also the the bandwidths costs to Tom’s system would be considerable at scale. However a 1% feed will provide a statistically significant measure of activity over just a few minutes.

My next project will be to use this feed to produce longer term trend analyses of how radio propagation changes over hours, days, and indeed months.

This has been a great example of how great improvements to system design can be made with relatively low effort and disruption, assuming one does not have management in the way. :-)