Introduction

The previous post in this series discussed an FT8 DoTDoA proof-of-concept that I built using web SDRs and software running on my home computer. Since then, I've been looking at ways to take advantage of the FT8 decoding that is already constantly being performed across the globe. The most obvious candidate seemed to be KiwiSDR but I've recently shifted my focus to OpenWebRX+.

There are two major factors in OpenWebRX+'s favor:

1. It uses WSJT-X's jt9 executable to decode WSJT modes. This makes it possible to apply my jt9 modifications by simply swapping my modified jt9 executable into the OpenWebRX+ server.
2. It can be configured to push decodes to an MQTT broker. This makes it easy to access the decode information from anywhere, provided that the broker is made visible to the internet.

The Distributed Architecture

The following figure illustrates the proposed distributed architecture. Anybody with an HF SDR can easily contribute decoded data by setting up an OpenWebRX+ server. Their decoded data will be sent to a centrqal MQTT broker by the OpenWebRX+ software on their server. Anybody wishing to process the data can connect to the MQTT broker which will send them an aggregated stream.

I've settled on this particular architecture with one centralized MQTT broker for two reasons:

• It simplifies the setup for contributers. Without a centralized MQTT broker, each contributor's server would need to run its own MQTT broker.
• It eliminates the need for contributors to map ports and allow incoming connection requests. All connection requests are outbound from the contributors, avoiding possible security issues related to open ports and incoming requests.

The rest of this post will discuss what you need to do in order to become a contributing participant by running your own server.

Install OpenWebRX+ on a Dedicated Server

Luarvique's instructions are probably the best place to start. Raspberry Pi images, Docker images, Ubuntu installs, Debian installs are discussed there.

Since you will likely want OpenWebRX+ to run 24/7, it's best to install it on a spare computer that you'll dedicate to the task. I tried a Raspberry Pi but it seemed to struggle when simultaneously monitoring/decoding multiple modes (e.g. FT4, FT8, and WSPR). Next, I tried an old Dell OptiPlex 3020 SFF and it performed admirably. Old computers like the Dell OptiPlex 3020 and HP EliteDesk 800 G1 are dirt cheap and easy to find.

Replace OpenWebRX+'s jt9 Executable

In a previous post, an enhanced version of jt9 was discussed. Your OpenWebRX+ server will need this same enhanced version but OpenWebRX+ installs the standard version. I'd suggest the following rough plan for replacing it:

• Download the WSJT-X source to your OpenWebRX+ server
• Make the enhancement to decoder.f90
• Build (but don't install) WSJT-X
• Find the new jt9 binary that was built and copy it over the jt9 that was installed by OpenWebRX+

Improve OpenWebRX+'s Jt9Decoder

Since we're now using the enhanced jt9, we need to make one small change to owrx/wsjt.py in order for OpenWebRX+ to correctly parse the more-precise output of the modified jt9. As shipped, owrx/wsjt.py includes a parsing class named Jt9Decoder but it doesn't quite work, for our purposes. To be exact, it unnecessarily assumes that each field in the output will have a specific width. Since the enhanced jt9 will produce wider "DT" values, we need to make Jt9Decoder more flexible by replacing it with the following:

class Jt9Decoder(Decoder):

    metadata_pattern = re.compile(r"\s*(\S+)\s+(\S+)\s+(\S+)\s+\S+\s+(.*\S)\s*")

    def parse(self, msg, dial_freq):
    # ft8 sample
    # '222100 -15 -0.0  508 ~  CQ EA7MJ IM66'
    # jt65 sample
    # '2352  -7  0.4 1801 #  R0WAS R2ABM KO85'
    # '0003  -4  0.4 1762 #  CQ R2ABM KO85'
    # fst4 sample
    # '**** -23  0.6 3023 `  <...> <...> R 591631 BI53PV'
    # MSK144 sample
    # '221602   8  0.4 1488 &  K1JT WA4CQG EM72'
    msg, timestamp = self.parse_timestamp(msg)
    match = self.metadata_pattern.match(msg)
    db_str, dt_str, df_str, wsjt_msg = match.groups()

    result = {
        "timestamp": timestamp,
        "db": float(db_str),
        "dt": float(dt_str),
        "freq": dial_freq + int(df_str),
        "msg": wsjt_msg
    }

    result.update(self.messageParser.parse(wsjt_msg))
    return result

This new code parses the metadata according to whitespace instead of field positions/widths and is compatible with both the original jt9 and my enhanced version.

I make this change by directly modifying the Python code that OpenWebRX+ installed on the server. In my case, that's:

/usr/lib/python3/dist-packages/owrx/wsjt.py

Note that this change will be overwritten if you subsequently do an apt upgrade that installs a newer version of OpenWebRX+, so either:

• Be prepared to make the same surgical change if things stop working after an upgrade.
• Run sudo apt-mark hold openwebrx to prevent automatic updates of OpenWebRX+. The hold approach is recommended.

Modify OpenWebRX+ To Post More FT8 Info

When it comes to posting FT8 messages to MQTT, note that standard OpenWebRX+ only posts those that have explicit locators. For our purposes, this is very bad. Our algorithm can potentially use any decoded message so we want to receive all of them via MQTT.

OpenWebRX+ has also chosen to post FT8 messages individually. For our purposes, this is inconvenient. Our algorithm works by examining pairs of messages sent in the same FT8 time period, so it would be more convenient if we received FT8 messages grouped by the time period in which they were sent.

Both of these issues can be addressed by enhancing OpenWebRX+ to post MQTT messages that look like this:

{
    "h2h_type": "org.ham2ham.cospots.v1", 
    "receiver": { "who": "kr0dak", "where": "DM42KJ" }, 
    "cospots": [
        {"dB":  -6, "time": 1762625085240, "freq": 14075492, "mode": "FT8", "msg": "CQ N3AZ EL09"}, 
        {"dB":   6, "time": 1762625085165, "freq": 14075151, "mode": "FT8", "msg": "KJ7WLL KN6RBP 73"}, 
        {"dB":   7, "time": 1762625085170, "freq": 14074936, "mode": "FT8", "msg": "AI7QQ K6GOH -10"}, 
        {"dB":  -2, "time": 1762625085425, "freq": 14075210, "mode": "FT8", "msg": "VE3TXI AA5HH EM21"}, 
        {"dB":  -7, "time": 1762625085235, "freq": 14074530, "mode": "FT8", "msg": "CQ POTA N6SVA CM97"}, 
        {"dB":  -6, "time": 1762625085470, "freq": 14076403, "mode": "FT8", "msg": "W8F W6GRE R-05"}, 
        {"dB": -17, "time": 1762625085325, "freq": 14075866, "mode": "FT8", "msg": "YO6OGW K5HK DM09"}, 
        {"dB":  -9, "time": 1762625086030, "freq": 14075573, "mode": "FT8", "msg": "WA8ZID KI5HCX -01"}, 
        {"dB": -18, "time": 1762625085165, "freq": 14075708, "mode": "FT8", "msg": "WE4RR KJ5CYB EM30"}, 
        {"dB":   0, "time": 1762625085355, "freq": 14076835, "mode": "FT8", "msg": "KN4TLV N4MNW 73"}, 
        {"dB": -11, "time": 1762625085425, "freq": 14074331, "mode": "FT8", "msg": "KS4YX KF0THN -04"}, 
        {"dB": -19, "time": 1762625084805, "freq": 14075921, "mode": "FT8", "msg": "CQ N1PRR DM33"}, 
        {"dB": -12, "time": 1762625085195, "freq": 14074747, "mode": "FT8", "msg": "AE6CH KB5A R-04"}, 
        {"dB": -15, "time": 1762625085290, "freq": 14075292, "mode": "FT8", "msg": "CQ AB8PS EN83"}, 
        {"dB": -18, "time": 1762625085180, "freq": 14075532, "mode": "FT8", "msg": "CQ N0VTY EN40"}, 
        {"dB":  -7, "time": 1762625085185, "freq": 14074896, "mode": "FT8", "msg": "VE2KJM AC7WY DN61"}, 
        {"dB": -19, "time": 1762625085360, "freq": 14074295, "mode": "FT8", "msg": "9U1RU NE0NS EN62"}, 
        {"dB":  -9, "time": 1762625084930, "freq": 14076399, "mode": "FT8", "msg": "W9POG KR0P RR73"}, 
        {"dB": -24, "time": 1762625085310, "freq": 14075875, "mode": "FT8", "msg": "S51VY KD2RUY -25"}, 
        {"dB": -11, "time": 1762625085185, "freq": 14075143, "mode": "FT8", "msg": "N1IPB AD9GE +08"}, 
        {"dB": -20, "time": 1762625085260, "freq": 14075485, "mode": "FT8", "msg": "KR4EOD KE8SAS R+07"}
    ]
}

I'm not exactly sure how best to add this functionality into OpenWebRX+ but I have settled on a reasonable hack for now. At the end of audio/chopper.py there is a function named sendResult. Replace the entire function definition with the following, being sure to maintain the same indentation:

    def sendResult(self, result:QueueJobResult):

        datas = []  # AMB is collecting these for h2h, later
        for line in result.lines:
            data = self.parser.parse(result.profile, result.frequency, line)
            if data is not None: # AMB
                # Presence of "  " is a simple check for a-priori and/or low-conf tags.
                if "  " not in data["msg"]:  datas.append(data)
            if data is not None and self.writer is not None:
                self.writer.write(pickle.dumps(data))

        # AMB cospots for h2h
        try:
            mode = result.profile.getMode()
            if mode not in ["FT4", "FT8"]: return  # I'm focusing on FT4 and FT8, for now.
            if len(datas) > 0:
                reporters = ReportingEngine.getSharedInstance().reporters
                mqtt_rptr = next(filter(lambda r: isinstance(r, MqttReporter), reporters))
                pm = Config.get()
                topic = f"h2h/{pm['receiver_name']}/cospots"  # DO NOT add a subtopic per mode
                cospots = []
                for data in datas:
                    cospot = {
                        "dB":   int(data["db"]),
                        "time": int(data["dt"]*1000) + data["timestamp"],
                        "freq": data["freq"],
                        "mode": data["mode"],
                        "msg":  data["msg"]
                    }
                    cospots.append(cospot)
                payload = {
                    'h2h_type': "org.ham2ham.cospots.v1",
                    'receiver': {
                        'who': pm["receiver_name"],
                        'where': Locator.fromCoordinates(pm["receiver_gps"]).upper(),
                        'default_resolutions' : {
                            "FT8": {'dB': 1.0, 'time': 5.0, 'freq': 1.0}
                            # FT4 and others would go here...
                        }
                    },
                    'cospot-count': len(cospots),
                    'cospots': cospots
                }
                mqtt_rptr.client.publish(topic, payload=json.dumps(payload))
        except Exception:
            logger.exception("error sending cospots to MQTT")

Configure OpenWebRX+ to use MQTT

In this section I'm going to use {you} as a placeholder for your callsign. Wherever you see that, just replace it with your actual callsign, in lower case.

The first thing you'll need to do is request an MQTT username and password from me. Then, using the OpenWebRX+ web interface, navigate to "Settings" > "Spotting and reporting" > "MQTT settings" and:

• Set "Broker address" to kr0dak.ddns.net:8883 (this is an alias for the HiveMQ MQTT)
• Set "Client ID" to {you}-owrx
• Set "Username" to {you}-owrx (this should be the username I provided)
• Set "Password" to the password I provided.
• Set "MQTT topic" to openwebrx/{you}

Testing

You can verify that everything is operating correctly by using an MQTT client like MQTT Explorer. We're going to configure MQTT Explorer to use the public, read-only account which is different than the account that I created specifically for you to send data to the MQTT broker. Set up MQTT Explorer as follows:

• Set "Name" to guest
• Set "Validate certificate" ON
• Set "Encryption" ON
• Set "Protocol" to mqtt://
• Set "Host" to kr0dak.ddns.net
• Set "Port" to 8883
• Set "Username" to guest
• Set "Password" to guest

Next, click on the "Advanced" button and add the following topics:

• h2h/#
• openwebrx/#

Now click on "Back", click on "Save", and click on "Connect". After a few seconds, you should see something similar to the following figure if you click on the triangles to expand the topics.

Check the following:

• Standard messages from my OpenWebRX+ should appear under openwebrx/kr0dak
• Standard messages from your OpenWebRX+ should appear under openwebrx/{you}
• Enhanced messages from my OpenWebRX+ changes should appear on h2h/kr0dak/cospots
• Enhanced messages from our OpenWebRX+ changes should appear on h2h/{you}/cospots

Conclusion

If even a very small number of additional people choose to set up similar servers (or choose to modify their existing servers accordingly) the result will be an easy-to-use and distributed source of timing information to enable FT8 (Do)TDoA experiments by anybody.

Given as few as three such servers, I'd begin modifying my analysis software (roughly described in the previous post) to make use of the data that would be constantly flowing from those servers via MQTT. And there are other projects that can be built on this sort of distributed system, which I'll likely discuss in future posts.

If I can help you get involved, please feel free to get in touch with me. I'm adrianboyko at that email site that almost everybody uses.

73s for now!