AAUSAT-4 decoding difference between gr_satellites and SatNOGS

[janvgils]

AAUSAT-4 data decoding differences

I know there is currently a thread on the SatNOGS community to standardize the decoding between multiple software builders and that AAUSAT is one of the satellites that is causing confusion.

When looking at the following SatNOGS observation that is done by the AAUSAT ground station, you can see there is a difference between the decoded data.

Maybe it is possible based on the data and the link at the bottom of this issue to solve the difference?

The SatNOGS observation that is used for the comparison https://network.satnogs.org/observations/3289222/

AAUSAT-4 decoded SatNOGS data:

data_obs/3289222/data_3289222_2020-12-13T00-47-42
00 AA 92 48 27 60 17 00 12 2B DF 5F D5 63 E7 7E 0E D9 C2 F6 17 00 0F 0F A3 69 AF 00 00 0A 55 00 00 00 00 0F 14 F8 42 F8 2D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF FF 00 00 00 00 00 00 00 00 00 00 00 00

Used gr_satellites cli command to replay the observation:

gr_satellites AAUSAT-4 --wavfile AAUSAT-4_satnogs_3289222_2020-12-13T00-47-30.wav --samp_rate 48e3 --hexdump

Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
* MESSAGE DEBUG PRINT PDU VERBOSE *
((transmitter . 2k4 FSK downlink) (rs_errors . 0) (iterations . -1))
pdu_length = 92
contents = 
0000: 00 56 00 aa 92 48 27 60 17 00 12 2b df 5f d5 63 
0010: e7 7e 0e d9 c2 f6 17 00 0f 0f a3 69 af 00 00 0a 
0020: 55 00 00 00 00 0f 14 f8 42 f8 2d 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 00 00 09 41 00 00 00 00 ff ff 00 00 
0050: 00 00 00 00 00 00 00 00 00 00 9d af 
***********************************
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
Volk warning: no arch found, returning generic impl
* MESSAGE DEBUG PRINT PDU VERBOSE *
((transmitter . 2k4 FSK downlink) (rs_errors . 0) (iterations . -1))
pdu_length = 92
contents = 
0000: 00 56 00 a6 92 48 27 60 17 00 12 2c 75 5f d5 64 
0010: 7d 7e 0e d9 c1 02 16 00 0d 0d a3 69 af 00 00 0a 
0020: 59 00 00 00 00 01 4a fe 9d 0b a6 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 00 00 09 41 00 00 00 00 ff ff 00 00 
0050: 00 00 00 00 00 00 00 00 00 00 10 f0 
***********************************

Here is also a link that I totally forgot that can maybe explain the behaviour: AAUSAT Information

[daniestevez]

Hi Jan,

Thanks for reporting this. Now I understand better the difference between SatNOGS and gr-satellites. In fact, @nickoe already mentioned this in his comment. He said that he preferred the HMAC to be included in the DB entries, but he saw that most of the entries stripped the HMAC, so he decided to strip it too in the decoder he did for SatNOGS.

From your link, I see that the HMAC is the last 2 bytes of the FEC data. As I mentioned in my comment, gr-satellites includes everything in the FEC frame. Now I see that this also includes the HMAC, which are in fact the last two bytes 10 f0 that gr-satellites shows, while the SatNOGS observation doesn't.

Now my question is: why most of the DB entries have the HMAC stripped? I understand that the SatNOGS decoder that Nick did is a relatively new thing, so these entries don't come from SatNOGS. Does the UZ7HO soundmodem strip the HMAC? Did an older version of gr-satellites strip the HMAC? (this could be checked in the git history).

I'm happy to change the gr-satellites decoder, but I think the decision for AAUSAT-4 should be given to Nick, since he is the operator of this satellite. Should we change all the decoders to include the HMAC in the DB entries or to strip it away?

[nickoe]

When I started work on my decoder there was no decoder in gr-satellites IIRC. You "thankfully" based your work on my endeavor to learn gnuradio, where I made my first "decoder", as documented in https://destevez.net/2016/05/a-bunch-of-data-from-aausat-4/ :)

As far as I understand most packages in the db before I started pushing data was introduced via the SIDS interface. And I think most data originated from DK3WN's software suite -- decoders and telemetry forwarder. So I suspect it was stripped here.

[janvgils]

Apart from the decoder running at the AAUSAT-4 ground station that is operated by Nick, there is no AAUSAT decoder within the SatNOGS network. So all the decoded data being send are coming from gr_satellites via SIDS or the SoundModem solution from UZ7HO in combination with the SIDS Telemetry Forwarder made by Mike DK3WN.

So when Nick gives his final solution this should also be changed by Andy UZ7HO.

[daniestevez]

When I started work on my decoder there was no decoder in gr-satellites IIRC. You "thankfully" based your work on my endeavor to learn gnuradio, where I made my first "decoder", as documented in https://destevez.net/2016/05/a-bunch-of-data-from-aausat-4/ :)

Sorry for messing up the timeline! Since you mentioned "I just made my AAUSAT4 satnogs decoder" I assumed you had added an AAUSAT-4 decoder to SatNOGS recently (which I guess is also true) and forgot that you wrote the original AAUSAT-4 decoder. As you've said, this was a very helpful resource for me to implement my AAUSAT-4 decoder (and this was before gr-satellites really existed as a project).

As Jan mentioned, most of the frames in the DB come from UZ7HO's soundmodem and from gr-satellites. I don't know what UZ7HO's soundmodem does, but I'm digging a bit in the old versions of gr-satellites and it seems it stripped the HMAC and it also stripped the length field. The decision to strip the length field actually comes from Jeppe's code, and the decision to strip the HMAC, who knows.

I believe this was changed in 361479d (December 2019), in which the current behaviour where the length field and HMAC are not stripped are introduced. So I guess this explains why most of the frames in the database don't have the length field and HMAC.

[janvgils]

To complete the information I have also used the UZ7HO HS Soundmodem version 0.21b with AAUSAT-4 support and that produces the following data:

1: [AAUSAT4] [22:41:51R]
[priority:0 src:0 src_port:18 dest:10 dest_port:42 flags:HMAC len:88 BIT_err:0 RS_err:0]
00 AA 92 48 27 60 17 00 12 2B DF 5F D5 63 E7 7E 0E D9 C2 F6 17 00 0F 0F A3 69 AF 00 00 0A 55 00 00 00 00 0F 14 F8 42 F8 2D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF FF 00 00 00 00 00 00 00 00 00 00 00 00 

1: [AAUSAT4] [22:44:20R]
[priority:0 src:0 src_port:18 dest:10 dest_port:26 flags:HMAC len:88 BIT_err:1 RS_err:0]
00 A6 92 48 27 60 17 00 12 2C 75 5F D5 64 7D 7E 0E D9 C1 02 16 00 0D 0D A3 69 AF 00 00 0A 59 00 00 00 00 01 4A FE 9D 0B A6 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF FF 00 00 00 00 00 00 00 00 00 00 00 00 

I hope this will be enough information to standardize the AAUSAT-4 demodulation.

[daniestevez]

Thanks Jan,

So it seems that UZ7HO also strips the length field and HMAC. Again, I don't really remember the timeline (and this time I would need to dig through old emails rather than the git history), but it may had happened that Andy UZ7HO based his decoder on mine and so chose to do the same thing that the old gr-satellites did.

I think we already have enough information about what each software does and the historical reasons for that, and we can go and decide what we want to do for the future.

[janvgils]

Thanks Daniel,

Nick will need to answer your question and then when it is clear gr_satellites and the SoundModem from Andy can be altered.
I don't think the Forwarder from Mike needs to change but lets see.

[nickoe]

I just had a small call with Daniel. We essentially agreed that it makes most sense to drop the length field and include the hmac. In principle we could also keep the length field, but it is sort of redundant because at this time you how big the frame is.

@janvgils I guess we need UZ7HO to agree and notify DK3WN for consistency.

[janvgils]

Thanks for the update, I have just informed the others that are involved.

[janvgils]

I will use the following observation to compare all the difference demodulation options:

https://network.satnogs.org/observations/3289222/

This is the data that was send/stored in the SatNOGS database:

00 AF 92 48 27 60 17 00 12 2D DC 5F D5 65 E4 7E 0E D9 C0 F6 16 00 09 09 A3 69 AF 00 00 0A 64 00 00 00 00 00 ED FF 8D 06 29 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF FF 00 00 00 00 00 00 00 00 00 00 00 00

The first new release that is available is from UZ7HO and that produces the following output:

This is the method used to decode:

replay the satnogs obs -> VB audio cable routing -> UZ7HO HS SoundModem -> GetKISS+ -> dev SIDS database

SoundModem Output:

1: [AAUSAT4] [09:14:17R]
[priority:0 src:0 src_port:18 dest:10 dest_port:62 flags:HMAC len:92 BIT_err:0 RS_err:0]
00 56 00 AF 92 48 27 60 17 00 12 2D DC 5F D5 65 E4 7E 0E D9 C0 F6 16 00 09 09 A3 69 AF 00 00 0A 64 00 00 00 00 00 ED FF 8D 06 29 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF FF 00 00 00 00 00 00 00 00 00 00 00 00 B5 71 

GetKISS Output:

2020-12-17 08:14:17.850
Packet number 8
0000: C0 00 00 56 00 AF 92 48 27 60 17 00 12 2D DC 5F 
0010: D5 65 E4 7E 0E D9 DB DC F6 16 00 09 09 A3 69 AF 
0020: 00 00 0A 64 00 00 00 00 00 ED FF 8D 06 29 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF 
0050: FF 00 00 00 00 00 00 00 00 00 00 00 00 B5 71 C0 
0060:

SIDS database entry:

005600AF924827601700122DDC5FD565E47E0ED9C0F616000909A369AF00000A640000000000EDFF8D0629000000000000000000000000000000000000000000000000000000094100000000FFFF000000000000000000000000B571

Apart from some leading and ending bytes this seems to be same, please share your thoughts so we can decide if this new UZ7HO version can be used in the future.

When the gr-satellites AAUSAT change becomes available I will test the same setup if necessary.

[daniestevez]

Hi Jan,

Thanks for running this test. The SIDS data base entry obtained with UZ7HO's soundmodem + GetKISS+ has the problem that it includes the length field (the initial 0056 ). Together with Nick we decided not to include this field in the DB. I think that UZ7HO's misinterpreted Nick's sentence "In principle we could also keep the length field", so I'm going to reply on the email thread we have with Nick, since this needs to be modified.

I'll also make the change in gr-satellites master branch today and will link this issue in the commit so that you can test it.

[daniestevez]

As an aside note to your test, Jan, the SatNOGS DB entry is also "wrong" according to the decision Nick and I made. It's missing the HMAC, which are the last two bytes (B571). Nick is going to fix this in the SatNOGS decoder.

[daniestevez]

I have just pushed a commit to master that removes the length field, as we decided.

Here is my test of this with the test recording in satellite-recordings. In the previous revision 31fd8df, we get

$ gr_satellites AAUSAT-4 --wavfile satellite-recordings/aausat_4.wav --samp_rate 48e3 --hexdump
* MESSAGE DEBUG PRINT PDU VERBOSE *
((transmitter . 2k4 FSK downlink) (rs_errors . 0) (iterations . -1))
pdu_length = 92
contents = 
0000: 00 56 00 b1 92 48 27 00 03 00 00 54 14 57 1f 01 
0010: 26 6e 0e db c7 fe f8 7f 10 11 a3 00 04 00 3e 02 
0020: 38 ff a5 18 00 11 3b 03 43 f7 1a 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 00 00 01 57 00 00 00 00 ff ff 00 00 
0050: 00 00 00 00 00 00 00 00 00 00 3b 00 
***********************************

In the new revision ac8e32f, we get

$ gr_satellites AAUSAT-4 --wavfile satellite-recordings/aausat_4.wav --samp_rate 48e3 --hexdump
* MESSAGE DEBUG PRINT PDU VERBOSE *
((transmitter . 2k4 FSK downlink) (rs_errors . 0) (iterations . -1))
pdu_length = 90
contents = 
0000: 00 b1 92 48 27 00 03 00 00 54 14 57 1f 01 26 6e 
0010: 0e db c7 fe f8 7f 10 11 a3 00 04 00 3e 02 38 ff 
0020: a5 18 00 11 3b 03 43 f7 1a 00 00 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 01 57 00 00 00 00 ff ff 00 00 00 00 
0050: 00 00 00 00 00 00 00 00 3b 00 
***********************************

We can also see that the HMAC (3b 00) is included.

[janvgils]

I have used the same observation as the previous tests, now with the latest gr-satellites version and see below the outcome.

https://network.satnogs.org/observations/3289222/

gr_satellites AAUSAT-4 --wavfile Recordings/AAUSAT-4_satnogs_3289222_2020-12-13T00-47-30.wav --samp_rate 48e3 --hexdump

* MESSAGE DEBUG PRINT PDU VERBOSE *
((transmitter . 2k4 FSK downlink) (rs_errors . 0) (iterations . -1))
pdu_length = 90
contents = 
0000: 00 af 92 48 27 60 17 00 12 2d dc 5f d5 65 e4 7e 
0010: 0e d9 c0 f6 16 00 09 09 a3 69 af 00 00 0a 64 00 
0020: 00 00 00 00 ed ff 8d 06 29 00 00 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 09 41 00 00 00 00 ff ff 00 00 00 00 
0050: 00 00 00 00 00 00 00 00 b5 71

[daniestevez]

After some email discussion with Nick, Andy UZ7HO, Jan and Mike, we have decided to keep both the packet length field and the HMAC in frames sent to the database. The main reason for this decision is that we are not so sure on why it was originally included, so we think that the safest option is to include it, even though it is probably redundant in most cases.

As Nick wrote:

Anyway, I think, lets keep both the length and hmac. Both are part
of the FEC data anyways, and the decoder has it for later debugging if
needed.

I will now revert ac8e32f.

[janvgils]

Decoding with the help of HS SoundModem version 2.6 and using the same observation.

'https://network.satnogs.org/observations/3289222/'

1: [AAUSAT4] [23:02:12R]
[priority:0 src:0 src_port:18 dest:10 dest_port:62 flags:HMAC len:92 BIT_err:0 RS_err:0]
00 56 00 AF 92 48 27 60 17 00 12 2D DC 5F D5 65 E4 7E 0E D9 C0 F6 16 00 09 09 A3 69 AF 00 00 0A 64 00 00 00 00 00 ED FF 8D 06 29 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF FF 00 00 00 00 00 00 00 00 00 00 00 00 B5 71

The kiss frame generated by GetKISS+

2020-12-17 22:02:12.410 Packet number 4 0000: C0 00 00 56 00 AF 92 48 27 60 17 00 12 2D DC 5F 0010: D5 65 E4 7E 0E D9 DB DC F6 16 00 09 09 A3 69 AF 0020: 00 00 0A 64 00 00 00 00 00 ED FF 8D 06 29 00 00 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0040: 00 00 00 00 00 00 00 00 00 09 41 00 00 00 00 FF 0050: FF 00 00 00 00 00 00 00 00 00 00 00 00 B5 71 C0 0060:

The SIDS database entry (this is development SIDS db)

005600AF924827601700122DDC5FD565E47E0ED9C0F616000909A369AF00000A640000000000EDFF8D0629000000000000000000000000000000000000000000000000000000094100000000FFFF000000000000000000000000B571

Decoding based on the reverted gr-satellites build

gr_satellites AAUSAT-4 --wavfile Recordings/OGG/AAUSAT-4_satnogs_3289222_2020-12-13T00-47-30.wav --samp_rate 48e3 --hexdump

* MESSAGE DEBUG PRINT PDU VERBOSE *
((transmitter . 2k4 FSK downlink) (rs_errors . 0) (iterations . -1))
pdu_length = 92
contents = 
0000: 00 56 00 af 92 48 27 60 17 00 12 2d dc 5f d5 65 
0010: e4 7e 0e d9 c0 f6 16 00 09 09 a3 69 af 00 00 0a 
0020: 64 00 00 00 00 00 ed ff 8d 06 29 00 00 00 00 00 
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0040: 00 00 00 00 00 00 09 41 00 00 00 00 ff ff 00 00 
0050: 00 00 00 00 00 00 00 00 00 00 b5 71 
***********************************

Currently this will be the "final" outcome and produces similar data by all demodulators.

Thank you to all that are involved, to reach this outcome.

[janvgils]

@nickoe Is all the data that is send by the different solution the same so it produces the same information at the database level?

If so, please let us know so we can close this issue.

[nickoe]

@janvgils Yes, everything looks good now. I think we can close this issue. Thank all you very much for the effort :)

[daniestevez]

Ok. Closing. Thanks everyone!