-
-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Submit decoded frames to SatNOGS observation #136
Comments
🪲 Missing frame reception timestamp for recordings When uploading frames to satnogs-network they must be submitted with the timestamp of reception as part of the filename. On a normal station this timestamp is created from the system clock in the gr-satnogs.frame_file_sink#L71-80. Thus we need some method to reconstruct this timestamp within gr-satellites for demodulated frames. 💡 Possible solution gr-satellites already supports the generation of reception timestamps when using the KISS output, but those timestamps are only correct when it is run in real-time during an observation, as they are similarly created from the system clock in pdu_to_kiss.py#L40-L56. For replayed recordings adding a @daniestevez What do you think about this solution & do you think it is possible? |
Thanks for your interest in this @kerel-fs! First of, if gr-satellites is run in real-time, this is a not an issue, as you said. Currently I'm thinking that the best way to run gr-satellites for integration with SatNOGS is in real-time, alongside gr-satnogs, rather than making it run afterwards on the IQ file generated by gr-satnogs. However, it would be nice to have some functionality to allow generating timestamps of recordings correctly. There is in fact an issue #92 open about this, but I haven't had time to work in it so far. Your suggestion is in fact how the older versions v1 and v2 of gr-satellites worked. The disadvantage is that this approach needs to play back the recording at 1x speed (with a throttle block). This is not very convenient. No one really wants to wait 10 minutes for a full pass recording to process in order to have accurate timestamps. gr-satellites is often capable of processing the full file in a matter of seconds. (As background information, in v1 and v2 it wasn't possible to process recordings as fast as allowed by the CPU. The flowgraphs only supported input by streaming in samples with UDP, so samples were lost if you streamed too fast, and streaming the recording at 1x speed was the only "supported" way to work. Therefore, it was reasonable to implement the timestamp handling as you suggested). As things work now, we read recordings as fast as the CPU allows (no throttle block at all in the flowgraph), so I think that timestamp handling should be implemented correctly by counting samples rather than using the UNIX clock. This is possible, but not so easy because GNU Radio doesn't have any notion of time or counting samples. My idea about how to do this is roughly:
So all this is doable but takes quite some time and I haven't set my head to doing it, so I guess it won't be done soon. While this approach makes total sense to me in applications much wider than gr-satellites, I haven't seen this done elsewhere. For bonus points this should be compatible with timestamp tags generated by UHD and similar radios. For example, if you have a meta-file including timestamps, then the timestamps should come from the file metadata rather than counting samples. In any case, I think that >90% of the use cases will be real-time processing. For the remaining cases, we could do as in gr-satellites v2 or even the user could do some hacky setup by using libfaketime or similar (I have never used this; it just came out of a Google search) and arranging for the recording to be played back in real-time (for instance by streaming it in real-time by UDP). So for the matter of the discussion in this thread, let's assume that gr-satellites is already producing good timestamps because it's running in real-time or the user has set-up something using libfaketime. Discussion about how is best to generate timestamps from a recording fits better in #92. TLDR: This only affects recording processing, and to allow processing the recordings faster than 1x this should be done by counting samples rather than using the UNIX clock/elapsed time. However, this takes significant effort and will not be done soon. Still, the real-time case is probably the most popular. |
Counting samples is exactly what I envisioned when I wrote "get the elapsed time" (ignoring for the fact that the recording could be missing samples). Thanks for the detailed explanation on how this could be properly implemented! Tbh I didn't expect counting samples to be so complicated (my gnuradio experience is limited).
Unfortunately I'll not be able to run gr-satellites in real-time (I'm not the station operator) so my use case here (and also the one described by @cpbridges elsewhere) is exactly one of the harder ones as it seems. :) |
Thanks for the clarification. I understood "get the elapsed time" as calling The difficulty of counting samples in GNU Radio comes mainly from having to deal with sample rate changes (especially variable rate, such as you would have for clock recovery), and because at some point you got to PDUs, so there aren't samples any more. As a workaround, would playing the recording at 1x speed be acceptable in your use-case? I'm thinking that it's easy to add a |
Yes, this would be fully acceptable for now! Running throttled is not a problem for me when dealing with a small number of observations to process (e.g. when running in a post-observation script or manually re-decoding selected recordings with gr-satellites), and the resulting accuracy wouldn't be much different than what we can expect when the timestamp was created by gr-satnogs (using system clock as well). |
Then I'll set to implement this. Probably during this afternoon. |
This is done now in 122ccea. I have tested that it is working with the KISS file sink. I haven't tested it with the SatNOGS DB submit block, since I don't have a recording with an accurate timestamp at hand and I don't want to fill the DB with tests. It should work, since most of the code is common. To use this, you need to include both the Please let me know if this works well for you, and in that case I'll add the description of this functionality to the documentation. |
Thanks for implementing this! I did some first test with observation 2901793 and the results are as expected (ie. inital offset due to setup time and potentially a small drift) and acceptable for the use case imho.
edit: The code for this short analysis can be found here. |
Thank you for the test. I'm wondering about the -4 s initial delta (this means that gr-satellites decoded the packet earlier than it should). Maybe the throttle block let some seconds of the wav file rush in at the beginning (it only regulates average rate). I also think an error of a few seconds is acceptable, any way. Keep me updated about how well this is working (I think there were some people interested in hooking up gr-satellites in the post-observation script) and if there is something else I can do to help. |
Just my 2 cents on testing: you can always use db-dev.satnogs.org for that. No worries about illegal frames or the amount of tests you're running. |
This idea originated here. gr-satellites is sometimes used as a post-observation script in SatNOGS. It would be good if it can send the demodulated frames to SatNOGS so that they appear in the "data" tab of the corresponding observation.
The text was updated successfully, but these errors were encountered: