Support Ublox NTRIP Client (No MQTT) SPARTN format

[jptalledo10]

PyGPSClient Feature Request Template

Ublox/Thingstream released a new service based on NTRIP/IP and providing SPARTN format corrections without any MQTT dependencies.

Wondering if this application can add this feature and provide a server for it to connect and retrieve those corrections

[semuadmin]

Hi @jptalledo10 ,

PyGPSClient can already parse (but not yet fully decode) SPARTN data from MQTT and L-Band sources, so in theory there should be no major difficulty in parsing SPARTN data from an NTRIP (i.e. HTTP) source.

The main issue is that, while the SPARTN protocol is ostensibly in the public domain, the documentation is (in my experience) relatively poor and u-blox's SPARTN services all require a paid subscription in order to decrypt the raw data, making it extremely difficult for an unpaid and unsponsored volunteer project like PyGPSClient to develop, test and implement it successfully. SPARTN is basically a money-spinner for u-blox.

The NTRIP RTCM3 protocol, on the other hand, is well documented, unencrypted and a great many NTRIP sources are available free of charge from public domain service providers like rtk2go.com and euref-ip.com.

u-blox have shown little interest in supporting open source projects like PyGPSClient and certainly no interest in assisting with subscription fees or testing, so at this point I don't feel greatly motivated to implement new SPARTN functionality in support of u-blox's revenue stream.

As things stand, the pyspartn library (which PyGPSClient uses to parse, decrypt and decode SPARTN data) is currently still in Alpha and cannot yet fully decode all SPARTN payloads. I suspect the outstanding issues are probably easily resolved (they're almost certainly a document interpretation problem rather than a decryption problem), but I've simply not had the time or frankly the incentive to look into them in any depth.

Having said this, contributions of all kinds are always welcome so if anyone out there is in a position to assist with the implementation of such functionality, I'd be more than happy to incorporate it.

[jptalledo10]

The good news is that Ntrip Spartn is not encrypted and I can share a developer account for debugging/testing if needed. Happy to discuss further.

[semuadmin]

The good news is that Ntrip Spartn is not encrypted and I can share a developer account for debugging/testing if needed. Happy to discuss further.

That would certainly be helpful - appreciate the offer. In point of fact, you don't actually need to decrypt SPARTN data in order to send it to an RTK-compatible GNSS receiver like the ZED-F9P - you can simply send the raw binary datastream along with the relevant decryption keys and the receiver will happily decrypt it itself. Native SPARTN decryption in pyspartn is provided more as a diagnostic and academic service to end users.

I've been reviewing the PointPerfect service description at https://developer.thingstream.io/guides/location-services/pointperfect-ntrip-distribution and it all looks relatively straightforward, particularly as it appears to support the standard NTRIP HTTP port 2101 in addition to an encrypted HTTPS (TLS) service on port 2102.

Supporting the service would require the following changes:

1. (necessary) Enhancements to the pygnssutils library - specifically the gnssntripclient class. It should be a simple matter to add a flag to signify either RTCM3 or SPARTN protocol, and then assign either pyrtcm.RTCMReader or pyspartn.SPARTNReader to parse the data into individual RTCMMessage or SPARTNMessage objects and forward the serialised objects to the receiver. Also minor change to accept TLS port 2102 in addition to 443.
2. (necessary) Add the additional input parameters to the PyGPSClient NTRIP Client GUI.
3. (optional) For HTTPS support, I may also need to add a parameter which points to the location of the necessary PointPerfect TLS certificates, assuming these are not already supported natively by the Python certifi library via its default cacerts.pem TLS certificate registry.
4. (optional) enhancements to the pyspartn library to fix the remaining SPARTN decode issues - something that's been on my backlog for a while anyway. Having access to a developer account would certainly facilitate this, but I can't guarantee when I'd be able to get to it.

I'm moving this issue to the PyGPSClient Ideas and Feature Requests Discussion Forum where we can continue the discussion, if that's OK with you.

Thanks again for the suggestion and any assistance you can offer.

[semuadmin]

FYI... https://github.com/semuconsulting/pygnssutils/tree/add-support-for-SPARTN-NTRIP-service

If you want to give this a quick test, you can try the following (substituting your PointPerfect NTRIP region and user credentials):

python3 -m pip install git+https://github.com/semuconsulting/pygnssutils.git@add-support-for-SPARTN-NTRIP-service
gnssntripclient -S ppntrip.services.u-blox.com -P 2101
gnssntripclient -S ppntrip.services.u-blox.com -P 2101 -M US --protocol SPARTN --ntripuser yourusername --ntrippassword yourpassword

The first gnssntripclient command should give you the sourcetable for the PointPerfect NTRIP service. The second should hopefully give you a partially decoded SPARTN data stream at the terminal. Let me know how you get on.

[semuadmin]

@jptalledo10

FYI I've just tested my provisional changes to pygnssutils.gnssntripclient myself using my own PointPerfect NTRIP credentials (it transpires that if you deactivate and reactivate any existing PointPerfect MQTT IP subscription, a new set of NTRIP credentials is created automatically) and it appears to work fine:

python3 -m pip install git+https://github.com/semuconsulting/pygnssutils.git@add-support-for-SPARTN-NTRIP-service
gnssntripclient -S "ppntrip.services.u-blox.com" -P 2101 -M EU --protocol SPARTN --ntripuser "**********-****-****-****-************" --ntrippassword "**********"

DEBUG server ppntrip.services.u-blox.com protocol SPARTN
2024-03-17 18:02:44.146279: Using mountpoint EU

2024-03-17 18:02:44.147213: <SPARTN(SPARTN-1X-HPAC-GLO, msgType=1, nData=72, eaf=0, crcType=2, frameCrc=5, msgSubtype=1, timeTagtype=0, gnssTimeTag=32532, solutionId=5, solutionProcId=11, crc=15482173, )>
2024-03-17 18:02:44.149103: <SPARTN(SPARTN-1X-HPAC-BEI, msgType=1, nData=401, eaf=0, crcType=2, frameCrc=3, msgSubtype=3, timeTagtype=0, gnssTimeTag=21736, solutionId=5, solutionProcId=11, crc=7697768, )>
2024-03-17 18:02:44.150977: <SPARTN(SPARTN-1X-HPAC-BEI, msgType=1, nData=413, eaf=0, crcType=2, frameCrc=0, msgSubtype=3, timeTagtype=0, gnssTimeTag=21736, solutionId=5, solutionProcId=11, crc=10946928, )>
2024-03-17 18:02:44.152735: <SPARTN(SPARTN-1X-HPAC-BEI, msgType=1, nData=390, eaf=0, crcType=2, frameCrc=6, msgSubtype=3, timeTagtype=0, gnssTimeTag=21736, solutionId=5, solutionProcId=11, crc=10553253, )>
2024-03-17 18:02:44.154557: <SPARTN(SPARTN-1X-HPAC-BEI, msgType=1, nData=407, eaf=0, crcType=2, frameCrc=11, msgSubtype=3, timeTagtype=0, gnssTimeTag=21736, solutionId=5, solutionProcId=11, crc=5303528, )>
2024-03-17 18:02:44.155090: <SPARTN(SPARTN-1X-HPAC-BEI, msgType=1, nData=90, eaf=0, crcType=2, frameCrc=8, msgSubtype=3, timeTagtype=0, gnssTimeTag=21736, solutionId=5, solutionProcId=11, crc=2030559, )>
2024-03-17 18:02:44.157060: <SPARTN(SPARTN-1X-HPAC-GAL, msgType=1, nData=459, eaf=0, crcType=2, frameCrc=15, msgSubtype=2, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=3065994, )>
2024-03-17 18:02:44.160058: <SPARTN(SPARTN-1X-HPAC-GAL, msgType=1, nData=518, eaf=0, crcType=2, frameCrc=0, msgSubtype=2, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=13578585, )>
2024-03-17 18:02:44.162254: <SPARTN(SPARTN-1X-HPAC-GAL, msgType=1, nData=559, eaf=0, crcType=2, frameCrc=3, msgSubtype=2, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=12483637, )>
2024-03-17 18:02:44.163973: <SPARTN(SPARTN-1X-HPAC-GAL, msgType=1, nData=495, eaf=0, crcType=2, frameCrc=6, msgSubtype=2, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=11408191, )>
2024-03-17 18:02:44.164305: <SPARTN(SPARTN-1X-HPAC-GAL, msgType=1, nData=99, eaf=0, crcType=2, frameCrc=15, msgSubtype=2, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=15962411, )>
2024-03-17 18:02:44.164925: <SPARTN(SPARTN-1X-OCB-GPS, msgType=0, nData=210, eaf=0, crcType=2, frameCrc=7, msgSubtype=0, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=12913608, )>
2024-03-17 18:02:44.165468: <SPARTN(SPARTN-1X-OCB-GLO, msgType=0, nData=188, eaf=0, crcType=2, frameCrc=6, msgSubtype=1, timeTagtype=0, gnssTimeTag=32532, solutionId=5, solutionProcId=11, crc=1522920, )>
2024-03-17 18:02:44.166021: <SPARTN(SPARTN-1X-OCB-BEI, msgType=0, nData=193, eaf=0, crcType=2, frameCrc=3, msgSubtype=3, timeTagtype=0, gnssTimeTag=21736, solutionId=5, solutionProcId=11, crc=15612021, )>
2024-03-17 18:02:44.166941: <SPARTN(SPARTN-1X-OCB-GAL, msgType=0, nData=191, eaf=0, crcType=2, frameCrc=6, msgSubtype=2, timeTagtype=0, gnssTimeTag=21750, solutionId=5, solutionProcId=11, crc=11664803, )>
etc...

I made some quick changes to PyGPSClient to accommodate the new functionality and, as you can see from this screen shot, it all seems to work fine - the SPARTN data is being successfully parsed from the NTRIP caster to the ZED-F9P receiver, the RXM-COR messages confirm they are being used (msgUsed = 2) and the fix status has duly changed to RTK-FLOAT (this was a quick indoor test so the results won't be spectacular):

If you're able to confirm you get similar results using the pygnssutils.gnssntripclient CLI utilty, I'll endeavour to get the changes into PyGPSClient RC 1.4.13 by the end of the month.

[jptalledo10]

That's great news and pretty quick turnaround. Thanks for your support. I will try it out

[jptalledo10]

@semuadmin Can you advise/suggest a better way to redirect the output from gnssntripclient?

is it like : --output socket 127.0.0.1:40002

Thanks

[semuadmin]

As it stands, the output argument is really only useful when using the GNSSNTRIPClient.run() method within a Python script, where output can represent either a serial stream, file, queue or socket instance (PyGPSClient, for example, invokes GNSSNTRIPClient internally with an output argument of queue). At the command line, stdout is the only available option - sorry, I should clarify this in the documentation.

Other CLI utilities within pygnssutils - e.g. gnssdump - can use an evaluable Python expression (e.g. lambda function) as an output argument, but this does not currently apply to the gnssntripclient CLI. I could add this at some point in the future but I think it would be of limited utility.

When I get a moment, I'll add a simple example script in the examples folder to illustrate how to output GNSSNTRIPClient output to a socket.

FYI Version 1.0.22 of pygnssutils, which includes support for the PointPerfect NTRIP SPARTN service, has now been released: https://github.com/semuconsulting/pygnssutils/releases/tag/v1.0.22.

I'm in the process of finalising testing on the corresponding PyGPSClient changes and these should be available in v1.4.13 within the next week or so: #122

[semuadmin]

u-blox PointPerfect NTRIP SPARTN service support (ppntrip.services.u-blox.com) is now available in PyGPSClient version 1.4.13.

Note that the underlying pyspartn library is still in Alpha and not yet fully functional. PyGPSClient can display incoming SPARTN transport-layer messages and their associated metadata (SPARTN-1X-HPAC-GPS*, SPARTN-1X-OCB-GLO* etc.) in the console, but the payloads are not yet fully decoded.

[jptalledo10]

Thanks for the quick turnaround.

Do you mind providing an example how to consume the SPARTN corrections as TCP socket stream?

[semuadmin]

Can you clarify your use case? An NTRIP service is itself a TCP socket stream, albeit in an HTTP(s) wrapper.

[jptalledo10]

I was trying to follow up with the example you can provide on how to use the GPS client CLI to provide the output so it can be consumed and received by a ntrip client. My use case is a ZED-F9P able to consume SPARTN Corrections over internet without U-Center. I don't think there are available NTRIP client supporting SPARTN corrections right?

…

On Thu, Mar 21, 2024, 7:51 AM SEMU Admin ***@***.***> wrote: Can you clarify your use case? An NTRIP service is itself a TCP socket stream, albeit in an HTTP(s) wrapper. — Reply to this email directly, view it on GitHub <#124 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH464ZE4PXPXHBUH6UFW6HTYZLCVZAVCNFSM6AAAAABE2NGNBSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNRUG43TI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[semuadmin]

Bear in mind you're in the Discussion channel for PyGPSClient, which is a Graphical User Interface application, similar to u-Center.

If you're after CLI functionality or NTRIP code examples, pygnssutils is the place to be.

See, for example, rtk_example.py, which illustrates how to connect to an NTRIP caster and pass the correction data (RTCM3 or SPARTN) to a suitable GNSS receiver e.g. a ZED-F9P. You can edit and run this script using u-blox's ppntrip.services.u-blox.com service and your own access credentials and receiver serial port configuration, and you should see something like the following log output (don't forget to change the DATATYPE argument to "SPARTN"):

python3 rtk_example.py

Starting GNSS reader/writer on /dev/tty.usbmodem101 @ 38400...

GNSS<< CFG-VALSET
Estimated horizontal accuracy: 1.166 m
Estimated horizontal accuracy: 1.159 m
Starting NTRIP client on ppntrip.services.u-blox.com:2101...

GNSS<< SPARTN-1X-HPAC-GLO
GNSS<< SPARTN-1X-HPAC-GLO
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
Estimated horizontal accuracy: 3.161 m
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> NAV-PVT
Estimated horizontal accuracy: 3.580 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
Estimated horizontal accuracy: 2.292 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.801 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.546 m
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
GNSS>> NAV-PVT
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
Estimated horizontal accuracy: 1.389 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
Estimated horizontal accuracy: 1.285 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.212 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.160 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.122 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
Estimated horizontal accuracy: 1.094 m
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.073 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.056 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.043 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
Estimated horizontal accuracy: 1.033 m
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
GNSS>> NAV-PVT
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
Estimated horizontal accuracy: 1.025 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.017 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.011 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
Estimated horizontal accuracy: 1.007 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 1.001 m
GNSS>> NAV-PVT
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
Estimated horizontal accuracy: 0.998 m
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> NAV-PVT
Estimated horizontal accuracy: 0.993 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
Estimated horizontal accuracy: 0.988 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 0.983 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 0.978 m
GNSS>> NAV-PVT
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
Estimated horizontal accuracy: 0.974 m
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
Estimated horizontal accuracy: 0.970 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 0.966 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 0.962 m
GNSS>> NAV-PVT
Estimated horizontal accuracy: 0.959 m
GNSS>> NAV-PVT
GNSS>> NAV-SAT
GNSS>> NAV-DOP
GNSS<< SPARTN-1X-GAD
GNSS<< SPARTN-1X-GAD
GNSS<< SPARTN-1X-HPAC-GPS
GNSS<< SPARTN-1X-HPAC-GPS
GNSS<< SPARTN-1X-HPAC-GPS
GNSS<< SPARTN-1X-HPAC-GPS
GNSS<< SPARTN-1X-HPAC-GPS
GNSS<< SPARTN-1X-HPAC-GLO
GNSS<< SPARTN-1X-HPAC-GLO
GNSS<< SPARTN-1X-HPAC-GLO
Estimated horizontal accuracy: 0.956 m
GNSS<< SPARTN-1X-HPAC-GLO
GNSS<< SPARTN-1X-HPAC-GLO
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-BEI
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-HPAC-GAL
GNSS<< SPARTN-1X-OCB-GPS
GNSS<< SPARTN-1X-OCB-GLO
GNSS<< SPARTN-1X-OCB-BEI
GNSS<< SPARTN-1X-OCB-GAL
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> RXM-COR
GNSS>> NAV-PVT
etc...

As more continuous SPARTN data is received, you should see the Estimated horizontal accuracy increase steadily - to a few cm in ideal conditions.

For brevity sake, you may find it helpful to disable NMEA output and enable the bare minimum UBX message types instead (i.e. NAV-PVT, NAV-DOP, RXM-COR), as I've done here. If you want to see the full parsed messages, rather than just the identities, set idonly = False.