Celestron AUX driver does not detect mount ceasing to track due (possibly) to meridian flip

[stevecoh1]

Describe the bug
The other night I set up my rig (Stellarmate Pro, Celestron AVX mount (GEM), etc.) to make 60-second captures of NGC_6946 over a 3-hour period from 2300 2024-08-19 to 0200 2024-08-20 local time (GMT-7hrs). This was set up using the Stellarmate App though, I imagine, the same results would have been obtained using KStars/Ekos directly.

The sequence ran to completion and shut itself down cleanly at 0200. 146 captures were made. However, while the first 69 captures showed reasonably good images of the target, the remaining 77 all showed streaks of light rather than stars, or the target. This was seen slightly on the 70th image, and progressively worse through the sequence.

Clearly, the mount had stopped tracking. My local astronomy club suggested that this was probably due to a meridian flip.

However, as may be seen in the attached log, no evidence of the driver having noticed the cessation of tracking occurred and the software continued to make captures, all of which showed streaks of light.

Noteworthy in the attached log:

The mount went through a few alignment iterations until finally achieving a satisfactory alignment on line 82117 of the log file at 2024-08-19T23:02:46.676 MST (between 2 and 3 minutes from the start of the job), when the state is recorded as "tracking".

[2024-08-19T23:02:46.676 MST DEBG ][ org.kde.kstars.ekos.mount] - New mount state for MF: "Tracking"

From that point there are no recorded mount state changes until 2024-08-20T02:00:03.679 MST, after completion of the job, when, as directed to the scheduler, the mount was parked (line 220990 of the log file).

[2024-08-20T02:00:03.679 MST DEBG ][ org.kde.kstars.ekos.mount] - Mount status changed from "Tracking" to "Parking"

I can confirm from looking at the rig in the morning, that the mount indeed moved to the park position.

To Reproduce
Difficult to reproduce, but I am attaching relevant images and full logging of the session. To reproduce, one would have to define a sequence that would be known to encompass a meridian flip, with full logging, as I did. One would need to be using the Celestron AVX mount.

Expected behavior
I am not sure what the correct behavior of the driver ought to be in such a situation, but at minimum I would expect something to be logged, if not shutting down the sequence altogether with an error message. But none of this happened.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Stellarmate OS
• Version: 1.8.4
• Driver version: official, as of 2024-08-19, not bleeding edge

Log Files
log_19-43-16.zip

[stevecoh1]

In case the actual FITS files and not just screenshots of them would be useful, I attach a zipped file of them here.
fitsfiles.zip

[knro]

Perhaps it's guiding issue? I don't see any flips in there. Apparently guide log was not active. Both Capture + Guide logs would be useful for this.

[jochym]

Actually, this is also an indicator of another deficiency in our code. We are doing things "blind". I.e. we are sending the commands and, usually, do nothing with the answer. My mount Wi-Fi sometimes hangs and stops responding. The driver happily sends further commands, not detecting the breakdown in communication. This break is probably of other nature, while it might be some error in comms. I think I/we need to review our error handling. That is my side-note. Unfortunately, I have no idea what this particular problem is. I think that we would need fool verbose logs to debug - since the origin of this park command may be anywhere from scheduler to the low-level part of the driver. @stevecoh1 Are you able to hunt for this bug a little by running your future sessions with full debugging? It is probably not that much of the disk space in comparison with images ?

[stevecoh1]

I certainly could do that. I thought it was pretty close to full debugging. I am not using a guide scope, so no guiding logging. Is there anything else that you would like me to turn on? I will do it.

Last night I felt the need to experiment with meridian flips. I tracked Vega, no astrophotography captures, just tracking as it crossed the meridian. I saw the setting in the mount module and turned on meridian flipping. I sat by the scope while it performed the meridian flip. Very impressive, actually.

It didn't immediately occur to me that "MF" stood for "meridian flip". (not the obscene connotation it has in English :-) ).

It is quite possible that WIFI comms played a role in this, but I think it was simply the lack of any logging when meridian flipping is turned off and therefore tracking stops.

Something like "WARNING: crossing Meridian, Meridian Flip not enabled, turning off tracking"

One other point. I noticed while scanning the logs after my successful MF experiment last night, that once the Meridian Flip had completed, Meridian flipping was disabled.

[2024-08-25T20:42:05.068 MST INFO ][   org.kde.kstars.ekos.capture] - "Telescope completed the meridian flip."
[2024-08-25T20:42:05.069 MST DEBG ][   org.kde.kstars.ekos.capture] - Capture State changes from "Meridian Flip" to "Idle"
[2024-08-25T20:42:05.076 MST DEBG ][     org.kde.kstars.ekos.mount] - Ha:  0.0118563  haLimit  2   "Pier Side: West (pointing East)"  haLimitReached  false  lastHa  0.0115775
[2024-08-25T20:42:05.374 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CAUX] CMD <MC_GET_POSITION>  APP ->  AZM "
[2024-08-25T20:42:05.424 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] CMD <50 01 10 01 00 00 00 03> "
[2024-08-25T20:42:05.425 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] RES (8 B): <3B 04 10 20 01 7E 9D 48> "
[2024-08-25T20:42:05.426 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] Got 4 bytes:  ; payload length field: 4 ; MSG: "
[2024-08-25T20:42:05.426 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] [00 04 10 20 01 7E] "
[2024-08-25T20:42:05.427 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CAUX] RES <MC_GET_POSITION>  AZM ->  APP [7E 9D 48] "
[2024-08-25T20:42:05.428 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CAUX] CMD <MC_GET_POSITION>  APP ->  ALT "
[2024-08-25T20:42:05.476 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] CMD <50 01 11 01 00 00 00 03> "
[2024-08-25T20:42:05.476 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] RES (8 B): <3B 04 11 20 01 1D 10 5E> "
[2024-08-25T20:42:05.476 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] Got 4 bytes:  ; payload length field: 4 ; MSG: "
[2024-08-25T20:42:05.477 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CSER] [00 04 11 20 01 1D] "
[2024-08-25T20:42:05.477 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[CAUX] RES <MC_GET_POSITION>  ALT ->  APP [1D 10 5E] "
[2024-08-25T20:42:05.478 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[DEBUG] Encoder [Axis1: 8297800 --> LST: 18:38:34 HA: -0:07:48 RA: 18:46:22] [Axis2: 1904734 --> DE: 40:52:16] "
[2024-08-25T20:42:05.479 MST DEBG ][           org.kde.kstars.indi] - Celestron Advanced VX Wired : "[ALIGNMENT] Mount -> Sky RA: 18:37:51 DE: 38:48:50 AZ: 358:47:22 AL: 83:20:42 HA:  2:01:24 Pier: PIER_WEST "
[2024-08-25T20:42:06.064 MST DEBG ][     org.kde.kstars.ekos.mount] - Setting meridian flip status to  "MOUNT_FLIP_NONE"
[2024-08-25T20:42:06.065 MST INFO ][     org.kde.kstars.ekos.mount] - "Status: inactive"
[2024-08-25T20:42:06.066 MST DEBG ][   org.kde.kstars.ekos.capture] - updateMFMountState:  "MOUNT_FLIP_NONE"
[2024-08-25T20:42:06.071 MST DEBG ][     org.kde.kstars.ekos.mount] - Ha:  0.0121348  haLimit  2   "Pier Side: West (pointing East)"  haLimitReached  false  lastHa  0.0118563

Is there a way to default-enable meridian flipping somewhere in the settings?