<html><head><style id="outgoing-font-settings">#response_container_BBPPID{font-family: initial; font-size:initial; color: initial;}</style></head><body style="background-color: rgb(255, 255, 255); background-image: initial; line-height: initial;"><div id="response_container_BBPPID" style="outline:none;" dir="auto" contenteditable="false"> <div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;">Thanks Wolfgang, </div><div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;"><br></div><div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;">I acknowledge what you analyse. </div><div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;"><br></div><div name="BB10" id="BB10_response_div_BBPPID" dir="auto" style="width:100%;">I added a test to verify that guiding would always be restarted if it was required (and disabled) at the time Focus was starting its procedure. But that makes Focus responsible for restarting Guide. So we are saying that it should be the same for Capture, with the risk that requests are mixed up. Doesn't that mean that Guide should integrate that task? Should we distinguish a failure, managed by Scheduler, and a procedure follow-up, managed by the module itself? </div> <div id="blackberry_signature_BBPPID" name="BB10" dir="auto"> <div id="_signaturePlaceholder_BBPPID" name="BB10" dir="auto"><p dir="ltr">eric.dejouhanet@gmail.com - https://astronomy.dejouha.net</p></div> </div></div><div id="_original_msg_header_BBPPID" dir="auto"> <table width="100%" style="border-spacing: 0px; display: table; outline: none;" contenteditable="false"><tbody><tr><td colspan="2" style="padding: initial; font-size: initial; text-align: initial;"> <div style="border-right: none; border-bottom: none; border-left: none; border-image: initial; border-top: 1pt solid rgb(181, 196, 223); padding: 3pt 0in 0in; font-family: Tahoma, "BB Alpha Sans", "Slate Pro"; font-size: 10pt;"> <div id="from"><b>De:</b> sterne-jaeger@t-online.de</div><div id="sent"><b>Envoyé:</b> 8 décembre 2020 07:51</div><div id="to"><b>À:</b> hy@murveit.com</div><div id="cc"><b>Cc:</b> eric.dejouhanet@gmail.com; mutlaqja@ikarustech.com</div><div id="subject"><b>Objet:</b> Re: scheduler issue</div></div></td></tr></tbody></table> <br> </div><!--start of _originalContent --><div name="BB10" dir="auto" style="background-image: initial; line-height: initial; outline: none;" contenteditable="false"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""></div>Dear all,<div class="">I’m not 100% sure, but for me it looks like the problem is located in the Capture module which - only in the case of a MF - is responsible for restarting the guiding procedure. </div><div class=""><br class=""></div><div class="">What we can see from the logs, the Capture module receives a GUIDE_CALIBRATION_ERROR event. For this event it calls processGuidingFailed(). There is the critical part:</div><div class=""><br class=""></div>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span><span style="color:#808000" class="">else</span><span style="color:#c0c0c0" class=""> </span><span style="color:#808000" class="">if</span><span style="color:#c0c0c0" class=""> </span>(<span style="color:#800000" class="">meridianFlipStage</span><span style="color:#c0c0c0" class=""> </span>==<span style="color:#c0c0c0" class=""> </span><span style="color:#800080" class="">MF_GUIDING</span>)</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span>{<!-- --></pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span><span style="color:#808000" class="">if</span><span style="color:#c0c0c0" class=""> </span>(++<span style="color:#800000" class="">retries</span><span style="color:#c0c0c0" class=""> </span>>=<span style="color:#c0c0c0" class=""> </span><span style="color:#000080" class="">3</span>)</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span>{<!-- --></pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span><span style="color:#00677c" class="">appendLogText</span>(<span style="color:#00677c" class="">i18n</span>(<span style="color:#008000" class="">"Post</span><span style="color:#c0c0c0" class=""> </span><span style="color:#008000" class="">meridian</span><span style="color:#c0c0c0" class=""> </span><span style="color:#008000" class="">flip</span><span style="color:#c0c0c0" class=""> </span><span style="color:#008000" class="">calibration</span><span style="color:#c0c0c0" class=""> </span><span style="color:#008000" class="">error.</span><span style="color:#c0c0c0" class=""> </span><span style="color:#008000" class="">Aborting..."</span>));</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span><span style="color:#00677c" class="">abort</span>();</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span>}</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span>}</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><span style="color:#c0c0c0" class=""> </span><span style="color:#800000" class="">autoGuideReady</span><span style="color:#c0c0c0" class=""> </span>=<span style="color:#c0c0c0" class=""> </span><span style="color:#808000" class="">false</span>;</pre>
<pre style="margin-top:0px;margin-bottom:0px" class=""><br class=""></pre><pre style="margin-top:0px;margin-bottom:0px" class=""><br class=""></pre><pre style="margin-top:0px;margin-bottom:0px" class=""><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class="">The meridianFlipStage == MF_GUIDING is true, but it happens for the first time, hence it does not abort.<br class=""><br class="">My explanation (without testing) is, that Capture is idling. The Scheduler also does nothing, since it only reacts upon guiding problems when it starts a job.</span></font></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class=""><br class=""></span></font></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class="">As a result, simply nothing happens - as we could see from the logs.</span></font></div><div class=""><br class=""></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class="">If somebody wants to fix it in the next two weeks, I could assist and advise. As a first step, I would warmly recommend creating a new test case for MF testing calibration error handling.</span></font></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class=""><br class=""></span></font></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class="">All the best</span></font></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class="">Wolfgang</span></font></div><div class=""><font color="#000000" face="Helvetica" class=""><span style="white-space:normal" class=""><br class=""></span></font></div></pre><div class=""><span style="color:rgb( 0 , 0 , 0 )" class="">—</span></div><div class=""><div class=""><div dir="auto" style="color:rgb( 0 , 0 , 0 );letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;word-wrap:break-word" class=""><div dir="auto" style="word-wrap:break-word" class=""><div style="color:rgb( 0 , 0 , 0 );font-family:'helvetica';font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">Wolfgang Reissenberger<br class=""><br class=""></div><div style="color:rgb( 0 , 0 , 0 );font-family:'helvetica';font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><a href="http://www.sterne-jaeger.de" class="">www.sterne-jaeger.de</a><br class="">TSA-120 + FSQ-85 | Avalon Linear + M-zero | Moravian G2-8300 + ASI 1600mm pro</div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">Am <a href="tel:08122020">08.12.2020</a> um 07:08 schrieb Hy Murveit <<a href="mailto:murveit@gmail.com" class="">murveit@gmail.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Just to be clear, at this point I'm not taking ownership of this, but really just acting as a bug reporter.<br class=""><div class="">If one of you wants to jump in and fix, or put it on a future TODO list, that would be great from my perspective.</div><div class="">On the other hand, if someone thinks I should fix it, then I'd need coaching. </div><div class="">I'd prefer the former, but would do the latter if necessary.</div><div class=""><br class=""></div><div class="">Hy</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 7, 2020 at 9:47 PM Eric Dejouhanet <<a href="mailto:eric.dejouhanet@gmail.com" class="">eric.dejouhanet@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex"><div style="background-color:rgb( 255 , 255 , 255 )" class=""><div id="gmail-m_1906432316115635784response_container_BBPPID" style="outline:none" dir="auto" class=""> <div id="gmail-m_1906432316115635784BB10_response_div_BBPPID" dir="auto" style="width:100%" class=""> Hello Hy, </div><div id="gmail-m_1906432316115635784BB10_response_div_BBPPID" dir="auto" style="width:100%" class=""><br class=""></div><div id="gmail-m_1906432316115635784BB10_response_div_BBPPID" dir="auto" style="width:100%" class="">From the code you pasted, state change is done before emitting state, so that seems proper. I think Jasem nailed it, there is a state management missing either in Capture, to report abort from guiding recovery failure, or Scheduler, to manage guiding failure. The fact it follows the meridian flip means, in my opinion, that we progressively approach the root cause of the remaining stability issues, and that we need to really isolate that flip process as much as we can. </div> <div id="gmail-m_1906432316115635784blackberry_signature_BBPPID" dir="auto" class=""> <div id="gmail-m_1906432316115635784_signaturePlaceholder_BBPPID" dir="auto" class=""><p dir="ltr" class=""><a href="mailto:eric.dejouhanet@gmail.com" class="">eric.dejouhanet@gmail.com</a> - <a href="https://astronomy.dejouha.net/" class="">https://astronomy.dejouha.net</a></p></div> </div></div><div id="gmail-m_1906432316115635784_original_msg_header_BBPPID" dir="auto" class=""> <table width="100%" style="border-spacing:0px;display:table;outline:none" class=""><tbody class=""><tr class=""><td colspan="2" class=""> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb( 181 , 196 , 223 );padding:3pt 0in 0in;font-family:'tahoma' , 'bb alpha sans' , 'slate pro';font-size:10pt" class=""> <div id="gmail-m_1906432316115635784from" class=""><b class="">De:</b> <a href="mailto:murveit@gmail.com" class="">murveit@gmail.com</a></div><div id="gmail-m_1906432316115635784sent" class=""><b class="">Envoyé:</b> 7 décembre 2020 22:56</div><div id="gmail-m_1906432316115635784to" class=""><b class="">À:</b> <a href="mailto:eric.dejouhanet@gmail.com" class="">eric.dejouhanet@gmail.com</a></div><div id="gmail-m_1906432316115635784reply_to" class=""><b class="">Répondre à:</b> <a href="mailto:hy@murveit.com" class="">hy@murveit.com</a></div><div id="gmail-m_1906432316115635784cc" class=""><b class="">Cc:</b> <a href="mailto:hy@murveit.com" class="">hy@murveit.com</a>; <a href="mailto:mutlaqja@ikarustech.com" class="">mutlaqja@ikarustech.com</a>; <a href="mailto:sterne-jaeger@t-online.de" class="">sterne-jaeger@t-online.de</a></div><div id="gmail-m_1906432316115635784subject" class=""><b class="">Objet:</b> Re: scheduler issue</div></div></td></tr></tbody></table> <br class=""> </div><div dir="auto" style="outline:none" class=""><div dir="ltr" class=""><div class="">Eric et al,</div><div class=""><br class=""></div><div class="">Not clear on how this works, but the code is copied below.</div><div class="">Note the emit newStatus().</div><div class="">What is the state change you are referring to?</div><div class=""><br class=""></div><div class="">FWIW, I've included the code snippet below (pretty complicated code to follow--there's a state machine in <a href="http://guide.cpp/" class="">guide.cpp</a> -- I guess that's either controlling the internal guider or PHD2), and there's a calibration state machine in <a href="http://internalguider.cpp/" class="">internalguider.cpp</a>. From what I can tell, I don't think it's the issue you described, but I'm not too familiar with these state machines, nor the scheduler.</div><div class=""><br class=""></div><div class="">Hy</div><br class=""><div class=""><br class=""></div><div class=""><b class="">From <a href="http://internalguider.cpp/" class="">internalguider.cpp</a>, line 505</b><br class="">void InternalGuider::processCalibration()<br class="">{<!-- --><br class=""> pmath->performProcessing();<br class=""><br class=""> <span style="background-color:rgb( 255 , 255 , 0 )" class="">if (pmath->isStarLost())</span><br class=""> {<!-- --><br class=""> emit newLog(i18n("Lost track of the guide star. Try increasing the square size or reducing pulse duration."));<br class=""> reset();<br class=""><br class=""> calibrationStage = CAL_ERROR;<br class=""> <span style="background-color:rgb( 255 , 255 , 0 )" class="">emit newStatus(Ekos::GUIDE_CALIBRATION_ERROR);</span><br class=""> emit calibrationUpdate(GuideInterface::CALIBRATION_MESSAGE_ONLY, i18n("Calibration Failed: Lost guide star."));<br class=""> return;<br class=""> }<br class=""></div><div class=""> ...</div><div class=""><br class=""></div><div class=""><b class="">From <a href="http://internalguider.cpp/" class="">internalguider.cpp</a>, line 542</b><br class=""></div><div class="">void InternalGuider::reset()<br class="">{<!-- --><br class=""> state = GUIDE_IDLE;<br class=""> //calibrationStage = CAL_IDLE;<br class=""> connect(guideFrame, SIGNAL(trackingStarSelected(int, int)), this, SLOT(trackingStarSelected(int, int)),<br class=""> Qt::UniqueConnection);<br class="">}<br class=""></div><div class=""><br class=""></div><div class=""><b class=""><a href="http://guide.cpp:2381/" class="">guide.cpp:2381</a></b></div><div class=""> connect(guider, &Ekos::GuideInterface::newStatus, this, &Ekos::Guide::setStatus);<br class=""></div><div class=""><br class=""></div><div class=""><b class=""><a href="http://guide.cpp:1935/" class="">guide.cpp:1935</a></b></div><div class="">void Guide::setStatus(Ekos::GuideState newState)<br class="">{<!-- --><br class=""> if (newState == state)<br class=""> {<!-- --><br class=""> // pass through the aborted state<br class=""> if (newState == GUIDE_ABORTED)<br class=""> emit newStatus(state);<br class=""> return;<br class=""> }<br class=""><br class=""> GuideState previousState = state;<br class=""><br class=""><span style="background-color:rgb( 255 , 255 , 0 )" class=""> state = newState;<br class=""> emit newStatus(state);</span><br class=""><br class=""> switch (state)<br class=""> {<!-- --><br class=""> ... </div><div class=""><br class=""> case GUIDE_IDLE:<br class=""> case GUIDE_CALIBRATION_ERROR:<br class=""> setBusy(false);<br class=""> manualDitherB->setEnabled(false);<br class=""> break;<br class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 7, 2020 at 12:31 PM Eric Dejouhanet <<a href="mailto:eric.dejouhanet@gmail.com" class="">eric.dejouhanet@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex"><div style="background-color:rgb( 255 , 255 , 255 )" class=""><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353response_container_BBPPID" style="outline:none" dir="auto" class=""> <div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353BB10_response_div_BBPPID" dir="auto" style="width:100%" class=""> Could it be that the emission of the guide failure is done before Guide's state is changed? That was the case for Focus, and Scheduler's immediate reaction for a new autofocus was thus rejected.</div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353BB10_response_div_BBPPID" dir="auto" style="width:100%" class=""><br class=""></div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353BB10_response_div_BBPPID" dir="auto" style="width:100%" class="">Unfortunately I haven't had time to check the log yet. </div> <div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353blackberry_signature_BBPPID" dir="auto" class=""> <div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353_signaturePlaceholder_BBPPID" dir="auto" class=""><p dir="ltr" class=""><a href="mailto:eric.dejouhanet@gmail.com" class="">eric.dejouhanet@gmail.com</a> - <a href="https://astronomy.dejouha.net/" class="">https://astronomy.dejouha.net</a></p></div> </div></div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353_original_msg_header_BBPPID" dir="auto" class=""> <table width="100%" style="border-spacing:0px;display:table;outline:none" class=""><tbody class=""><tr class=""><td colspan="2" class=""> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb( 181 , 196 , 223 );padding:3pt 0in 0in;font-family:'tahoma' , 'bb alpha sans' , 'slate pro';font-size:10pt" class=""> <div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353from" class=""><b class="">De:</b> <a href="mailto:murveit@gmail.com" class="">murveit@gmail.com</a></div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353sent" class=""><b class="">Envoyé:</b> 7 décembre 2020 21:04</div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353to" class=""><b class="">À:</b> <a href="mailto:mutlaqja@ikarustech.com" class="">mutlaqja@ikarustech.com</a></div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353reply_to" class=""><b class="">Répondre à:</b> <a href="mailto:hy@murveit.com" class="">hy@murveit.com</a></div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353cc" class=""><b class="">Cc:</b> <a href="mailto:hy@murveit.com" class="">hy@murveit.com</a>; <a href="mailto:sterne-jaeger@t-online.de" class="">sterne-jaeger@t-online.de</a>; <a href="mailto:eric.dejouhanet@gmail.com" class="">eric.dejouhanet@gmail.com</a></div><div id="gmail-m_1906432316115635784gmail-m_-3979288803047069353subject" class=""><b class="">Objet:</b> Re: scheduler issue</div></div></td></tr></tbody></table> <br class=""> </div><div dir="auto" style="outline:none" class=""><div dir="ltr" class="">It was indeed after a meridian flip.<div class="">Jo also sent the .analyze file to me (attached) and here's a zoom'ed in screen shot from the time of the issue.</div><div class=""><br class=""></div><div class=""><img alt="Screen Shot 2020-12-07 at 12.02.19 PM.png" width="562" height="295" class=""><br class=""></div><div class=""><br class=""></div><div class="">Hy</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 7, 2020 at 11:43 AM Jasem Mutlaq <<a href="mailto:mutlaqja@ikarustech.com" class="">mutlaqja@ikarustech.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex"><div dir="ltr" class=""><div class="">Hello Hy,</div><div class=""><br class=""></div><div class="">Do you know why it was calibrating? this wasn't after a meridian flip correct? What's happening is that scheduler handles calibration failures IF it was in the steps..i.e.</div><div class=""><br class=""></div><div class="">Track --> Focus --> Align --> Guide --> Capture. If at "Guide" calibration fails then it handles that. Right now, after capturing, the scheduler just LOGS the guide calibration results but does not handle them. Capture module should have been aborted if calibration fails, and then that would have been handled by the scheduler... but again, what would cause calibration in the middle of capture? meridian flip?</div><div class=""><br class=""></div><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">--</div><div class="">Best Regards,<br class="">Jasem Mutlaq<br class=""></div><div class=""><br class=""></div></div></div></div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Dec 6, 2020 at 11:30 PM Hy Murveit <<a href="mailto:murveit@gmail.com" class="">murveit@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex"><div dir="ltr" class="">Eric, Jasem,<br class=""><div class=""><br class=""></div><div class="">Reporting a possible scheduler bug.</div><div class=""><br class=""></div><div class="">Jo (@ElCorazon) sent me a log</div><div class=""><a href="https://www.dropbox.com/s/n8icvn90fhunjfl/log_20-53-07.txt.gz?dl=0" style="font-family:'verdana'" class="">https://www.dropbox.com/s/n8icvn90fhunjfl/log_20-53-07.txt.gz?dl=0</a><br class=""></div><div class="">which I analyzed and my conclusion is that star detection caused guider calibration to fail at 01:15:54 (see below)</div><div class=""><br class=""></div><div class="">[2020-12-05T01:15:54.614 CST INFO ][ <a href="http://org.kde.kstars.ekos.guide/" class="">org.kde.kstars.ekos.guide</a>] - "Lost track of the guide star. Try increasing the square size or reducing pulse duration."<br class="">[2020-12-05T01:15:54.617 CST DEBG ][ <a href="http://org.kde.kstars.ekos.capture/" class="">org.kde.kstars.ekos.capture</a>] - Guiding state changed from "Calibrating" to "Calibration error"<br class=""></div><div class=""><br class=""></div><div class="">and (ignoring that issue) the scheduler recognized, I suppose, that guider failed</div><div class=""><br class=""></div><div class="">[2020-12-05T01:15:54.624 CST DEBG ][ <a href="http://org.kde.kstars.ekos.scheduler/" class="">org.kde.kstars.ekos.scheduler</a>] - Guide State "Calibration error"<br class=""></div><div class=""><br class=""></div><div class="">but the scheduler didn't seem to restart the guiding calibration. Basically nothing happens until 1:57:56 when I assume Jo restarted things.</div><div class=""><br class=""></div><div class="">[2020-12-05T01:57:56.852 CST INFO ][ <a href="http://org.kde.kstars.ekos.scheduler/" class="">org.kde.kstars.ekos.scheduler</a>] - Scheduler is stopping...<br class=""></div><div class=""><br class=""></div><div class="">I assume the scheduler should try and restart the guider, but there are no .guide nor .scheduler messages between 1:15:54 and 1:57:56</div><div class=""><br class=""></div><div class="">Hy</div></div>
</blockquote></div>
</blockquote></div>
</div></div></blockquote></div>
</div></div></blockquote></div>
</div></blockquote></div><br class=""></div><!--end of _originalContent --></div></body></html>