KStars v3.5.0 Release Date?

Robert Lancaster rlancaste at gmail.com
Sun Nov 15 03:34:19 GMT 2020 

Hi Wolfgang, I was just working on something in stellarsolver that should improve this.  I was working to make sure the temporary files, the solved files, and the cancel files are all managed better, deleted when they are done, and that the solvers use them and don’t get killed prematurely.  I think I saw an issue similar to what you were reporting in my testing this evening and I think I fixed it.  Please see if it fixes it.

> On Nov 14, 2020, at 3:01 PM, Wolfgang Reissenberger <sterne-jaeger at openfuture.de> wrote:
> 
> Robert, all,
> I had the issue again when trying to solve a wide field image around NGC6888, which contains very dense star fields. I am using the 1-Default profile without any change.
> 
> If I leave the „Parallel Algorithm“ option from the Astrometry Parameters on „Auto“, Kstars solves the image very fast, but remains on 100%. It seems that the in parallel running threads were hanging.
> 
> I am using the following versions:
> KStars: 57c44d05c3e1f9895d84c7f4f73950975e8eddb7
> StellarSolver: 2d7eba6685c1bcd77c0525e88b3d24b2fcd474a9
> 
> Anything I could test right now?
> 
> Wolfgang
> 
>> Am 10.11.2020 um 15:50 schrieb Robert Lancaster <rlancaste at gmail.com <mailto:rlancaste at gmail.com>>:
>> 
>> Hi Wolfgang,
>> 
>> So I just want to clarify something you said here, there are a couple of parallel things and that can be a little confusing, so I just want to make sure we are talking about the same things.  The cause of the confusion is the terminology that astrometry.net <http://astrometry.net/> uses
>> 
>> 1. Load all Indexes in Memory / Load all indexes in Parallel.  This is the inParallel option for astrometry.net <http://astrometry.net/>.   In the options I tried to call this “Load all Indexes in Memory” to attempt to avoid the confusion with the Parallel Algorithm.  This has nothing to do with parallelization in different threads or processors.  It has to do with memory management.  The astrometry.net <http://astrometry.net/> solver can load the indexes and search them one after the other, or it can try to load all the indexes at once and then solve.  The second option is much much faster, but comes with risk.  astrometry.net <http://astrometry.net/> does NOT check to see if it has enough RAM before it tries to solve,  They have big warnings in the documentation about using this option.  If you don’t have enough RAM, it could use all the RAM and crash.
>> 
>> I programmed StellarSolver to check the available RAM prior to starting the solve.  If there is not enough RAM, it is supposed to turn off the option.  The user can also disable the option entirely, so that there is never a problem.  But you really do want the option turned on if your system can handle it.  We had some issues earlier about the RAM calculation.  I think the “inParallel” option causes the greatest crash risk.  I would really like it if somebody could look over the code for determining enough RAM and see if it is good now.  One thought that I have is that we can make the calculation more conservative and we could change the option to have 3 choices, Auto, on, or off.  So that if a user is really brave, or convinced they have enough RAM for sure, they could turn the option on regardless of the risk, If they are risk averse, they could turn it off, but most users could just leave it on auto.  What do you think?
>> 
>> 2. Parallelization Algorithm for solving.   I am assuming this second option is what you meant in your email.  This one is entirely of my creation and is what makes StellarSolver stellar.  Modern computers really have great capacity for computing in parallel and it causes a HUGE performance boost to use this capability, even on a Pi, since the PI has 4 processors. 
>> 
>> I programmed StellarSolver to have 2 different parallel algorithms, one that solves simultaneously at multiple “depths” and one that solves simultaneously at different scales.  If you set it to Auto, it will select the appropriate one based on whether you specified the scale or position (or neither).  If the image has both scale AND position, it does NOT solve in parallel and goes back to solving with a single thread.
>> 
>> When Jasem wanted to me to de-thread the StellarSolver and make it so that just the solvers are threads, I had to make a bunch of changes and one change I forgot was to make the star extraction before parallel solving asynchronous.  That does mean that when doing a parallel solve, it might look like things have frozen for a moment during the star extraction before the threads start up.  I have already fixed this, but it is in the releaseExperiment branch of StellarSolver, not in Master.  I would like to get this fix integrated before we release, but I will need to test this thoroughly first as I mentioned in a previous email.  I am wondering if this freezing behavior was what caused the “crash” you observed?
>> 
>> Thanks,
>> 
>> Rob
>> 
>> 
>>> On Nov 10, 2020, at 8:03 AM, Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>> 
>>> OK, I did a quick check on my RPi4 with Parallel Algorithm set to „Auto“ - and it works super fast! But since it is daytime, I can only test the „Load and Slew“ option. So maybe the WCS info in the file gave hints that are not present for normal capture and slew or sync.
>>> 
>>> I need to check it under real conditions, which might be tricky due to the fog hanging around here…
>>> 
>>> Wolfgang
>>>> Am 10.11.2020 um 11:16 schrieb Jasem Mutlaq <mutlaqja at ikarustech.com <mailto:mutlaqja at ikarustech.com>>:
>>>> 
>>>> Alright, let's look at this:
>>>> 
>>>> 1. Parallel algorithm: This is related to SOLVER, not image partitioning. It should work fine on Rpi4 and the checks are more reliable now as Robert worked on that.
>>>> 2. WCS Polar Align: Can this be reproduced with simulators?
>>>> 
>>>> --
>>>> Best Regards,
>>>> Jasem Mutlaq
>>>> 
>>>> 
>>>> 
>>>> On Tue, Nov 10, 2020 at 10:48 AM Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>>> It wasn’t that bad. The problem was that KStars went to 100% CPU usage and died (or I killed it, do not exactly remember). I’ll try to reproduce it...
>>>> 
>>>>> Am 10.11.2020 um 08:45 schrieb Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>>:
>>>>> 
>>>>> OK, well I believe it was fixed a week ago, so if you can still recreate it, you should report it. 
>>>>> It should be fixed before release if it is still freezing the Pi.
>>>>> 
>>>>> Hy
>>>>> 
>>>>> On Mon, Nov 9, 2020 at 11:42 PM Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>>>> OK, I have to check it. The problem occurred only a few days ago and I think I’m always on bleeding edge...
>>>>> 
>>>>>> Am 10.11.2020 um 08:38 schrieb Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>>:
>>>>>> 
>>>>>> Wolfgang: I believe Rob and/or Jasem fixed the issue with parallel algorithm bringing down the RPi4 a while back.
>>>>>> I have the solver on auto parallelism and load all indexes in memory, and it seems to work fine (and in parallel).
>>>>>> Similarly, for star extraction, Jasem implemented a threaded extraction that also automatically determines how many threads to use and seems fine on the RPi4.
>>>>>> 
>>>>>> Eric: I believe these parallel options are the defaults. Hopefully users won't need to configure things like this.
>>>>>> For star detection, I don't believe you can turn it off.
>>>>>> For star detection Jasem split the frame before detection (into at most num-threads parts--4 for the RPi4).
>>>>>> For align, I'm not sure how Rob divided things.
>>>>>> 
>>>>>> Hy
>>>>>> 
>>>>>> On Mon, Nov 9, 2020 at 11:07 PM Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>>>>> Hi all,
>>>>>> I think we are close to finishing the release. I personally would opt to wait for another week and keep an eye stability.
>>>>>> 
>>>>>> Maybe we should take another look if the default settings in the StellarSolver profiles work a) for typical camera/scope combinations and b) for all platforms.
>>>>>> 
>>>>>> For example with my RPi, I needed to change the Parallel Algorithm to „None“ because parallelity brought KStars down. Is the default setting „None“ and I changed it somewhen? With all the new parameters I would prefer having a robust setup and leave it to the user to optimize speed.
>>>>>> 
>>>>>> @Jasem: please take a closer look to MR!122, since it fixed 4(!) regressions I introduced with my capture counting fix MR!114. Hopefully now we have at least a proper coverage with automated tests...
>>>>>> 
>>>>>> Wolfgang
>>>>>> 
>>>>>>> Am 09.11.2020 um 22:04 schrieb Jasem Mutlaq <mutlaqja at ikarustech.com <mailto:mutlaqja at ikarustech.com>>:
>>>>>>> 
>>>>>>> Hello Folks,
>>>>>>> 
>>>>>>> So back to this topic, any major blockers to the KStars 3.5.0 release now?
>>>>>>> 
>>>>>>> 1. Remote Solver should be fixed now.
>>>>>>> 2. StellarSolver Profiles are more optimized now.
>>>>>>> 3. Handbook not updated yet, but we can probably work on this shortly.
>>>>>>> 4. Couple of pending MRs to take care of.
>>>>>>> 
>>>>>>> How about Friday the 13th?
>>>>>>> 
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Jasem Mutlaq
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Nov 5, 2020 at 3:41 AM Robert Lancaster <rlancaste at gmail.com <mailto:rlancaste at gmail.com>> wrote:
>>>>>>> Hi Eric,
>>>>>>> 
>>>>>>> Ok so then we would be changing the way we do version numbering with this, right?
>>>>>>> I believe now we typically add features in each new iteration 3.4.1, 3.4.2, etc etc
>>>>>>> and when it is really big like StellarSolver, then we make it a big release like 3.5.0
>>>>>>> 
>>>>>>> With this new paradigm, we wouldn’t put new features into the master of the main 3.5 branch
>>>>>>> But instead we would work on a new 3.6 branch, and then bug fixes would go into the 3.5 branch
>>>>>>> to make each new minor release, like 3.5.1, 3.5.2 etc.
>>>>>>> 
>>>>>>> Do I have this correct?
>>>>>>> 
>>>>>>> If this is right, then it would be longer before users see new features in the main branch, but the 
>>>>>>> tradeoff is that the main branch would have a LOT more stability.  I see this as a big positive.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> > On Nov 4, 2020, at 5:54 PM, Eric Dejouhanet <eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>> wrote:
>>>>>>> > 
>>>>>>> > Hello Hy,
>>>>>>> > 
>>>>>>> > Version 3.5.0 is only the beginning of the 3.5.x series, with more
>>>>>>> > bugfixes on each iteration (and possibly, only bugfixes).
>>>>>>> > So I have no problem leaving unresolved issues in 3.5.0.
>>>>>>> > 
>>>>>>> > For instance, the Focus module now has a slight and unforeseeable
>>>>>>> > delay after the capture completes.
>>>>>>> > The UI reflects the end of the capture only, not the end of the detection.
>>>>>>> > This makes the UI Focus test quite difficult to tweak, as running an
>>>>>>> > average of the HFR over multiple frames now has an unknown duration.
>>>>>>> > Right now, the test is trying to click the capture button too soon 2
>>>>>>> > out of 10 attempts.
>>>>>>> > But this won't block 3.5 in my opinion (and now that I understood the
>>>>>>> > problem, I won't work on it immediately).
>>>>>>> > 
>>>>>>> > In terms of reporting problems, the official way is stil bugs.kde.org <http://bugs.kde.org/>,
>>>>>>> > but there's quite a cleanup/followup to do there.
>>>>>>> > I'd say we can use issues in invent.kde.org <http://invent.kde.org/> to discuss planned
>>>>>>> > development around a forum/bugzilla issue or invent proposal (like
>>>>>>> > agile stories).
>>>>>>> > There are milestones associated with several issues (although I think
>>>>>>> > they should be reviewed and postponed).
>>>>>>> > And we can certainly write a punchlist: check the board at
>>>>>>> > https://invent.kde.org/education/kstars/-/milestones/3 <https://invent.kde.org/education/kstars/-/milestones/3>
>>>>>>> > 
>>>>>>> > Le mer. 4 nov. 2020 à 22:38, Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>> a écrit :
>>>>>>> >> 
>>>>>>> >> Eric,
>>>>>>> >> 
>>>>>>> >> I would add to your list:
>>>>>>> >> 
>>>>>>> >> - KStars Handbook (review update sections to reflect 3.5.0) and finally (perhaps manually if necessary) put the latest handbook online.
>>>>>>> >> 
>>>>>>> >> - Review the extraction settings. I spent a bit of time looking at the default HFR settings, and based on some experimentation (truth be told, with a limited amount of data) adjust things a little differently than my first guess (which was basically focus' settings).
>>>>>>> >> Rob: My intuition is that I should adjust the default StellarSolver star-extraction settings for Focus and Guide as well in stellarsolverprofile.cpp. I don't know whether you've already verified them, and want to release them as they are, or whether they are a first shot and you'd welcome adjustment?
>>>>>>> >> 
>>>>>>> >> Also, Eric, I suppose I should be adding these things here: https://invent.kde.org/education/kstars/-/issues <https://invent.kde.org/education/kstars/-/issues>
>>>>>>> >> Is that right? Sorry about that--ok, after this thread ;) But seriously, your email is a good summary, and from that link
>>>>>>> >> it doesn't seem as easy to see which are "must do by 3.5.0" and which are "nice to have someday".
>>>>>>> >> A 3.5.0 punchlist would be a nice thing to have.
>>>>>>> >> 
>>>>>>> >> Hy
>>>>>>> >> 
>>>>>>> >> On Wed, Nov 4, 2020 at 12:58 PM Eric Dejouhanet <eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>> wrote:
>>>>>>> >>> 
>>>>>>> >>> Hello,
>>>>>>> >>> 
>>>>>>> >>> Where do we stand now in terms of bugfixing towards 3.5.0?
>>>>>>> >>> 
>>>>>>> >>> - StellarSolver has all features in, and 1.5 is finally out at Jasem's PPA.
>>>>>>> >>> - However Gitlab CI still complains about that lib package (see
>>>>>>> >>> https://invent.kde.org/education/kstars/-/jobs/75941 <https://invent.kde.org/education/kstars/-/jobs/75941>)
>>>>>>> >>> - Unitary tests are being fixed progressively, mount tests are down to
>>>>>>> >>> ~20 minutes (yeees!)
>>>>>>> >>> - From my tests, the remote Astrometry INDI driver is not usable
>>>>>>> >>> anymore from Ekos.
>>>>>>> >>> - The issue raised with flat frames is confirmed fixed (at least by me).
>>>>>>> >>> - Meridian flip is OK (but I had not enough time to test TWO flips in a row).
>>>>>>> >>> - Memory leaks are still being researched in Ekos.
>>>>>>> >>> - There is an issue when duplicating an entry in a scheduler job,
>>>>>>> >>> where the sequence associated is copied from the next job.
>>>>>>> >>> 
>>>>>>> >>> Could we get a 3.6 branch where we will merge development of new features?
>>>>>>> >>> And master for bugfixing 3.5.x until we merge 3.6 new features in?
>>>>>>> >>> (we'd still have to port bugfixes from master to 3.6)
>>>>>>> >>> I don't think the opposite, master for 3.6 and a separate living
>>>>>>> >>> 3.5.x, is doable in the current configuration (build, ppas, MRs...).
>>>>>>> >>> 
>>>>>>> >>> --
>>>>>>> >>> -- eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com> - https://astronomy.dejouha.net <https://astronomy.dejouha.net/>
>>>>>>> > 
>>>>>>> > 
>>>>>>> > 
>>>>>>> > -- 
>>>>>>> > -- eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com> - https://astronomy.dejouha.net <https://astronomy.dejouha.net/>
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20201114/8653373e/attachment-0001.htm>

More information about the Kstars-devel mailing list