KStars v3.5.0 Release Date?

Robert Lancaster rlancaste at gmail.com
Thu Nov 12 06:13:58 GMT 2020 

Hi Hy,

Yeah, we can definitely think about it, I was just throwing out some ideas.  There are multiple options.  I have one big concern though about automatically performing a calculation where we exclude stars, but I don’t have a problem with it if the calculation is part of a parameter that can be controlled by the user or the program.. The reason is this: I think there are several applications of the SEP algorithm. Right now, we are using it for finding stars for the solver, and we are also using it to find the HFR of the stars in the image.  Neither of those tasks really cares about whether we keep all the stars, and they both perform better when some stars are excluded, it just seems (as of right now) that they like different stars to be excluded and thus the two different parameters.  But my concern is that there is a very significant third application of this algorithm that we haven’t really tapped into yet.  I do start to tap into that feature in the StellarSolver Tester program, but it is not used at all yet in KStars (though it could be in the future).  

That feature is the ability to perform photometry on galaxies, variable stars, and other astronomical objects.  As soon as we get this perfected into KStars, I would like to create a small program based on StellarSolver that can automatically load a list of FITS files, plate solve them, do SEP, and automatically make light curves for all the stars in the field.  I already partially implemented this feature in the tester, you can currently click on the object in the image and it shows you all the photometry info about it in a table.  Likewise you can click the table and it identifies the star.  So my concern is that if we automatically exclude stars without a user setting to control it, then we will have serious problems in photometry because it will exclude some objects of interest.

So for your initial Keep parameter, I have no problem with keeping it as it is as long as we can expose it to the user in the interface, or if we make it one of the other star filtering parameters.  But I wouldn’t want to make it a purely internal parameter that can’t be turned off by the user.  For similar reasons we should probably make sure that no stars are excluded during the partitioning process, but we can look into fixing that later, or allowing the ability to turn off the SEP partitioning if they are getting left out.

Thanks,

Rob

> On Nov 11, 2020, at 9:04 PM, Hy Murveit <murveit at gmail.com> wrote:
> 
> Rob,
> 
> Thanks for looking into this. I think you should keep initialKeep as it is, code-wise. I'd also be OK with implementing it as some constant multiplier of keepSize (e.g. initialKeep = N * keepSize) where N is 3 or 5 and not exposed to the user. 
> 
> I also disagree about the name for initialKeep. It is just an internal parameter to speed things up by not running the SEP HFR routine on all the stars extracted by SEP. That is, imagine you're only interested in 100 stars. sep_extract() can find 10,000 stars in some busy images. Running the HFR calculations (the slowest part of the SEP code) on 10000 stars would be very slow and unnecessary if you're just going to throw away most of them and return 100. 
> 
> So, as I don't think anyone should adjust this parameter for "quality" reasons, just for speed reasons, I'm happy with the name, and think keepBright could be misleading. Or, as I say, no name at all and set it to say something like: min(300, 5 * keepSize). [ Really the larger keepSize is, the smaller the multiplier might be--e.g. if you were keeping 1000, perhaps 2*keepSize ].
> 
> Hy
> 
> 
> On Wed, Nov 11, 2020 at 4:21 PM Robert Lancaster <rlancaste at gmail.com <mailto:rlancaste at gmail.com>> wrote:
> Hi Hy,
> 
> Yes, I forgot that you added that parameter and then we didn’t add it to the interface and saving functions!  
> 
> So I just went about rectifying this a few minutes ago.  I just added the initial keep parameter to the tester and file saving functions and pushed the changes to StellarSolver.  I think that will make a difference particularly if it wasn’t being saved in the saved options profiles.  I can also add it to the Editor in KStars.  But maybe this explains why it was taking so long on Eric’s computer.  Maybe he saved it and the parameter wasn’t saved.
> 
> I am wondering though if maybe we want to reword these 2 parameters before we release.  Right now they are called keepNum and initialKeep.  The initialKeep is a filter based on the size of the star that is applied before HFR is done and the keepNum is a filter based on the magnitude that is applied after the star extraction is all finished.  The keepNum is most useful for Solving because we want the brightest stars for solving. The initialKeep is most useful for SEP since we tend to want the biggest stars for calculating HFR.  Also it is extremely important for HFR that the filtering happens before the HFR calculations are done  Maybe we should call the keepNum something like keepBright and the initialKeep something like keepSize?
> 
> On a similar note when I was adding the parameter into the tester just now and doing some testing, I found that Jasem’s changes to partition the image might make your initialKeep parameter not work like you intended it to work Hy.  Now the initial keep parameter is being applied to each partition instead of being applied to the whole image.  So if you set an initial Keep of 50 stars, but then partition the image into 50 chunks, while you might expect initialKeep to limit it to 50 stars, in fact you are limiting it to 50 x 50 stars.  Perhaps as a simple solution, we could just divide the initial keep parameter by the number of chunks?
> 
>> On Nov 10, 2020, at 3:45 PM, Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>> wrote:
>> 
>> Initial Keep is a key parameter wrt time. 
>> I see that it isn't exposed in the profile editor. 
>> Rob, how is that set when someone reduces the KEEP parameter?
>> 
>> On Tue, Nov 10, 2020 at 12:15 PM Eric Dejouhanet <eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>> wrote:
>> Robert, following your reply, I configured my SmallScaleSolving Focus configuration to return 5 stars only. I expected the HFR to be computed faster, as there would be less stars in the list (well, compared to...29?). But the duration remained the same, more than two minutes. I also observe that the cpu is at 70-80% usage on the two cores during that time. 
>> 
>> What I can do is add the same log output to Focus as is done in Align and see the difference. 
>> eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com> - https://astronomy.dejouha.net <https://astronomy.dejouha.net/>
>> De: rlancaste at gmail.com <mailto:rlancaste at gmail.com>
>> Envoyé: 10 novembre 2020 15:13
>> À: eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>
>> Cc: sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>; mutlaqja at ikarustech.com <mailto:mutlaqja at ikarustech.com>; hy at murveit.com <mailto:hy at murveit.com>; kstars-devel at kde.org <mailto:kstars-devel at kde.org>
>> Objet: Re: KStars v3.5.0 Release Date?
>> 
>> One simple reason, HFR calculations.  They are very computation intensive.  If you want the HFR, it takes much longer
>> 
>>> On Nov 10, 2020, at 9:04 AM, Eric Dejouhanet <eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>> wrote:
>>> 
>>> On my system, as an end-user, I don't understand why the star detection takes less than a second in the alignment module (SmallScaleSolving), and more than twenty in the Focus module (tweaked version of SmallSizedStars).
>>> 
>>> So I did configure Focus to use the SmallScaleSolving settings from the Align module. I used the CCD Simulator to send one of my NGC6888 frames to Focus, and ran a capture. The Focus module took FIVE minutes to detect 29 stars. I thought the whole thing was crashed. 
>>> 
>>> The Align module took FOUR seconds with the same configuration on the same 31MB frame to detect 50+ stars. It said it wasn't parallel by itself. It did a downsample by 3.
>>> 
>>> So this is probably not a question of detection, there is another culprit somewhere. During the execution of the Focus detection, the cpu wasn't 100%. 
>>> eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com> - https://astronomy.dejouha.net <https://astronomy.dejouha.net/>
>>> De: sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>
>>> Envoyé: 10 novembre 2020 14:03
>>> À: mutlaqja at ikarustech.com <mailto:mutlaqja at ikarustech.com>
>>> Cc: hy at murveit.com <mailto:hy at murveit.com>; rlancaste at gmail.com <mailto:rlancaste at gmail.com>; kstars-devel at kde.org <mailto:kstars-devel at kde.org>; eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>
>>> Objet: Re: KStars v3.5.0 Release Date?
>>> 
>>> OK, I did a quick check on my RPi4 with Parallel Algorithm set to „Auto“ - and it works super fast! But since it is daytime, I can only test the „Load and Slew“ option. So maybe the WCS info in the file gave hints that are not present for normal capture and slew or sync.
>>> 
>>> I need to check it under real conditions, which might be tricky due to the fog hanging around here…
>>> 
>>> Wolfgang
>>>> Am 10.11.2020 um 11:16 schrieb Jasem Mutlaq <mutlaqja at ikarustech.com <mailto:mutlaqja at ikarustech.com>>:
>>>> 
>>>> Alright, let's look at this:
>>>> 
>>>> 1. Parallel algorithm: This is related to SOLVER, not image partitioning. It should work fine on Rpi4 and the checks are more reliable now as Robert worked on that.
>>>> 2. WCS Polar Align: Can this be reproduced with simulators?
>>>> 
>>>> --
>>>> Best Regards,
>>>> Jasem Mutlaq
>>>> 
>>>> 
>>>> 
>>>> On Tue, Nov 10, 2020 at 10:48 AM Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>>> It wasn’t that bad. The problem was that KStars went to 100% CPU usage and died (or I killed it, do not exactly remember). I’ll try to reproduce it...
>>>> 
>>>>> Am 10.11.2020 um 08:45 schrieb Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>>:
>>>>> 
>>>>> OK, well I believe it was fixed a week ago, so if you can still recreate it, you should report it. 
>>>>> It should be fixed before release if it is still freezing the Pi.
>>>>> 
>>>>> Hy
>>>>> 
>>>>> On Mon, Nov 9, 2020 at 11:42 PM Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>>>> OK, I have to check it. The problem occurred only a few days ago and I think I’m always on bleeding edge...
>>>>> 
>>>>>> Am 10.11.2020 um 08:38 schrieb Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>>:
>>>>>> 
>>>>>> Wolfgang: I believe Rob and/or Jasem fixed the issue with parallel algorithm bringing down the RPi4 a while back.
>>>>>> I have the solver on auto parallelism and load all indexes in memory, and it seems to work fine (and in parallel).
>>>>>> Similarly, for star extraction, Jasem implemented a threaded extraction that also automatically determines how many threads to use and seems fine on the RPi4.
>>>>>> 
>>>>>> Eric: I believe these parallel options are the defaults. Hopefully users won't need to configure things like this.
>>>>>> For star detection, I don't believe you can turn it off.
>>>>>> For star detection Jasem split the frame before detection (into at most num-threads parts--4 for the RPi4).
>>>>>> For align, I'm not sure how Rob divided things.
>>>>>> 
>>>>>> Hy
>>>>>> 
>>>>>> On Mon, Nov 9, 2020 at 11:07 PM Wolfgang Reissenberger <sterne-jaeger at openfuture.de <mailto:sterne-jaeger at openfuture.de>> wrote:
>>>>>> Hi all,
>>>>>> I think we are close to finishing the release. I personally would opt to wait for another week and keep an eye stability.
>>>>>> 
>>>>>> Maybe we should take another look if the default settings in the StellarSolver profiles work a) for typical camera/scope combinations and b) for all platforms.
>>>>>> 
>>>>>> For example with my RPi, I needed to change the Parallel Algorithm to „None“ because parallelity brought KStars down. Is the default setting „None“ and I changed it somewhen? With all the new parameters I would prefer having a robust setup and leave it to the user to optimize speed.
>>>>>> 
>>>>>> @Jasem: please take a closer look to MR!122, since it fixed 4(!) regressions I introduced with my capture counting fix MR!114. Hopefully now we have at least a proper coverage with automated tests...
>>>>>> 
>>>>>> Wolfgang
>>>>>> 
>>>>>>> Am 09.11.2020 <tel:09112020> um 22:04 schrieb Jasem Mutlaq <mutlaqja at ikarustech.com <mailto:mutlaqja at ikarustech.com>>:
>>>>>>> 
>>>>>>> Hello Folks,
>>>>>>> 
>>>>>>> So back to this topic, any major blockers to the KStars 3.5.0 release now?
>>>>>>> 
>>>>>>> 1. Remote Solver should be fixed now.
>>>>>>> 2. StellarSolver Profiles are more optimized now.
>>>>>>> 3. Handbook not updated yet, but we can probably work on this shortly.
>>>>>>> 4. Couple of pending MRs to take care of.
>>>>>>> 
>>>>>>> How about Friday the 13th?
>>>>>>> 
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Jasem Mutlaq
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Nov 5, 2020 at 3:41 AM Robert Lancaster <rlancaste at gmail.com <mailto:rlancaste at gmail.com>> wrote:
>>>>>>> Hi Eric,
>>>>>>> 
>>>>>>> Ok so then we would be changing the way we do version numbering with this, right?
>>>>>>> I believe now we typically add features in each new iteration 3.4.1, 3.4.2, etc etc
>>>>>>> and when it is really big like StellarSolver, then we make it a big release like 3.5.0
>>>>>>> 
>>>>>>> With this new paradigm, we wouldn’t put new features into the master of the main 3.5 branch
>>>>>>> But instead we would work on a new 3.6 branch, and then bug fixes would go into the 3.5 branch
>>>>>>> to make each new minor release, like 3.5.1, 3.5.2 etc.
>>>>>>> 
>>>>>>> Do I have this correct?
>>>>>>> 
>>>>>>> If this is right, then it would be longer before users see new features in the main branch, but the 
>>>>>>> tradeoff is that the main branch would have a LOT more stability.  I see this as a big positive.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Rob
>>>>>>> 
>>>>>>> > On Nov 4, 2020, at 5:54 PM, Eric Dejouhanet <eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>> wrote:
>>>>>>> > 
>>>>>>> > Hello Hy,
>>>>>>> > 
>>>>>>> > Version 3.5.0 is only the beginning of the 3.5.x series, with more
>>>>>>> > bugfixes on each iteration (and possibly, only bugfixes).
>>>>>>> > So I have no problem leaving unresolved issues in 3.5.0.
>>>>>>> > 
>>>>>>> > For instance, the Focus module now has a slight and unforeseeable
>>>>>>> > delay after the capture completes.
>>>>>>> > The UI reflects the end of the capture only, not the end of the detection.
>>>>>>> > This makes the UI Focus test quite difficult to tweak, as running an
>>>>>>> > average of the HFR over multiple frames now has an unknown duration.
>>>>>>> > Right now, the test is trying to click the capture button too soon 2
>>>>>>> > out of 10 attempts.
>>>>>>> > But this won't block 3.5 in my opinion (and now that I understood the
>>>>>>> > problem, I won't work on it immediately).
>>>>>>> > 
>>>>>>> > In terms of reporting problems, the official way is stil bugs.kde.org <http://bugs.kde.org/>,
>>>>>>> > but there's quite a cleanup/followup to do there.
>>>>>>> > I'd say we can use issues in invent.kde.org <http://invent.kde.org/> to discuss planned
>>>>>>> > development around a forum/bugzilla issue or invent proposal (like
>>>>>>> > agile stories).
>>>>>>> > There are milestones associated with several issues (although I think
>>>>>>> > they should be reviewed and postponed).
>>>>>>> > And we can certainly write a punchlist: check the board at
>>>>>>> > https://invent.kde.org/education/kstars/-/milestones/3 <https://invent.kde.org/education/kstars/-/milestones/3>
>>>>>>> > 
>>>>>>> > Le mer. 4 nov. 2020 à 22:38, Hy Murveit <murveit at gmail.com <mailto:murveit at gmail.com>> a écrit :
>>>>>>> >> 
>>>>>>> >> Eric,
>>>>>>> >> 
>>>>>>> >> I would add to your list:
>>>>>>> >> 
>>>>>>> >> - KStars Handbook (review update sections to reflect 3.5.0) and finally (perhaps manually if necessary) put the latest handbook online.
>>>>>>> >> 
>>>>>>> >> - Review the extraction settings. I spent a bit of time looking at the default HFR settings, and based on some experimentation (truth be told, with a limited amount of data) adjust things a little differently than my first guess (which was basically focus' settings).
>>>>>>> >> Rob: My intuition is that I should adjust the default StellarSolver star-extraction settings for Focus and Guide as well in stellarsolverprofile.cpp <http://stellarsolverprofile.cpp/>. I don't know whether you've already verified them, and want to release them as they are, or whether they are a first shot and you'd welcome adjustment?
>>>>>>> >> 
>>>>>>> >> Also, Eric, I suppose I should be adding these things here: https://invent.kde.org/education/kstars/-/issues <https://invent.kde.org/education/kstars/-/issues>
>>>>>>> >> Is that right? Sorry about that--ok, after this thread ;) But seriously, your email is a good summary, and from that link
>>>>>>> >> it doesn't seem as easy to see which are "must do by 3.5.0" and which are "nice to have someday".
>>>>>>> >> A 3.5.0 punchlist would be a nice thing to have.
>>>>>>> >> 
>>>>>>> >> Hy
>>>>>>> >> 
>>>>>>> >> On Wed, Nov 4, 2020 at 12:58 PM Eric Dejouhanet <eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com>> wrote:
>>>>>>> >>> 
>>>>>>> >>> Hello,
>>>>>>> >>> 
>>>>>>> >>> Where do we stand now in terms of bugfixing towards 3.5.0?
>>>>>>> >>> 
>>>>>>> >>> - StellarSolver has all features in, and 1.5 is finally out at Jasem's PPA.
>>>>>>> >>> - However Gitlab CI still complains about that lib package (see
>>>>>>> >>> https://invent.kde.org/education/kstars/-/jobs/75941 <https://invent.kde.org/education/kstars/-/jobs/75941>)
>>>>>>> >>> - Unitary tests are being fixed progressively, mount tests are down to
>>>>>>> >>> ~20 minutes (yeees!)
>>>>>>> >>> - From my tests, the remote Astrometry INDI driver is not usable
>>>>>>> >>> anymore from Ekos.
>>>>>>> >>> - The issue raised with flat frames is confirmed fixed (at least by me).
>>>>>>> >>> - Meridian flip is OK (but I had not enough time to test TWO flips in a row).
>>>>>>> >>> - Memory leaks are still being researched in Ekos.
>>>>>>> >>> - There is an issue when duplicating an entry in a scheduler job,
>>>>>>> >>> where the sequence associated is copied from the next job.
>>>>>>> >>> 
>>>>>>> >>> Could we get a 3.6 branch where we will merge development of new features?
>>>>>>> >>> And master for bugfixing 3.5.x until we merge 3.6 new features in?
>>>>>>> >>> (we'd still have to port bugfixes from master to 3.6)
>>>>>>> >>> I don't think the opposite, master for 3.6 and a separate living
>>>>>>> >>> 3.5.x, is doable in the current configuration (build, ppas, MRs...).
>>>>>>> >>> 
>>>>>>> >>> --
>>>>>>> >>> -- eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com> - https://astronomy.dejouha.net <https://astronomy.dejouha.net/>
>>>>>>> > 
>>>>>>> > 
>>>>>>> > 
>>>>>>> > -- 
>>>>>>> > -- eric.dejouhanet at gmail.com <mailto:eric.dejouhanet at gmail.com> - https://astronomy.dejouha.net <https://astronomy.dejouha.net/>
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20201112/86bd3cd9/attachment-0001.htm>

More information about the Kstars-devel mailing list