KStars v3.5.0 Release Date?

[Hy Murveit]

Rob, I agree that the user should control whether we're losing stars,
though for KStars we should give him/her reasonable defaults.
However, InitialKeep and Keep are really the same kind of thing. If you are
setting Keep to 100, then I think it's OK to automatically compute
initialKeep from Keep. If the user has stated that he/she only wants to
keep 100 stars, then why does he care what the program does internally with
initialKeep? In any event, I'm also OK exposing it if you like.

Hy

On Wed, Nov 11, 2020 at 10:14 PM Robert Lancaster <rlancaste at gmail.com>
wrote:

> Hi Hy,
>
> Yeah, we can definitely think about it, I was just throwing out some
> ideas.  There are multiple options.  I have one big concern though about
> automatically performing a calculation where we exclude stars, but I don’t
> have a problem with it if the calculation is part of a parameter that can
> be controlled by the user or the program.. The reason is this: I think
> there are several applications of the SEP algorithm. Right now, we are
> using it for finding stars for the solver, and we are also using it to find
> the HFR of the stars in the image.  Neither of those tasks really cares
> about whether we keep all the stars, and they both perform better when some
> stars are excluded, it just seems (as of right now) that they like
> different stars to be excluded and thus the two different parameters.  But
> my concern is that there is a very significant third application of this
> algorithm that we haven’t really tapped into yet.  I do start to tap into
> that feature in the StellarSolver Tester program, but it is not used at all
> yet in KStars (though it could be in the future).
>
> That feature is the ability to perform photometry on galaxies, variable
> stars, and other astronomical objects.  As soon as we get this perfected
> into KStars, I would like to create a small program based on StellarSolver
> that can automatically load a list of FITS files, plate solve them, do SEP,
> and automatically make light curves for all the stars in the field.  I
> already partially implemented this feature in the tester, you can currently
> click on the object in the image and it shows you all the photometry info
> about it in a table.  Likewise you can click the table and it identifies
> the star.  So my concern is that if we automatically exclude stars without
> a user setting to control it, then we will have serious problems in
> photometry because it will exclude some objects of interest.
>
> So for your initial Keep parameter, I have no problem with keeping it as
> it is as long as we can expose it to the user in the interface, or if we
> make it one of the other star filtering parameters.  But I wouldn’t want to
> make it a purely internal parameter that can’t be turned off by the user.
> For similar reasons we should probably make sure that no stars are excluded
> during the partitioning process, but we can look into fixing that later, or
> allowing the ability to turn off the SEP partitioning if they are getting
> left out.
>
> Thanks,
>
> Rob
>
> On Nov 11, 2020, at 9:04 PM, Hy Murveit <murveit at gmail.com> wrote:
>
> Rob,
>
> Thanks for looking into this. I think you should keep initialKeep as it
> is, code-wise. I'd also be OK with implementing it as some constant
> multiplier of keepSize (e.g. initialKeep = N * keepSize) where N is 3 or 5
> and not exposed to the user.
>
> I also disagree about the name for initialKeep. It is just an internal
> parameter to speed things up by not running the SEP HFR routine on all the
> stars extracted by SEP. That is, imagine you're only interested in 100
> stars. sep_extract() can find 10,000 stars in some busy images. Running the
> HFR calculations (the slowest part of the SEP code) on 10000 stars would be
> very slow and unnecessary if you're just going to throw away most of them
> and return 100.
>
> So, as I don't think anyone should adjust this parameter for "quality"
> reasons, just for speed reasons, I'm happy with the name, and think
> keepBright could be misleading. Or, as I say, no name at all and set it to
> say something like: min(300, 5 * keepSize). [ Really the larger keepSize
> is, the smaller the multiplier might be--e.g. if you were keeping 1000,
> perhaps 2*keepSize ].
>
> Hy
>
>
> On Wed, Nov 11, 2020 at 4:21 PM Robert Lancaster <rlancaste at gmail.com>
> wrote:
>
>> Hi Hy,
>>
>> Yes, I forgot that you added that parameter and then we didn’t add it to
>> the interface and saving functions!
>>
>> So I just went about rectifying this a few minutes ago.  I just added the
>> initial keep parameter to the tester and file saving functions and pushed
>> the changes to StellarSolver.  I think that will make a difference
>> particularly if it wasn’t being saved in the saved options profiles.  I can
>> also add it to the Editor in KStars.  But maybe this explains why it was
>> taking so long on Eric’s computer.  Maybe he saved it and the parameter
>> wasn’t saved.
>>
>> I am wondering though if maybe we want to reword these 2 parameters
>> before we release.  Right now they are called keepNum and initialKeep.  The
>> initialKeep is a filter based on the size of the star that is applied
>> before HFR is done and the keepNum is a filter based on the magnitude that
>> is applied after the star extraction is all finished.  The keepNum is most
>> useful for Solving because we want the brightest stars for solving. The
>> initialKeep is most useful for SEP since we tend to want the biggest stars
>> for calculating HFR.  Also it is extremely important for HFR that the
>> filtering happens before the HFR calculations are done  Maybe we should
>> call the keepNum something like keepBright and the initialKeep something
>> like keepSize?
>>
>> On a similar note when I was adding the parameter into the tester just
>> now and doing some testing, I found that Jasem’s changes to partition the
>> image might make your initialKeep parameter not work like you intended it
>> to work Hy.  Now the initial keep parameter is being applied to each
>> partition instead of being applied to the whole image.  So if you set an
>> initial Keep of 50 stars, but then partition the image into 50 chunks,
>> while you might expect initialKeep to limit it to 50 stars, in fact you are
>> limiting it to 50 x 50 stars.  Perhaps as a simple solution, we could just
>> divide the initial keep parameter by the number of chunks?
>>
>> On Nov 10, 2020, at 3:45 PM, Hy Murveit <murveit at gmail.com> wrote:
>>
>> Initial Keep is a key parameter wrt time.
>> I see that it isn't exposed in the profile editor.
>> Rob, how is that set when someone reduces the KEEP parameter?
>>
>> On Tue, Nov 10, 2020 at 12:15 PM Eric Dejouhanet <
>> eric.dejouhanet at gmail.com> wrote:
>>
>>> Robert, following your reply, I configured my SmallScaleSolving Focus
>>> configuration to return 5 stars only. I expected the HFR to be computed
>>> faster, as there would be less stars in the list (well, compared to...29?).
>>> But the duration remained the same, more than two minutes. I also observe
>>> that the cpu is at 70-80% usage on the two cores during that time.
>>>
>>> What I can do is add the same log output to Focus as is done in Align
>>> and see the difference.
>>>
>>> eric.dejouhanet at gmail.com - https://astronomy.dejouha.net
>>> *De:* rlancaste at gmail.com
>>> *Envoyé:* 10 novembre 2020 15:13
>>> *À:* eric.dejouhanet at gmail.com
>>> *Cc:* sterne-jaeger at openfuture.de; mutlaqja at ikarustech.com;
>>> hy at murveit.com; kstars-devel at kde.org
>>> *Objet:* Re: KStars v3.5.0 Release Date?
>>>
>>> One simple reason, HFR calculations.  They are very computation
>>> intensive.  If you want the HFR, it takes much longer
>>>
>>> On Nov 10, 2020, at 9:04 AM, Eric Dejouhanet <eric.dejouhanet at gmail.com>
>>> wrote:
>>>
>>> On my system, as an end-user, I don't understand why the star detection
>>> takes less than a second in the alignment module (SmallScaleSolving), and
>>> more than twenty in the Focus module (tweaked version of SmallSizedStars).
>>>
>>> So I did configure Focus to use the SmallScaleSolving settings from the
>>> Align module. I used the CCD Simulator to send one of my NGC6888 frames to
>>> Focus, and ran a capture. The Focus module took FIVE minutes to detect 29
>>> stars. I thought the whole thing was crashed.
>>>
>>> The Align module took FOUR seconds with the same configuration on the
>>> same 31MB frame to detect 50+ stars. It said it wasn't parallel by itself.
>>> It did a downsample by 3.
>>>
>>> So this is probably not a question of detection, there is another
>>> culprit somewhere. During the execution of the Focus detection, the cpu
>>> wasn't 100%.
>>>
>>> eric.dejouhanet at gmail.com - https://astronomy.dejouha.net
>>> *De:* sterne-jaeger at openfuture.de
>>> *Envoyé:* 10 novembre 2020 14:03
>>> *À:* mutlaqja at ikarustech.com
>>> *Cc:* hy at murveit.com; rlancaste at gmail.com; kstars-devel at kde.org;
>>> eric.dejouhanet at gmail.com
>>> *Objet:* Re: KStars v3.5.0 Release Date?
>>>
>>> OK, I did a quick check on my RPi4 with Parallel Algorithm set to „Auto“
>>> - and it works super fast! But since it is daytime, I can only test the
>>> „Load and Slew“ option. So maybe the WCS info in the file gave hints that
>>> are not present for normal capture and slew or sync.
>>>
>>> I need to check it under real conditions, which might be tricky due to
>>> the fog hanging around here…
>>>
>>> Wolfgang
>>>
>>> Am 10.11.2020 um 11:16 schrieb Jasem Mutlaq <mutlaqja at ikarustech.com>:
>>>
>>> Alright, let's look at this:
>>>
>>> 1. Parallel algorithm: This is related to SOLVER, not image
>>> partitioning. It should work fine on Rpi4 and the checks are more reliable
>>> now as Robert worked on that.
>>> 2. WCS Polar Align: Can this be reproduced with simulators?
>>>
>>> --
>>> Best Regards,
>>> Jasem Mutlaq
>>>
>>>
>>>
>>> On Tue, Nov 10, 2020 at 10:48 AM Wolfgang Reissenberger <
>>> sterne-jaeger at openfuture.de> wrote:
>>>
>>>> It wasn’t that bad. The problem was that KStars went to 100% CPU usage
>>>> and died (or I killed it, do not exactly remember). I’ll try to reproduce
>>>> it...
>>>>
>>>> Am 10.11.2020 um 08:45 schrieb Hy Murveit <murveit at gmail.com>:
>>>>
>>>> OK, well I believe it was fixed a week ago, so if you can still
>>>> recreate it, you should report it.
>>>> It should be fixed before release if it is still freezing the Pi.
>>>>
>>>> Hy
>>>>
>>>> On Mon, Nov 9, 2020 at 11:42 PM Wolfgang Reissenberger <
>>>> sterne-jaeger at openfuture.de> wrote:
>>>>
>>>>> OK, I have to check it. The problem occurred only a few days ago and I
>>>>> think I’m always on bleeding edge...
>>>>>
>>>>> Am 10.11.2020 um 08:38 schrieb Hy Murveit <murveit at gmail.com>:
>>>>>
>>>>> Wolfgang: I believe Rob and/or Jasem fixed the issue with parallel
>>>>> algorithm bringing down the RPi4 a while back.
>>>>> I have the solver on auto parallelism and load all indexes in memory,
>>>>> and it seems to work fine (and in parallel).
>>>>> Similarly, for star extraction, Jasem implemented a threaded
>>>>> extraction that also automatically determines how many threads to use and
>>>>> seems fine on the RPi4.
>>>>>
>>>>> Eric: I believe these parallel options are the defaults. Hopefully
>>>>> users won't need to configure things like this.
>>>>> For star detection, I don't believe you can turn it off.
>>>>> For star detection Jasem split the frame before detection (into at
>>>>> most num-threads parts--4 for the RPi4).
>>>>> For align, I'm not sure how Rob divided things.
>>>>>
>>>>> Hy
>>>>>
>>>>> On Mon, Nov 9, 2020 at 11:07 PM Wolfgang Reissenberger <
>>>>> sterne-jaeger at openfuture.de> wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I think we are close to finishing the release. I personally would opt
>>>>>> to wait for another week and keep an eye stability.
>>>>>>
>>>>>> Maybe we should take another look if the default settings in the
>>>>>> StellarSolver profiles work a) for typical camera/scope combinations and b)
>>>>>> for all platforms.
>>>>>>
>>>>>> For example with my RPi, I needed to change the Parallel Algorithm to
>>>>>> „None“ because parallelity brought KStars down. Is the default setting
>>>>>> „None“ and I changed it somewhen? With all the new parameters I would
>>>>>> prefer having a robust setup and leave it to the user to optimize speed.
>>>>>>
>>>>>> @Jasem: please take a closer look to MR!122, since it fixed 4(!)
>>>>>> regressions I introduced with my capture counting fix MR!114. Hopefully now
>>>>>> we have at least a proper coverage with automated tests...
>>>>>>
>>>>>> Wolfgang
>>>>>>
>>>>>> Am 09.11.2020 <09112020> um 22:04 schrieb Jasem Mutlaq <
>>>>>> mutlaqja at ikarustech.com>:
>>>>>>
>>>>>> Hello Folks,
>>>>>>
>>>>>> So back to this topic, any major blockers to the KStars 3.5.0 release
>>>>>> now?
>>>>>>
>>>>>> 1. Remote Solver should be fixed now.
>>>>>> 2. StellarSolver Profiles are more optimized now.
>>>>>> 3. Handbook not updated yet, but we can probably work on this shortly.
>>>>>> 4. Couple of pending MRs to take care of.
>>>>>>
>>>>>> How about Friday the 13th?
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Jasem Mutlaq
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 5, 2020 at 3:41 AM Robert Lancaster <rlancaste at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> Ok so then we would be changing the way we do version numbering with
>>>>>>> this, right?
>>>>>>> I believe now we typically add features in each new iteration 3.4.1,
>>>>>>> 3.4.2, etc etc
>>>>>>> and when it is really big like StellarSolver, then we make it a big
>>>>>>> release like 3.5.0
>>>>>>>
>>>>>>> With this new paradigm, we wouldn’t put new features into the master
>>>>>>> of the main 3.5 branch
>>>>>>> But instead we would work on a new 3.6 branch, and then bug fixes
>>>>>>> would go into the 3.5 branch
>>>>>>> to make each new minor release, like 3.5.1, 3.5.2 etc.
>>>>>>>
>>>>>>> Do I have this correct?
>>>>>>>
>>>>>>> If this is right, then it would be longer before users see new
>>>>>>> features in the main branch, but the
>>>>>>> tradeoff is that the main branch would have a LOT more stability.  I
>>>>>>> see this as a big positive.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> > On Nov 4, 2020, at 5:54 PM, Eric Dejouhanet <
>>>>>>> eric.dejouhanet at gmail.com> wrote:
>>>>>>> >
>>>>>>> > Hello Hy,
>>>>>>> >
>>>>>>> > Version 3.5.0 is only the beginning of the 3.5.x series, with more
>>>>>>> > bugfixes on each iteration (and possibly, only bugfixes).
>>>>>>> > So I have no problem leaving unresolved issues in 3.5.0.
>>>>>>> >
>>>>>>> > For instance, the Focus module now has a slight and unforeseeable
>>>>>>> > delay after the capture completes.
>>>>>>> > The UI reflects the end of the capture only, not the end of the
>>>>>>> detection.
>>>>>>> > This makes the UI Focus test quite difficult to tweak, as running
>>>>>>> an
>>>>>>> > average of the HFR over multiple frames now has an unknown
>>>>>>> duration.
>>>>>>> > Right now, the test is trying to click the capture button too soon
>>>>>>> 2
>>>>>>> > out of 10 attempts.
>>>>>>> > But this won't block 3.5 in my opinion (and now that I understood
>>>>>>> the
>>>>>>> > problem, I won't work on it immediately).
>>>>>>> >
>>>>>>> > In terms of reporting problems, the official way is stil
>>>>>>> bugs.kde.org,
>>>>>>> > but there's quite a cleanup/followup to do there.
>>>>>>> > I'd say we can use issues in invent.kde.org to discuss planned
>>>>>>> > development around a forum/bugzilla issue or invent proposal (like
>>>>>>> > agile stories).
>>>>>>> > There are milestones associated with several issues (although I
>>>>>>> think
>>>>>>> > they should be reviewed and postponed).
>>>>>>> > And we can certainly write a punchlist: check the board at
>>>>>>> > https://invent.kde.org/education/kstars/-/milestones/3
>>>>>>> >
>>>>>>> > Le mer. 4 nov. 2020 à 22:38, Hy Murveit <murveit at gmail.com> a
>>>>>>> écrit :
>>>>>>> >>
>>>>>>> >> Eric,
>>>>>>> >>
>>>>>>> >> I would add to your list:
>>>>>>> >>
>>>>>>> >> - KStars Handbook (review update sections to reflect 3.5.0) and
>>>>>>> finally (perhaps manually if necessary) put the latest handbook online.
>>>>>>> >>
>>>>>>> >> - Review the extraction settings. I spent a bit of time looking
>>>>>>> at the default HFR settings, and based on some experimentation (truth be
>>>>>>> told, with a limited amount of data) adjust things a little differently
>>>>>>> than my first guess (which was basically focus' settings).
>>>>>>> >> Rob: My intuition is that I should adjust the default
>>>>>>> StellarSolver star-extraction settings for Focus and Guide as well in
>>>>>>>  stellarsolverprofile.cpp. I don't know whether you've already
>>>>>>> verified them, and want to release them as they are, or whether they are a
>>>>>>> first shot and you'd welcome adjustment?
>>>>>>> >>
>>>>>>> >> Also, Eric, I suppose I should be adding these things here:
>>>>>>> https://invent.kde.org/education/kstars/-/issues
>>>>>>> >> Is that right? Sorry about that--ok, after this thread ;) But
>>>>>>> seriously, your email is a good summary, and from that link
>>>>>>> >> it doesn't seem as easy to see which are "must do by 3.5.0" and
>>>>>>> which are "nice to have someday".
>>>>>>> >> A 3.5.0 punchlist would be a nice thing to have.
>>>>>>> >>
>>>>>>> >> Hy
>>>>>>> >>
>>>>>>> >> On Wed, Nov 4, 2020 at 12:58 PM Eric Dejouhanet <
>>>>>>> eric.dejouhanet at gmail.com> wrote:
>>>>>>> >>>
>>>>>>> >>> Hello,
>>>>>>> >>>
>>>>>>> >>> Where do we stand now in terms of bugfixing towards 3.5.0?
>>>>>>> >>>
>>>>>>> >>> - StellarSolver has all features in, and 1.5 is finally out at
>>>>>>> Jasem's PPA.
>>>>>>> >>> - However Gitlab CI still complains about that lib package (see
>>>>>>> >>> https://invent.kde.org/education/kstars/-/jobs/75941)
>>>>>>> >>> - Unitary tests are being fixed progressively, mount tests are
>>>>>>> down to
>>>>>>> >>> ~20 minutes (yeees!)
>>>>>>> >>> - From my tests, the remote Astrometry INDI driver is not usable
>>>>>>> >>> anymore from Ekos.
>>>>>>> >>> - The issue raised with flat frames is confirmed fixed (at least
>>>>>>> by me).
>>>>>>> >>> - Meridian flip is OK (but I had not enough time to test TWO
>>>>>>> flips in a row).
>>>>>>> >>> - Memory leaks are still being researched in Ekos.
>>>>>>> >>> - There is an issue when duplicating an entry in a scheduler job,
>>>>>>> >>> where the sequence associated is copied from the next job.
>>>>>>> >>>
>>>>>>> >>> Could we get a 3.6 branch where we will merge development of new
>>>>>>> features?
>>>>>>> >>> And master for bugfixing 3.5.x until we merge 3.6 new features
>>>>>>> in?
>>>>>>> >>> (we'd still have to port bugfixes from master to 3.6)
>>>>>>> >>> I don't think the opposite, master for 3.6 and a separate living
>>>>>>> >>> 3.5.x, is doable in the current configuration (build, ppas,
>>>>>>> MRs...).
>>>>>>> >>>
>>>>>>> >>> --
>>>>>>> >>> -- eric.dejouhanet at gmail.com - https://astronomy.dejouha.net
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > -- eric.dejouhanet at gmail.com - https://astronomy.dejouha.net
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20201111/40910b19/attachment-0001.htm>