NGC Thumbnails

Wed Apr 19 16:25:59 UTC 2017

All,

I tweaked the settings as much as I could and I did a run of the NGC objects to see how many of them came out usable.  I used data from the DSS1R for RED, DSS for GREEN and DSS1B for BLUE.  I used the same sizing parameters for the original monochrome pulls.

I am attaching a link to the onedrive shared file for you guys to pull and review:

https://1drv.ms/u/s!AnSnRFrNRtfXviM4XnBnN9neLucR

You will probably notice quite a few of the images have a yellow or green overall appearance.  I am not 100% sure if that is because of missing data or the histeq method I chose this time.  Probably a bit of both.

Some of the images appear to be quite usable like the attached NGC1432 and NGC2024.  That being said, many of the objects came out like NGC10, NGC40 and NGC56.  There are also ones that have significant source data issues like NGC101.

My question to the group is this, Do we feel there is enough good picture data like 1432 and 2024 to warrant the time and effort to comb through the 7840 objects to get the color objects out and include them in the mix.  Once we identify the 'good' data images, I have to rename and convert them to PNG for inclusion.  I would also intend to split up the excel spreadsheet so that the ones we deemed worthy would be pulled in RGB if we ever needed to pull them again.  The remaining black and white images would be either candidates for further experimentation to get the color data right in them or they would be left as monochrome in the thumbnails collection.

Please weigh in on this and also let me know if you would be willing to QA these images and give me a listing of the good ones you find.  This is a fairly time consuming effort and I would like some help as well as guidance on what others feel are decent images or not.

John

________________________________
From: John S. <myxlflik at hotmail.com>
Sent: Tuesday, April 18, 2017 3:01 PM
To: Jasem Mutlaq; KStars Development Mailing List
Subject: Re: NGC Thumbnails

Jasem,

You can use my last name (Sadler) for the credit on the packages.  I played around with the data some more and have gotten a few images to come out acceptably, but most of completely out of balance when you convert them to RGB.  I think I found out why.  One of my underlying assumptions about the DSS data appears to be incorrect.  Although the original DSS survey is monochrome, it does not appear to be filtered.  It appears to be visible light.  Obviously when you assign full spectrum to one channel it's going to play all kinds of havok with the balance of the channels.  I am not certain if there is a practical way to deal with the data, but perhaps a 'green' channel can be extracted by subtracting the red and blue channels from the DSS green channel.  This all would require significant effort but may yield results and be worthwhile if we can find a way to do it in a batch manner.  I can certainly pull all three channels separately and start playing to see if I get any kind of real results.  I will let you know what I find out.

John

________________________________
From: John S. <myxlflik at hotmail.com>
Sent: Tuesday, April 18, 2017 9:27 AM
To: Jasem Mutlaq; KStars Development Mailing List
Subject: Re: NGC Thumbnails

Great.  I am continuing to tweak and play with settings on the object pulls to see if I can get better data.  The problem I think is that they don't have a setting to adjust the levels and it seems like the blue channel is too weak overall to make for a proper color balance.  The last thing I want to do here is manually adjust 13000 images.  Even with a batch process it feels like a lot of effort.  I will keep you posted as to what I find.  I will start 'versioning' the tarball files as I change them so that you will know if it's newer than what you have now.

________________________________
From: Kstars-devel <kstars-devel-bounces at kde.org> on behalf of Jasem Mutlaq <mutlaqja at ikarustech.com>
Sent: Tuesday, April 18, 2017 8:13 AM
To: KStars Development Mailing List
Subject: Re: NGC Thumbnails

Yup we can publish an updated version if we get better data

On Tue, Apr 18, 2017 at 2:59 PM, John S. <myxlflik at hotmail.com<mailto:myxlflik at hotmail.com>> wrote:

Oh wow.  I thought you wanted to try to work on them some more.  Assuming we can get better images for  this, what will the process be to update this now that it is incorporated?  Just publish a new version and have them download it via that data downloads in Kstars?

________________________________
From: Jasem Mutlaq <mutlaqja at ikarustech.com<mailto:mutlaqja at ikarustech.com>>
Sent: Tuesday, April 18, 2017 4:38 AM
To: John S.
Cc: Hans; KStars Development Mailing List
Subject: Re: NGC Thumbnails

I just added them to KStars. I used your name "John S." and email address. Please let me know if you want to change any of those as they appear in the download dialog. I'll try to add previews as well since we do not have previews for _any_ download before.

On Mon, Apr 17, 2017 at 7:44 PM, John S. <myxlflik at hotmail.com<mailto:myxlflik at hotmail.com>> wrote:

Did anyone have any further input on this?  Is it worth pursuing and adding to Kstars?

________________________________
From: John S. <myxlflik at hotmail.com<mailto:myxlflik at hotmail.com>>
Sent: Saturday, April 15, 2017 11:00 AM

To: Hans
Cc: KStars Development Mailing List
Subject: Re: NGC Thumbnails

Here is a link to the current NGC file tarball and the rudimentary install and removal scripts.  This has only been tested on Linux Mint, but suspect it will be fine on all versions:

https://1drv.ms/u/s!AnSnRFrNRtfXviEYxBIM2pNlmuc6

Here is a link to the current IC file tarball.  Same caveats as the NGC file:

https://1drv.ms/u/s!AnSnRFrNRtfXviLmrzlp6Ru6N7KZ

The images are all of the DSS2R (Red Plate) survey only.  There is an ability to pull data into RGB using the DSS2R, DSS and DSS2B surveys (or really any other combination for that matter).  I have found that in nearly all cases, the colors needed subsequent adjustment in order to make the images look palatable.  I honestly felt the red channel ones and gray scale worked out just find overall.  I know there are some outright bad images in there and I haven't had the time or inclination to weed through the 13,000 objects I pulled to clean it up as of yet.  I can certainly do it, but it would largely depend on how strongly this group felt about it's inclusion.  I agree it should be an add-on package and not part of the base install.  We will need to make sure to credit the sources of the images as well.

One thing we may need to consider is directory management.  Downloading all of these images puts 13,000+ files in the $HOME/.local/share/kstars directory.  That's a pretty good chunk of files for a single directory.  I may make sense to compile these into some sort of database file or put them in a directory structure.  I am not a developer though and have NO skill when it comes to that aspect of this project.

John

________________________________
From: John S. <myxlflik at hotmail.com<mailto:myxlflik at hotmail.com>>
Sent: Saturday, April 15, 2017 10:15 AM
To: Hans
Cc: KStars Development Mailing List
Subject: Re: NGC Thumbnails

I started this project for my own personal gratification. I personally don't mind that some of the objects are subpar quality. Most of those objects are very interesting to me anyway. However, I would like to make sure that it meets certain quality levels. I am open to input for how to rectify the situation. I believe the limits of the survey itself will limit the outlet quality though. If you want I'll post the links to the two tarballs for the NGC and IC objects and you can download them a try for yourself.

John

> On Apr 15, 2017, at 9:02 AM, Hans <hans at lambermont.dyndns.org<mailto:hans at lambermont.dyndns.org>> wrote:
>
> Agreed, this is something very nice to have. And it is tedious work too.  For
> images that don't look too good I expect we'll even get negative remarks, but
> that should not inhibit this project. To me having a skimpy image is far better
> than no image at all.
>
> Is the idea to have an image library or to dynamically download them on demand ?
>
> -- Hans
>
> Jasem Mutlaq wrote on 20170415:
>
>> I'm attaching a few examples so you can see why we can't use them as is in
>> KStars.
>>
>>> On Sat, Apr 15, 2017 at 9:43 AM, Akarsh Simha <akarshsimha at gmail.com<mailto:akarshsimha at gmail.com>> wrote:
>>>
>>> Hi
>>>
>>> This is something very nice to have!
>>>
>>> A caveat to consider is the licensing for DSS imagery. They are not
>>> available for commercial use, so we must ship this as an add-on through
>>> KNewStuff ("Get New Data")
>>>
>>> Other sources of images are SDSS, the Hubble Legacy Archive (hla.stsci.edu<http://hla.stsci.edu>)
>>> and the recently released PanStarrs 1 survey. I think the PanStarrs 1
>>> survey has a large sky coverage (possibly even all sky -- I haven't
>>> checked). SDSS has a somewhat small footprint, and the Hubble only has
>>> objects of interest.
>>>
>>> I didn't quite get what the problem involved is. The DSS is made from
>>> digitizing scanned plates mostly from the Palomar Observatory and UKSTU
>>> telescope. The result is that many plates have artefacts, and they are
>>> completely uncalibrated. Planetary nebulae and bright emission nebulae tend
>>> to be overexposed. There is no way around these problems other than going
>>> to other surveys.
>>>
>>> Regards
>>> Akarsh
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Jasem Mutlaq
>
>
>
>
>

--
Best Regards,
Jasem Mutlaq

--
Best Regards,
Jasem Mutlaq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb-ngc1432_rgb.jpg
Type: image/jpeg
Size: 163906 bytes
Desc: thumb-ngc1432_rgb.jpg
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0006.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb-ngc2024_rgb.jpg
Type: image/jpeg
Size: 89041 bytes
Desc: thumb-ngc2024_rgb.jpg
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0007.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb-ngc10_rgb.jpg
Type: image/jpeg
Size: 8146 bytes
Desc: thumb-ngc10_rgb.jpg
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0008.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb-ngc40_rgb.jpg
Type: image/jpeg
Size: 6950 bytes
Desc: thumb-ngc40_rgb.jpg
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0009.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb-ngc56_rgb.jpg
Type: image/jpeg
Size: 12005 bytes
Desc: thumb-ngc56_rgb.jpg
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0010.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb-ngc101_rgb.jpg
Type: image/jpeg
Size: 7476 bytes
Desc: thumb-ngc101_rgb.jpg
URL: <http://mail.kde.org/pipermail/kstars-devel/attachments/20170419/67eb0434/attachment-0011.jpg>