[Okular-devel] Engaging

[Fabio D'Urso]

On Sunday, March 31, 2013 04:06:49 PM Yuval Levy wrote:
> Thank you Fabio for the short off-list insights that made me feel
> confident to write this back:
> 
> On 13-03-30 09:39 AM, Fabio D'Urso wrote:
> > I'm glad you liked the new annotation features :)
> 
> "like" is an understatement.  In my opinion *PDF* annotations are the
> killer feature of Okular, and Okular as a *PDF* viewer is a killer app
> to enable the survival of free desktops along proprietary desktops.
> 
> > I don't see inconsistency between "Save as" and "Save copy as" as
> > functionalities, but it could be because I "just know" what they do :)
> 
> OK, I was trying to bring this point as a usability argument for menu
> simplicity.  I'll venture on thinner ice now:  Okular declares itself on
> its home page as "universal" document reader.  Embedded *PDF*
> annotations seem to me more universal than XML annotations because they
> are more widely supported and they enable me to communicate
> "universally", with more users.  Everything else follows.
> 
> > I see the "Annotations only (XML)" option a little hard to do:
>
> Well, I suggested this one because I did not want to take away existing
> functionality from users who might have a strong attachment to it.  I do
> not want to impose my personal views on anybody.  That said, IMHO there
> is no need to save *PDF* annotation to XML at all.  Newer versions of
> Okular should reflect this and stop using XML for the PDF annotations.
> Even better:  on first start they should check for existing XML
> annotations to PDF files and offers a migration of all of my documents
> (with three options: Yes, No, Remind me later).  If I had a choice 18
> months ago between XML and embedded annotation, my choice would have
> been for embedded.  Now I am stuck with XML annotations that I'd rather
> convert all to embedded ones.
> 
> > what would you
> > do if the user chooses this option on a already annotated PDF file? Would
> > you only save modified annotations to XML? What about deleted ones?
> > Currently we refuse to use the XML format on already annotated PDF
> > documents (that's why you get the "annotation chnages will not be saved
> > automatically" warning in such documents).
> 
> With all due respect, what Okular does now is unnecessarily complex.
> Why not save annotation into PDF documents automatically?  Of course
> with a backup copy of the original file, to account for data integrity
> concerns such as <https://bugs.kde.org/show_bug.cgi?id=301774#c3>.  The
> backup copy could be optional in the preference, but the option should
> be on by default.

I'm all for it. Keep in mind that PDF annotation write support is relatively 
new so we took some time to see if it causes issues.

> > Yuval, are you willing to work on it?
> 
> As I wrote to you privately, there are two constrains: skill and time.
> Skill can be compensated for by training and mentoring.  Indeed with a
> little of encouragement from you I was able to build Okular on my end.
> Time is, unfortunately, a more critical constrain.  I almost completely
> stopped contributing to (other) Open Source projects 18 months ago.  It
> is very unlikely that my situation will change in the next 18 (or more
> likely 30) months.  Despite all the good will I can put forward, my
> contributions will be very limited.
> 
> Here an initial contribution.  Could somebody with edit privilege for
> the website add the following two FAQs under "Compiling Okular" ?
> 
> Q: Cmake quits with 'ERROR: cmake/modules/FindKDE4Internal.cmake not found'
> 
> A: Missing development headers.  On Ubuntu (12.10) type the command
> 'sudo apt-get install kde-workspace-dev'
> 
> Q: Cmake ends stating that the following REQUIRED packages could NOT be
> located on your system: QImageBlitz (kdesupport or higher)
> 
> A: Missing a required library.  On Ubuntu (12.10) type the command 'sudo
> apt-get install libqimageblitz-dev'

They are standard KDE4 libraries, they are part of the "proper KDE 4 
environment". After all, I don't think it's *too* hard to locate them, is it?

> I believe Okular is critically important to the survival of free
> desktops alongside proprietary ones.  The ability to read documents in a
> universal format (*PDF*) while respecting their integrity; to annotate
> them; and to exchange these documents with annotations in a
> standardized/agreed way across systems and organizational boundaries are
> a critical functionality of an office machine.  Okular is, to my
> knowledge, the best tool for the task in the free software world.
> 
> > A regular "Save" option has already been requested in bug 301774, so this
> > should probably be discussed there.
> 
> Thank you for pointing me to the bug report.  You will excuse me for
> continuing the discussion here rather than there, for two reasons: (1)
> that bug report is flawed; and (2) this discussion is about more than
> just a mere functionality / bug report.
> 
> Let me first handle what is IMHO a flaw of the bug report.  Frankly (and
> maybe this is a criticism of the KDE bug tracker) it is scary.  That bug
> report (and an even worse example is bug report 151614) looks to me like
> a thread in a web-based forum.  I had to read through flames and other
> irrelevant stuff to distill the essence of the bug report.  This is bad
> use of potential contributor time.  That bug tracker needs cleaning.
> Anyway, with over 400 bugs, somebody should sift through that bug
> tracker and clean it.
> 
> The first thing I would expect in a bug tracker is that the bug
> description should be editable.  Does not seem to be the case on this
> bug tracker, but it might be that I do not have sufficient access rights
> to do so.

That's correct, I do see an edit button.

> What is now the description would be comment #1 on the bug.
> Whoever takes ownership of the report (usually the user who reports it)
> should put a proper analysis in the body of the ticket, and that
> analysis should be updated based on insights from the discussion thread
> so that reading through the thread becomes optional.

In theory *all* comments should be constructive and relevant. Those bugs are 
examples of what happens when the discussion goes out of control.
Anyway, I'm not the right person to speak about this, nor this is the right 
mailing list to discuss KDE's bug tracker (it's the same one for all 
projects). Let me just say two things:
 - Sometimes users create bug reports and forget about them. Even if they
   do keep caring about it, they may not have the knowledge/skills to
   manage it
 - Moderation (and summarization in particular) takes (developers') time

> That was easy.  Well, you could ask me to clean up the current bug
> tracker, and I might agree and raise to the task, but there is a more
> difficult issue to solve first.  Difficult one, because there are
> trade-offs to be made, values and goals for the project to be decided.
> You can decide them, I can't.
> 
> Reading the two bug threads, I see two conflicting type of universality:
> 
> *1* Document-Format Universality:
> <https://bugs.kde.org/show_bug.cgi?id=151614#c23> stands for the
> proposition that Okular needs the XML format as long as not all back-end
> document formats support embedding annotations.  That means forever.
> Even if all back-ends could process annotations, there will inevitably
> be differences and incompatibilities.  How to add text or vector
> annotation to a TIFF or a JPEG?  It is IMHO impossible to annotate all
> document formats in the exact same way.  Okular's own XML format makes
> sense because it enables to add a uniform annotation layer on top of
> each document, no matter its original format.  This, however, comes at a
> very high cost, since it is generally bound to the user's profile and as
> such far from being "universal".

Let's split the idea from the implementation: the idea is to be able to let 
users annotate all supported formats in the most standard-compliant way we 
can, and fall back to a custom okular-only format if the standard way is not 
supported/doesn't exist.

Our current implementation of this fallback buries annotations in the docdata/ 
directory by default. You can use the document archive format to export them 
and be able to share them with other okular users. This is not what you call 
"user universal", but it's the best we can do and, imho, it's better than 
nothing.

> *2* User Universality:  For users, universality is the ability to share
> documents with annotation across platforms and organizational
> boundaries.  From this perspective, the cost imposed by document
> universality are excessive since the result is that documents with
> annotations are difficult to share even between two users of the same
> machine (the XML annotations are stored in a user-specific folder!).
> 
> The question becomes: what is more important to you?  Universal support
> of document formats or universal support of users across platforms?

So, if you are asking to get rid of annotation capabilities for all formats 
but PDF I say no.
If you are asking to stop "hiding" annotations in the docdata/ directory I say 
that we can discuss about it. IIRC there have been proposals about adding 
configuration options to store .xml files in the same directory as the 
document. I personally agree, but I haven't thought thoroughly about this yet.

> <http://okular.kde.org/> states that Okular is universal in that it
> works on multiple platforms.

I've always thought "universal" meant a lot of document formats. I think the 
multiplatform thing refers to the fact that this is a KDE 4 application, and 
KDE 4 applications run on multiple platforms thanks to portable underlying 
libraries.

> IMHO this should not be read in a pure
> technical way (i.e. works on Linux, Windows, Mac, etc.), but in a more
> holistic way: it works for me and it works for you because we can both
> access the same original document, in its integrity, and the comments we
> each added to it.  The interchange is at the document level and that's
> where universality should reside, IMHO.
> 
> Remember the Unix principle: single-purpose tools.  I don't need Okular
> to support a plethora of document formats.  Some of them are nice to
> have.  Some of them are plain redundant and a drag on development of
> what IMHO is the key purpose of Okular: read and annotate PDF documents.

Okular is labelled as a document viewer, not as "annotator" :)
Annotations are only a feature.

> For TIFFs and JPEG there is Gwenview (and if I wanted to annotate them
> with Okular, I can use ImageMagick convert to embed them in a PDF).  ODF
> is well taken care natively by LibreOffice (which can save to PDF as
> well).  A bunch of the formats listed on
> <http://okular.kde.org/formats.php> result in a drag on the one unique
> proposition for which there is critical need that Okular seems designed
> to satisfy: reading and annotating *PDF*.

That's just wrong, as I said Okular is much more than a PDF annotator. You 
have to think about other users' needs too.

> I think I have already exposed too much of my own vision and it is not
> up to me, an outsider, to express a vision for Okular.  It is up to you,
> the insiders.  What do you want Okular to be?
> 
> I would recommend taking a hard look at what kind of universality you
> want.  I would suggest defining three tiers of support for document types:
> (1) fully supported
> (2) partially supported
> (3) not supported
> 
> Fully supported would be with embedded annotations ONLY (migrate from
> XML to embedded annotation).
> 
> Partially supported would be read-only, or maybe with XML annotations.
> 
> Not supported: make a conscious decision that Okular does not add much
> value to supporting the format; and that support for the format does not
> add value to Okular either. Drop them.  Make Oklar leaner and focus your
> limited resources on the top two tiers.
> 
> Personally I would put ODT, TIFF, and JPEG in (3).  And PDF and epub in (1).
> 
> Full support would mean *migrating* from XML annotation to embedded
> annotation whenever possible.  It means stepping up to a better level of
> support: now the annotation are truly universally available, as opposed
> to only within Okular, and only within the one user account with the XML
> file.

I don't see any advantage from keeping (2) and discarding (3), and I don't see 
why (2) documents should be read-only.

That'd be a regression for a lot of use cases. If we did that literally, all 
existing annotations on non-PDF documents would suddenly be useless and users 
would start to have issues with documents they used to be able to open.

> And to prevent disasters, Okular can (and should), on save, rename the
> original document to something like document.timestamp.bak.pdf and save
> the new document with the new embedded comments as document.pdf.
> Without asking any question.  With an optional preference setting to
> take risks and not to save a backup copy.
> 
> Depending on the view you take regarding the purpose of Okular, you may
> like my proposal or you may find it heretically subvertive.  I am not
> suggesting you should do it, I am just saying that this is how I would
> do it if I had the time and the skill.
>
> > A quick note about your SSD consideration: annotations are not the only
> > things we write to docdata/, we also write the viewport history (aka
> > "restore last seen page when you reopen a document") and filled form
> > values.
> 
> I have seen this when analyzing the XML files.  I would suggest using a
> temporary file and save from the temporary file to the persistent file
> in regular intervals, configurable by preference.  If the preference is
> 0, keep the current behaviour (let the user make the trade-off between
> work-preservation and SSD-preservation).  If the preference is -1, never
> write to a persistent file (i.e. let the user write only when they go
> for the Save function).  The assumption is that /tmp is in RAM disk
> (where it should be for an SSD-only system).

I don't see the benefit in going through a temporary file. Are you saying that 
you suggest a configurable "Autosave metadata every X minutes" option (with 
"never" being an option too)? Can you open a "wish" bug about it?

> >> Another functionality that would be helpful to me and I have not found a
> >> way to get around:  I now have hundreds of marked up PDFs.  The
> >> annotations are all in $(kde4-config
> >> --localprefix)/share/apps/okular/docdata/ and I would like to export all
> >> of my library of marked up PDFs into PDFs with embedded annotations.
> >> Any idea how I could do this?
> > 
> > There's no automated way to do this. You have to open and "Save As" each
> > file separately.
> 
> Here is a user account in favour of a "migration path".
> [snip]

Yes, I see your point and, don't get me wrong, I agree it would be useful. 
OTOH, this doesn't seem to me a very big priority (i.e. there are more urgent 
things to do). Please open a wish about this too, so people can vote it. If it 
gets enough votes maybe we will prioritize it.

As a side note, the beauty of FOSS software is that everyone can scratch his 
own itches. If you want such a feature you can either do it yourself or hire 
someone to do it (obviously it's always better to discuss the design in 
advance if you want it to be merged upstream). We have had cases of paid 
people working on Okular in the past.

> > This is bug 159601. I've started some work on it, check out the
> > "configurable- review-tools" git branch. I've got a few more patches on
> > my laptop waiting to be polished and pushed :)
> > 
> > You can find some (old) screenshots here:
> >  - http://mail.kde.org/pipermail/okular-devel/2012-July/012047.html (look
> >  for>  
> >    the attachments at the end)
> >  
> >  - http://mail.kde.org/pipermail/okular-devel/2012-August/012054.html
> 
> Looks great!  Now that I built my own Okluar I would like to apply the
> patches and play with them.  Unfortunately the mailing list archive
> displays them inside the HTML which is rather inconvenient; and I do not
> see the patches attached to the bug report.  Can you please forward a
> zip file with the patches or share them in another convenient way? or do
> I really have to copy&paste them individually from my web browser into
> text files?

Those are old patches, I've started to polish them and publish them on git. 
Just do a "git checkout configurable-review-tools" and recompile. Note that 
not all patches have been pushed to git yet.

> > Thank you for caring about us and Okular,
> > Please let us know if you are willing to work on these issues, we always
> > need manpower :)
> 
> Thank you for being so welcoming.  I hope that I did not ruff any
> feathers with my comments, especially in light of the sensitivities that
> I start to understand based on what I saw in the threads for bugs 301774
> and 151614.
> 
> At the risk of repeating myself, I believe there are no alternatives in
> the free software world that are as good as Okular in matching or
> improving on annotation functionality that is available on proprietary
> desktops through Adobe's Reader.  It is vital to the survival of the
> Linux desktop that Okular thrives.  I am willing to work on the issues
> discussed above, or on reducing the bug tracker to a manageable
> dimension.  I would suggest first a discussion amongst contributors
> about Document-Format Universality vs User Universality.  I believe that
> making clear which one of the two has higher priority will solve a lot
> of the conflicts that manifest themselves in the less than harmonious
> comments.  It is also necessary to prioritize bug reports.

Sorry if I repeat myself one more time: I don't agree on removing support for 
non-PDF annotations nor "3rd-class" documents formats.
Personally, my top priority is to prevent users from losing their work.

As a side note, I've been told that you are not subscribed to this mailing 
list. In fact someone had to manually approve your messages. Please send a 
email to okular-devel-request at kde.org with the subject "subscribe".

One more thing: it's usually much better to split different ideas in several 
small discussions rather than making a single huge post. It's easier to focus 
on each idea and follow the discussion in general.

Fabio