[Owncloud] Automating what Frank means when he says "Please give it a really good hard test." Help build the test suite!

[Thomas Müller]

Am Freitag, dem 21.09.2012 um 11:09 schrieb Arthur Schiwon:
> Hey Jörn,
> 
> thanks a lot, it's too long of course, but i read it anyway.
> 
> You know i would really love to automatically test the LDAP backend. However, 
> it requires an available LDAP server, otherwise there is not much to test. Is 
> it possible to have one running on Jenkins?
>

Hi Arthur,

My plan in terms of executing the test suite is based on heavy usage of automated vm
creation using vagrant (http://vagrantup.com/)

This is necessary to test ownCloud with a great variety of server configurations:
[sqlite, mysql, pg, (oracle?)] x [apache, nginx, lighttpd, (IIS?)] x [....]

ldap - among other authentication backends will 'just' add another dimension to this configuration space.
 
> Plus, it is not enough to check if configuring and login works, but also all 
> the other lovely things which are used for local users. Then however, some 
> features do not work (e.g. current encryption, because password changes are 
> unknown). Do you have a fluffy sweet recipe for this, too? :)
> 

>From a pure behavior driven testing approach I would expect that all scenarios should 
work in any configuration combination.

Exceptions - as usual - prove the rule! ;-)

Take care,

Tom aka DeepDiver

> Cheers
> Arthur
> 
> On Friday, September 21, 2012 12:34:42 AM Jörn Friedrich Dreyer wrote:
> > TLDR: Describe a feature with scenarios in gherkin[1]! Do it! Do it now!
> > 
> > Have you, too, asked yourself what Frank actually means when he says
> > "Please give it a really good hard test."? I always imagine a dark
> > cellar, chained to a desk, where I have to sit late into the night to
> > click through every possible action ownCloud offers. What else would a
> > good hard test be?
> > 
> > But I don't want to ruin my evenings like this. I prefer sitting in my
> > garden, sipping a cold beer (you are hereby invited to join me, should
> > you visit Hannover). There must be a better solution! I mean, we are
> > software developers ... we automate things! So, Tom (DeepDiver) and I
> > started working on setting up an automated test suite that emulates a
> > user sitting in front of a browser, doing all the "really good hard
> > test"ing. Now we need to tell it what to do!
> > 
> > As an example I just committed some Gherkin scenarios for the core files
> > app[1]. The nice thing about them is: they are readable by your mom!
> > Take for example, this excerpt:
> > 
> > <gherkin>
> > Feature: files
> >   In order to organize my files
> >   As a mom
> >   I want to have basic file management
> > 
> >   Background
> >     Given I am logged in
> >     # these are the files hosted on demo.owncloud.org
> >     And I am in the files app
> > 
> >   Scenario: create files and folders
> >     When I click on the new button
> >     And I click on the new <type> action
> >     And I enter <filename>
> >     Then I should see <filename>
> >     And <filename> should have mimetype <mimetype>
> >     And <filename> should have size <size>
> > 
> >     Examples:
> >       | type   | mimetype             | filename               | size |
> >       | file   | text/plain           | simplefile.txt         | 0    |
> >       | file   | text/plain           | äöü ß ÄÖÜ € @.txt      | 0    |
> >       | file   | text/x-c             | test.cc                | 0    |
> >       | file   | text/x-php           | test.php               | 0    |
> >       | folder | httpd/unix-directory | Testfolder             | 0    |
> >       | folder | httpd/unix-directory | Testfolder with spaces | 0    |
> > 
> > </gherkin>
> > 
> > Not that hard to understand, is it?
> > 
> > Until now, this is basically a checklist, a tester could use to do
> > Franks bidding. Now, as I said earlier, I tend to find this kind of
> > tedious task ... very boring. Enter cucumber[2]: with a little glue
> > code, these gherkin scenarios can be executed as unit tests! The clue
> > code translates "I click on the new button" into code that cucumber uses
> > to execute actions in a browser and check results.
> > 
> > At this point I admit that I sometimes forget to execute the test suite
> > before commiting. Enter our continuous integration server[3] (thx again
> > tom for your awesome work there, too!):
> > Jenkins will monitor the commits and execute the cucumber test suite
> > after each checkin. He currently posts the result of the simpletest test
> > suite into the IRC dev channel allowing you to blame me whenever I break
> > a feature our users expect ;)
> > 
> > In the end, this process will enable us to automate large parts of QA by
> > testing each commit against every feature with combination of sqlite,
> > mysql, Postgresql, Oracle, Firefox and Chrome/Chromium. We will always
> > be able to say "Feature A works, which means that you can do scenario X,
> > Y and Z". I cannot stress enough how awwwesome ... err ... sweeeeeeet
> > this is!
> > 
> > Sure,there is still a ton of work left to do, and as we all know the
> > first steps are always the hardest. Fortunately, even your mom, dad,
> > sister, brother or significant other can help us with the first step -
> > writing textual scenarios for features:
> > 
> >   1. Go to http://demo.owncloud.org
> >   2. Pick an app from the navigation bar
> >   3. Keep the gherkin syntax page open for reference
> > https://github.com/cucumber/cucumber/wiki/Gherkin
> >   4. Learn from feature examples
> > https://github.com/aslakhellesoy/cucumber-rails-test/tree/master/features
> > (all the *.feature files)
> >   5. Start with a general Feature description:
> > 
> >     Feature: [the app name is enough for now]
> >       In order to [goal]
> >       [stakeholder]
> >       wants [behavior]
> > 
> > 6. Flesh out some scenarios with Given-When-Then:
> > https://github.com/cucumber/cucumber/wiki/Given-When-Then
> > 
> >     Scenario: browsing the gallery
> >       Given I am logged in
> >         And I have uploaded the demo files
> >        When I go to the gallery
> >        Then I want to see a thumbnail of <image>
> > 
> >     Examples:
> >       | image                                            |
> >       | Demo Image - ccc.jpg                             |
> >       | Demo Image - Laser Towards Milky Ways Centre.jpg |
> >       | Demo Image - Northern Lights.jpg                 |
> > 
> > 7. Send the .feature as a response to this message or directly to me.
> > 
> > I'll take care of aggregating the first round of scenarios into
> > features. We'll then create a public repository on github and start
> > writing the glue code (Step Definitions[4]). But thats another topic for
> > the future.
> > 
> > Now, go ahead, write a few feature scenarios, and ask your neighbor to
> > do it, too! It'll save you lots of hours of silly clicking in the night!
> > And it will bring world peace!
> > 
> > so long
> > 
> > Jörn
> > 
> > [1]
> > https://gitorious.org/owncloud-acceptance-test-suite/owncloud-acceptance-tes
> > t-suite [2] http://cukes.info/
> > [3] http://ci.tmit.eu/
> > [4] https://github.com/cucumber/cucumber/wiki/Step-Definitions
> _______________________________________________
> Owncloud mailing list
> Owncloud at kde.org
> https://mail.kde.org/mailman/listinfo/owncloud