[Owncloud] Automating what Frank means when he says "Please give it a really good hard test." Help build the test suite!

[Arthur Schiwon]

Hey Jörn,

thanks a lot, it's too long of course, but i read it anyway.

You know i would really love to automatically test the LDAP backend. However, 
it requires an available LDAP server, otherwise there is not much to test. Is 
it possible to have one running on Jenkins?

Plus, it is not enough to check if configuring and login works, but also all 
the other lovely things which are used for local users. Then however, some 
features do not work (e.g. current encryption, because password changes are 
unknown). Do you have a fluffy sweet recipe for this, too? :)

Cheers
Arthur

On Friday, September 21, 2012 12:34:42 AM Jörn Friedrich Dreyer wrote:
> TLDR: Describe a feature with scenarios in gherkin[1]! Do it! Do it now!
> 
> Have you, too, asked yourself what Frank actually means when he says
> "Please give it a really good hard test."? I always imagine a dark
> cellar, chained to a desk, where I have to sit late into the night to
> click through every possible action ownCloud offers. What else would a
> good hard test be?
> 
> But I don't want to ruin my evenings like this. I prefer sitting in my
> garden, sipping a cold beer (you are hereby invited to join me, should
> you visit Hannover). There must be a better solution! I mean, we are
> software developers ... we automate things! So, Tom (DeepDiver) and I
> started working on setting up an automated test suite that emulates a
> user sitting in front of a browser, doing all the "really good hard
> test"ing. Now we need to tell it what to do!
> 
> As an example I just committed some Gherkin scenarios for the core files
> app[1]. The nice thing about them is: they are readable by your mom!
> Take for example, this excerpt:
> 
> <gherkin>
> Feature: files
>   In order to organize my files
>   As a mom
>   I want to have basic file management
> 
>   Background
>     Given I am logged in
>     # these are the files hosted on demo.owncloud.org
>     And I am in the files app
> 
>   Scenario: create files and folders
>     When I click on the new button
>     And I click on the new <type> action
>     And I enter <filename>
>     Then I should see <filename>
>     And <filename> should have mimetype <mimetype>
>     And <filename> should have size <size>
> 
>     Examples:
>       | type   | mimetype             | filename               | size |
>       | file   | text/plain           | simplefile.txt         | 0    |
>       | file   | text/plain           | äöü ß ÄÖÜ € @.txt      | 0    |
>       | file   | text/x-c             | test.cc                | 0    |
>       | file   | text/x-php           | test.php               | 0    |
>       | folder | httpd/unix-directory | Testfolder             | 0    |
>       | folder | httpd/unix-directory | Testfolder with spaces | 0    |
> 
> </gherkin>
> 
> Not that hard to understand, is it?
> 
> Until now, this is basically a checklist, a tester could use to do
> Franks bidding. Now, as I said earlier, I tend to find this kind of
> tedious task ... very boring. Enter cucumber[2]: with a little glue
> code, these gherkin scenarios can be executed as unit tests! The clue
> code translates "I click on the new button" into code that cucumber uses
> to execute actions in a browser and check results.
> 
> At this point I admit that I sometimes forget to execute the test suite
> before commiting. Enter our continuous integration server[3] (thx again
> tom for your awesome work there, too!):
> Jenkins will monitor the commits and execute the cucumber test suite
> after each checkin. He currently posts the result of the simpletest test
> suite into the IRC dev channel allowing you to blame me whenever I break
> a feature our users expect ;)
> 
> In the end, this process will enable us to automate large parts of QA by
> testing each commit against every feature with combination of sqlite,
> mysql, Postgresql, Oracle, Firefox and Chrome/Chromium. We will always
> be able to say "Feature A works, which means that you can do scenario X,
> Y and Z". I cannot stress enough how awwwesome ... err ... sweeeeeeet
> this is!
> 
> Sure,there is still a ton of work left to do, and as we all know the
> first steps are always the hardest. Fortunately, even your mom, dad,
> sister, brother or significant other can help us with the first step -
> writing textual scenarios for features:
> 
>   1. Go to http://demo.owncloud.org
>   2. Pick an app from the navigation bar
>   3. Keep the gherkin syntax page open for reference
> https://github.com/cucumber/cucumber/wiki/Gherkin
>   4. Learn from feature examples
> https://github.com/aslakhellesoy/cucumber-rails-test/tree/master/features
> (all the *.feature files)
>   5. Start with a general Feature description:
> 
>     Feature: [the app name is enough for now]
>       In order to [goal]
>       [stakeholder]
>       wants [behavior]
> 
> 6. Flesh out some scenarios with Given-When-Then:
> https://github.com/cucumber/cucumber/wiki/Given-When-Then
> 
>     Scenario: browsing the gallery
>       Given I am logged in
>         And I have uploaded the demo files
>        When I go to the gallery
>        Then I want to see a thumbnail of <image>
> 
>     Examples:
>       | image                                            |
>       | Demo Image - ccc.jpg                             |
>       | Demo Image - Laser Towards Milky Ways Centre.jpg |
>       | Demo Image - Northern Lights.jpg                 |
> 
> 7. Send the .feature as a response to this message or directly to me.
> 
> I'll take care of aggregating the first round of scenarios into
> features. We'll then create a public repository on github and start
> writing the glue code (Step Definitions[4]). But thats another topic for
> the future.
> 
> Now, go ahead, write a few feature scenarios, and ask your neighbor to
> do it, too! It'll save you lots of hours of silly clicking in the night!
> And it will bring world peace!
> 
> so long
> 
> Jörn
> 
> [1]
> https://gitorious.org/owncloud-acceptance-test-suite/owncloud-acceptance-tes
> t-suite [2] http://cukes.info/
> [3] http://ci.tmit.eu/
> [4] https://github.com/cucumber/cucumber/wiki/Step-Definitions