[Owncloud] Automating what Frank means when he says "Please give it a really good hard test." Help build the test suite!

[Jörn Friedrich Dreyer]

TLDR: Describe a feature with scenarios in gherkin[1]! Do it! Do it now!

Have you, too, asked yourself what Frank actually means when he says
"Please give it a really good hard test."? I always imagine a dark
cellar, chained to a desk, where I have to sit late into the night to
click through every possible action ownCloud offers. What else would a
good hard test be?

But I don't want to ruin my evenings like this. I prefer sitting in my
garden, sipping a cold beer (you are hereby invited to join me, should
you visit Hannover). There must be a better solution! I mean, we are
software developers ... we automate things! So, Tom (DeepDiver) and I
started working on setting up an automated test suite that emulates a
user sitting in front of a browser, doing all the "really good hard
test"ing. Now we need to tell it what to do!

As an example I just committed some Gherkin scenarios for the core files
app[1]. The nice thing about them is: they are readable by your mom!
Take for example, this excerpt:

<gherkin>
Feature: files
  In order to organize my files
  As a mom
  I want to have basic file management

  Background
    Given I am logged in
    # these are the files hosted on demo.owncloud.org
    And I am in the files app

  Scenario: create files and folders
    When I click on the new button
    And I click on the new <type> action
    And I enter <filename>
    Then I should see <filename>
    And <filename> should have mimetype <mimetype>
    And <filename> should have size <size>

    Examples:
      | type   | mimetype             | filename               | size |
      | file   | text/plain           | simplefile.txt         | 0    |
      | file   | text/plain           | äöü ß ÄÖÜ € @.txt      | 0    |
      | file   | text/x-c             | test.cc                | 0    |
      | file   | text/x-php           | test.php               | 0    |
      | folder | httpd/unix-directory | Testfolder             | 0    |
      | folder | httpd/unix-directory | Testfolder with spaces | 0    |
</gherkin>

Not that hard to understand, is it?

Until now, this is basically a checklist, a tester could use to do
Franks bidding. Now, as I said earlier, I tend to find this kind of
tedious task ... very boring. Enter cucumber[2]: with a little glue
code, these gherkin scenarios can be executed as unit tests! The clue
code translates "I click on the new button" into code that cucumber uses
to execute actions in a browser and check results.

At this point I admit that I sometimes forget to execute the test suite
before commiting. Enter our continuous integration server[3] (thx again
tom for your awesome work there, too!):
Jenkins will monitor the commits and execute the cucumber test suite
after each checkin. He currently posts the result of the simpletest test
suite into the IRC dev channel allowing you to blame me whenever I break
a feature our users expect ;)

In the end, this process will enable us to automate large parts of QA by
testing each commit against every feature with combination of sqlite,
mysql, Postgresql, Oracle, Firefox and Chrome/Chromium. We will always
be able to say "Feature A works, which means that you can do scenario X,
Y and Z". I cannot stress enough how awwwesome ... err ... sweeeeeeet
this is!

Sure,there is still a ton of work left to do, and as we all know the
first steps are always the hardest. Fortunately, even your mom, dad,
sister, brother or significant other can help us with the first step -
writing textual scenarios for features:

  1. Go to http://demo.owncloud.org
  2. Pick an app from the navigation bar
  3. Keep the gherkin syntax page open for reference
https://github.com/cucumber/cucumber/wiki/Gherkin
  4. Learn from feature examples
https://github.com/aslakhellesoy/cucumber-rails-test/tree/master/features (all
the *.feature files)
  5. Start with a general Feature description:

    Feature: [the app name is enough for now]
      In order to [goal]
      [stakeholder]
      wants [behavior]

6. Flesh out some scenarios with Given-When-Then:
https://github.com/cucumber/cucumber/wiki/Given-When-Then

    Scenario: browsing the gallery
      Given I am logged in
        And I have uploaded the demo files
       When I go to the gallery
       Then I want to see a thumbnail of <image>

    Examples:
      | image                                            |
      | Demo Image - ccc.jpg                             |
      | Demo Image - Laser Towards Milky Ways Centre.jpg |
      | Demo Image - Northern Lights.jpg                 |
	
7. Send the .feature as a response to this message or directly to me.

I'll take care of aggregating the first round of scenarios into
features. We'll then create a public repository on github and start
writing the glue code (Step Definitions[4]). But thats another topic for
the future.

Now, go ahead, write a few feature scenarios, and ask your neighbor to
do it, too! It'll save you lots of hours of silly clicking in the night!
And it will bring world peace!

so long

Jörn

[1]
https://gitorious.org/owncloud-acceptance-test-suite/owncloud-acceptance-test-suite
[2] http://cukes.info/
[3] http://ci.tmit.eu/
[4] https://github.com/cucumber/cucumber/wiki/Step-Definitions

-- 
Jörn Friedrich Dreyer (jfd at owncloud.com)
Software Developer
ownCloud GmbH

Your Data, Your Cloud, Your Way!

ownCloud GmbH, GF: Markus Rex, Holger Dyroff
Schloßäckerstrasse 26a, 90443 Nürnberg, HRB 28050 (AG Nürnberg)