[Owncloud] Automating what Frank means when he says "Please give it a really good hard test." Help build the test suite!
thomas.mueller at tmit.eu
Fri Sep 21 09:20:25 UTC 2012
Am Freitag, dem 21.09.2012 um 11:09 schrieb Arthur Schiwon:
> Hey Jörn,
> thanks a lot, it's too long of course, but i read it anyway.
> You know i would really love to automatically test the LDAP backend. However,
> it requires an available LDAP server, otherwise there is not much to test. Is
> it possible to have one running on Jenkins?
My plan in terms of executing the test suite is based on heavy usage of automated vm
creation using vagrant (http://vagrantup.com/)
This is necessary to test ownCloud with a great variety of server configurations:
[sqlite, mysql, pg, (oracle?)] x [apache, nginx, lighttpd, (IIS?)] x [....]
ldap - among other authentication backends will 'just' add another dimension to this configuration space.
> Plus, it is not enough to check if configuring and login works, but also all
> the other lovely things which are used for local users. Then however, some
> features do not work (e.g. current encryption, because password changes are
> unknown). Do you have a fluffy sweet recipe for this, too? :)
>From a pure behavior driven testing approach I would expect that all scenarios should
work in any configuration combination.
Exceptions - as usual - prove the rule! ;-)
Tom aka DeepDiver
> On Friday, September 21, 2012 12:34:42 AM Jörn Friedrich Dreyer wrote:
> > TLDR: Describe a feature with scenarios in gherkin! Do it! Do it now!
> > Have you, too, asked yourself what Frank actually means when he says
> > "Please give it a really good hard test."? I always imagine a dark
> > cellar, chained to a desk, where I have to sit late into the night to
> > click through every possible action ownCloud offers. What else would a
> > good hard test be?
> > But I don't want to ruin my evenings like this. I prefer sitting in my
> > garden, sipping a cold beer (you are hereby invited to join me, should
> > you visit Hannover). There must be a better solution! I mean, we are
> > software developers ... we automate things! So, Tom (DeepDiver) and I
> > started working on setting up an automated test suite that emulates a
> > user sitting in front of a browser, doing all the "really good hard
> > test"ing. Now we need to tell it what to do!
> > As an example I just committed some Gherkin scenarios for the core files
> > app. The nice thing about them is: they are readable by your mom!
> > Take for example, this excerpt:
> > <gherkin>
> > Feature: files
> > In order to organize my files
> > As a mom
> > I want to have basic file management
> > Background
> > Given I am logged in
> > # these are the files hosted on demo.owncloud.org
> > And I am in the files app
> > Scenario: create files and folders
> > When I click on the new button
> > And I click on the new <type> action
> > And I enter <filename>
> > Then I should see <filename>
> > And <filename> should have mimetype <mimetype>
> > And <filename> should have size <size>
> > Examples:
> > | type | mimetype | filename | size |
> > | file | text/plain | simplefile.txt | 0 |
> > | file | text/plain | äöü ß ÄÖÜ € @.txt | 0 |
> > | file | text/x-c | test.cc | 0 |
> > | file | text/x-php | test.php | 0 |
> > | folder | httpd/unix-directory | Testfolder | 0 |
> > | folder | httpd/unix-directory | Testfolder with spaces | 0 |
> > </gherkin>
> > Not that hard to understand, is it?
> > Until now, this is basically a checklist, a tester could use to do
> > Franks bidding. Now, as I said earlier, I tend to find this kind of
> > tedious task ... very boring. Enter cucumber: with a little glue
> > code, these gherkin scenarios can be executed as unit tests! The clue
> > code translates "I click on the new button" into code that cucumber uses
> > to execute actions in a browser and check results.
> > At this point I admit that I sometimes forget to execute the test suite
> > before commiting. Enter our continuous integration server (thx again
> > tom for your awesome work there, too!):
> > Jenkins will monitor the commits and execute the cucumber test suite
> > after each checkin. He currently posts the result of the simpletest test
> > suite into the IRC dev channel allowing you to blame me whenever I break
> > a feature our users expect ;)
> > In the end, this process will enable us to automate large parts of QA by
> > testing each commit against every feature with combination of sqlite,
> > mysql, Postgresql, Oracle, Firefox and Chrome/Chromium. We will always
> > be able to say "Feature A works, which means that you can do scenario X,
> > Y and Z". I cannot stress enough how awwwesome ... err ... sweeeeeeet
> > this is!
> > Sure,there is still a ton of work left to do, and as we all know the
> > first steps are always the hardest. Fortunately, even your mom, dad,
> > sister, brother or significant other can help us with the first step -
> > writing textual scenarios for features:
> > 1. Go to http://demo.owncloud.org
> > 2. Pick an app from the navigation bar
> > 3. Keep the gherkin syntax page open for reference
> > https://github.com/cucumber/cucumber/wiki/Gherkin
> > 4. Learn from feature examples
> > https://github.com/aslakhellesoy/cucumber-rails-test/tree/master/features
> > (all the *.feature files)
> > 5. Start with a general Feature description:
> > Feature: [the app name is enough for now]
> > In order to [goal]
> > [stakeholder]
> > wants [behavior]
> > 6. Flesh out some scenarios with Given-When-Then:
> > https://github.com/cucumber/cucumber/wiki/Given-When-Then
> > Scenario: browsing the gallery
> > Given I am logged in
> > And I have uploaded the demo files
> > When I go to the gallery
> > Then I want to see a thumbnail of <image>
> > Examples:
> > | image |
> > | Demo Image - ccc.jpg |
> > | Demo Image - Laser Towards Milky Ways Centre.jpg |
> > | Demo Image - Northern Lights.jpg |
> > 7. Send the .feature as a response to this message or directly to me.
> > I'll take care of aggregating the first round of scenarios into
> > features. We'll then create a public repository on github and start
> > writing the glue code (Step Definitions). But thats another topic for
> > the future.
> > Now, go ahead, write a few feature scenarios, and ask your neighbor to
> > do it, too! It'll save you lots of hours of silly clicking in the night!
> > And it will bring world peace!
> > so long
> > Jörn
> > 
> > https://gitorious.org/owncloud-acceptance-test-suite/owncloud-acceptance-tes
> > t-suite  http://cukes.info/
> >  http://ci.tmit.eu/
> >  https://github.com/cucumber/cucumber/wiki/Step-Definitions
> Owncloud mailing list
> Owncloud at kde.org
More information about the Owncloud