testing UIs and improving a11y all at once!

[Joseph P. De Veaugh-Geiss]

Reviving an old thread with an update from SoK23 (see below).

On 8/10/22 12:47, Nicolas Fella wrote:
> Am 10.08.22 um 12:29 schrieb Harald Sitter:
>> Servus,
>>
>> A while ago I prototyped a "new" approach to UI testing and I'm
>> wondering if there's general interest in doing more Plasma testing
>> using it. I'm able to invest time in polishing the experience for us.
>>
>> Very rough prototype: 
>> https://invent.kde.org/sitter/selenium-webdriver-at-spi
>>
>> The testing is run through the accessibility API we have on Linux
>> https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/ as well as the
>> testing API selenium https://www.selenium.dev/ respectively the more
>> specific appium https://appium.io/
>>
>> Architecturally appium is an extension to selenium and selenium is a
>> client-server system where the client is the test and the server is a
>> so called webdriver. Webdriver is a standardized well-defined API of
>> various UI interactions https://www.w3.org/TR/webdriver/ and we'd
>> implement one based on the a11y APIs (the feature sets match fairly
>> well).
>>
>> Since selenium has wide spread use across the industry we get to use
>> excellent tooling on the client without any extra work from us. And
>> because it is so wide spread the stuff is generally very well
>> maintained. All we need to maintain is the webdriver that interacts
>> with the a11y API.
>>
>> The way this type of testing works is by UI interaction and state
>> validation. There is a kcalc test available in the prototype repo [1]
>> - the test operates the various UI elements to perform a calculation
>> and then checks that the output UI element contains the expected
>> value.
>>
>> A simple plasma test might open kickoff, and launch one of the
>> favorites, then validate that indeed a new window has opened.
>>
>> Since all this is driven by the a11y API there is the additional
>> advantage of making us notice a11y problems and deal with them,
>> resulting in bettery a11y support in the long run. Two birds, one
>> stone!
>>
>> What do you reckon?
>>
>> [1] 
>> https://invent.kde.org/sitter/selenium-webdriver-at-spi/-/blob/91486b50995ade23c6b03a54f77347c263c2f03a/calculatortest.py
>>
>> HS
> 
> +energy-efficiency at kde.org
> 
> Servus Harald,
> 
> that looks very cool!
> 
> It could also come in handy for energy consumption measurements where we
> need an automated and predictable way to emulate user interaction.
> 
> The idea of using a11y API for that came up, but AFAIK there's no work
> to actually do it on our side.
> 
> That's three birds one stone I guess?
> 
> Cheers
> 
> Nico
> 
> 
> PS: I pondered about different ways to do GUI automation for energy
> consumption measurements in my master thesis recently, but I left the
> a11y way for future work. If you are interested you can find the thesis
> at https://nx9294.your-storageshare.de/s/PDQK4DEyPMW8AHM
> 

Season of KDE student Nitin Tejuja has done some great work writing 
scripts in Selenium to be used for energy consumption measurements and 
accessibility testing. Nitin's work is based on selenium-at-spi 
developed by Harald.

You can read more at the KDE Eco blog:

*Final Weeks Of SoK23 With KDE Eco: Writing Scripts In Selenium To 
Reproduce KdeEcoTest Script*

   https://eco.kde.org/blog/2023-04-20-sok23-kde-eco-selenium-project-3/

Also, check out the step-by-step guide to be included soon as Chapter 4 
in the energy measurement handbook (draft):

https://invent.kde.org/teams/eco/be4foss/-/merge_requests/79/diffs#9535d50cb0a5be5a4f5e08ebc2a839c07afdb834

 > Selenium AT-SPI the future Kde tool to test accessibility and measure 
power consomption.
 > A step by step guide to use Selenium AT-SPI to automatise system 
testing of QT applications

You can support Nitin's work by boosting the Mastodon post promoting the 
above blog post:

   https://floss.social/@be4foss/110231776673032085

Cheers,
Joseph

-- 
Joseph P. De Veaugh-Geiss
KDE Internal Communications & KDE Eco Community Manager
OpenPGP: 8FC5 4178 DC44 AD55 08E7 DF57 453E 5746 59A6 C06F
Matrix: @joseph:kde.org

Generally available Monday-Thursday from 10-16h CET/CEST. Outside of 
these times it may take a little longer for me to respond.

KDE Eco: Building Energy-Efficient Free Software!
Website: https://eco.kde.org
Mastodon: @be4foss at floss.social