Automated Tests Working Group

This is the home page of the Automated Test Working Group, that is the result of combining the old System Testing Working Group and the Unit Testing Working Group.

Team Members

See the full list at: http://volunteers.joomla.org/working-groups/automated-tests-working-group

Goals

• to improve the Quality of Joomla CMS software
• to create learning material about "how to test" for the Joomla Community

Roadmap

The next steps in this team are:

GSoC 2014

• focus on Google Summer of Code 2014 Project: Webdriver_system_tests_for_CMS

Sauce Labs integration

• Test SauceLabs and Travis integration with the repo at GitHub: https://saucelabs.com/opensource/travis. It will require the use of Paratest, to run several tests at the same time, otherwise it will take too long (more than 8 hours just the /Adminitrator/ tests)

Improvements in current tests

• (done) Create an Assert in all test that makes sure there is not any warning, even if the test passes. See this example:
  File:System tests-example of warning.png
• (done) Create a graphical test coverage report
• Move to Codeception framework. Some tests are being done here: https://github.com/joomla-projects/com_localise/tree/codeception#testing-with-codeception
• Create a Best Practice in System testing Joomla! extensions with https://github.com/joomla-projects/com_localise component.
  • reorganise folders inside codeception for _step objects and _pages (we need to wait for Codeception 2.1, see: http://phptest.club/t/issue-with-codeception-autoload-and-namespaces/192 )
  • Run tests in parallel, see: https://twitter.com/JAVI_GOMEZ/status/523422301384368128

Travis

We have Travis complaining about this:

E.................E.........
I'm sorry but your test run exceeded 50.0 minutes. 
One possible solution is to split up your test run.

We need to find a solution, for example: http://blog.travis-ci.com/2012-11-28-speeding-up-your-tests-by-parallelizing-them/

Test Joomla extension

The goal of this project is to test com_localise extension https://github.com/joomla-projects/com_localise/ to create a proof of concept and documentation to teach others how to test their Joomla Extensions.

After com_localise our efforts will be moved to com_weblinks https://github.com/joomla-extensions/weblinks/issues/10

Create a infrastructure for fast testing the CMS

We plan to run create a battery of webservers able to run several instances of the system tests with different configuration: browser, screen size, testing area...

The idea is to create a server infraestructure capable to create a set of virtual machines and run individually part of the system testing suite on each of them. Then send a signal back to issues.joomla.org to change the status of a pull request to OK or KO depending on the result of running the tests.

Documents

• Writing System Tests for Joomla!
• GSOC-Webdriver_system_tests_for_CMS repository: https://github.com/joomla-projects/GSOC-Webdriver_system_tests_for_CMS
• Writting system tests with Codeception: Testing Joomla Extensions with Codeception
• Javier Gomez presentation of Joomla System Testing at Joomla World Conference 2014: http://prezi.com/qsf17leurgbe/automated-system-testing-in-joomla/

Projects

• com_localise: https://github.com/joomla-projects/com_localise/tree/codeception project for testing a Joomla extension with Codeception framework adding parallel testing and Saucelabs or BrowserStack integration.
• https://github.com/joomla-projects/Codeception/ fork of original Codeception project for bug fixing and contributing to the main project.
• The Joomla Docker Container for System Tests