Dev/Tests – Indico

Context Navigation

Version 104 (modified by jnguyenx, 3 years ago) (diff)
--

0. Before starting

Tests are in folder Indicop and they are organized according to their type. Tests will only succeed if you have a working installation of Indico.

To run all the tests in the test suite:

python setup.py test

or with coverage:

python setup.py test --coverage --jscoverage

Then reports will be generated in folder indicop/report.

In short, here are all the python packages you have to install with easy_install:

sudo easy_install nose
sudo easy_install figleaf
sudo easy_install twill
sudo easy_install selenium

and all the linux packages needed to install with apt-get:

sudo apt-get install lcov
sudo apt-get install pylint
sudo apt-get install rhino

If you don't use linux, go through all the sections below to install the required packages.

A configuration file named tests.conf is needed in the folder Indicop. Take a look at the example configuration file to make yours.

1. Unit tests

1.a. Program to install

We are using the framework Nosetests. You can just install with:

sudo easy_install nose

In case of problems, follow this Installation Guide for Nosetests

For code coverage, install Figleaf, which is based on coverage.py. You can install with:

sudo easy_install figleaf

1.b. Writing unit tests

You can either use the standard UnitTest package provided in python, or you can code in Nosetests style, here is how I organize a test file:

#Note that the 2 following functions will be called for module, very handy if we need to run a single test
#So we do not need to specify a setup and teardown for each class
def setup_module():
    DBMgr.getInstance().startRequest()

def teardown_module():
    DBMgr.getInstance().abort()
    DBMgr.getInstance().endRequest()

class TestCategories(unittest.TestCase):
    
    def testBasicAddAndRemoveConferences(self):
        [...]
        c1._addConference(conf1)
        assert (croot.getNumConferences()==0)
        [...]

The complete documentation of nosetests can be found here Note that TestCategories is heriting from Python's unittest module, this is not necessary, but since nosetests is based on this default module, it is fully compatible with its synthax. So when you write your tests, you can either use unittest synthax or nosetests' one.

1.c. Naming and Files convention

A test file should correspond to a source file. For example: MaKaC_tests/conference_test.py must only have tests for MaKaC/conference.py. Classes and functions names should also match.

Of course, at some point testing many different things in a single file is also required. In this case, use explicit file names, like test_move_widget.py.

Nosetests uses this regular expression (?:^|[b_.-])[Tt]est to fetch tests, so our naming conventions will be:

Every file name has to end with indicop_test.py
Every class has to start with Test with a capital T class TestIndicop(unittest.TestCase)
Every function has to start with test def testIndicop()

1.d. Running tests

To run all the unit tests:

python setup.py test --unit

This will generate a report in indicop/report/pyunit.txt.

If you need to run only a specific test, use the flag --specify and use the nosetests convention:

file.py:class.function

For example:

python setup.py test --specify=unit/MaKaC_tests/conference_test.py:TestCategories.testBasicAddAndRemoveConferences

Or you can also specify a set of tests like this:

python setup.py test --specify=unit/MaKaC_tests/conference_test.py:TestCategories

and also only a file or a folder like this:

python setup.py test --specify=unit/MaKaC_tests

If you use this specify flag the output will be displayed directly in the console and NOT be written in the report.

1.e. Enable code coverage

To enable the code coverage, add the option --coverage:

python setup.py test --unit --coverage

This will show you which parts of the code were executed during the unit tests you ran. An HTML report will be output in several files, you can find them in indicop/report/pycoverage. Open the file index.html to see the summary of the coverage. These files will be overwritten each time a coverage test is run.

Note that class and function definitions are executed on import, which is why def blabla(self): is green in the HTML report files. The contents of the functions themselves are not executed.

2. Functional tests

2.a. Programs to install

Note: Selenium is going to be merged with Google's webdriver (http://selenium.googlecode.com/) *

You need:

Firefox
XPather (Firefox add-on)
Selenium IDE (firefox add-on)
Twill, which you can install with:
```
 sudo easy_install twill
```
Selenium egg, which you can install with:
```
 sudo easy_install selenium
```
java, which needs to be in your PATH.

2.b. Writing tests

Think of Selenium as the dumbest bot possibly existing :)

Handy functions are located in the file seleniumTestCase.py. A dummy user is automatically created from this Class to carry on the creation of conferences and so.

Here is the template of a Selenium's test file:

import time
from seleniumTestCase import SeleniumTestCase

class ExampleTest(SeleniumTestCase):
    def setUp(self):
        SeleniumTestCase.setUp(self)
    
    [...]

    def tearDown(self):
        SeleniumTestCase.tearDown(self)

if __name__ == "__main__":
    unittest.main()

Have a look at the file example_test.py to see how functional tests are written.

If you create a new conference, you might want this conference to be deleted automatically if a test fails. Add this line just after the creation of the new conference or meeting:

SeleniumTestCase.setConfID(self, sel.get_location())

Since it is very time consuming and painful to debug Selenium's generated code, tests should be small and test specific part of Indico. Of course, a few overall tests are needed.

Use Firefox's Selenium IDE to record your tests and set the output format to python, but be rigorous with the following points:

Selenium does not recognize Javascript auto-complete when a text is entered.
Solution: Enter the whole word by hand.
Javascript Calendar does not work either.
Solution: Enter the date by hand.
Do not make clicks or assertions on element containing IDs.
Solution: Use XPath instead. Xpaths are easy to find with Firefox's plugin XPather.
When replaying, Selenium can get confused with buttons' names and can end up clicking on the wrong button.
Solution: Once again, use XPath instead. Xpaths are easy to find with Firefox's plugin XPather.
Popups and AJAX requests are very bothersome to test with Selenium. The latter would not wait for a page or a tab to load if we do not explicitly tell to do so.
Solution: Use Selenium waitForText function whenever a window pops up, a tab is loading or an AJAX request is sent.

And since Selenium is so unreliable about AJAX, to ensure that the action is performed, we do some brute forcing. PROVIDE EXAMPLE

So this generated code from Selenium:

sel.click("addLink")
for i in range(60):
    try:
        if "Session" == sel.get_text("link=Session"): break
    except: pass
    time.sleep(1)
else: self.fail("time out")

Becomes:

sel.click("addLink")
for i in range(60):
    try:
        if "Session" == sel.get_text("link=Session"): break
        else: sel.click("addLink")
    except: pass
    time.sleep(1)
else: self.fail("time out")

Finally, a test might work at several times and suddenly fail in the next run because of the reasons mentioned earlier. Running a test several times is a good indication to say if it is reliable or not. Use the script, it will run the test 20 times and see if it fails. TODO: Where to put the script?

2.c. Naming conventions

Files are organized by topic. For example, if you want to test your new widget for the timetable, the test file has to be in the folder timetable. Overall tests are in folder general.

Names have to be explicit and conformed to nosetests conventions. For example: moveWidget_test.py.

2.d. Running tests

python setup.py test --functional

This will generate a report in indicop/report/pyfunctional.txt.

Or you can user the --specify flag as previously described for unit tests.

2.e. Summary

Watch this tutorial video TODO

3. Source analysis

3.a. Program to install

The only tool needed is

Pylint and its associated libraries
Python Abstract Syntax Tree New Generation and
Logilab common.
pylint needs to be in your PATH!

In order to install all this and have pylint in your PATH, you can simply do:

sudo apt-get pylint

3.b. Running analysis

python setup.py test --pylint

This will generate a report in indicop/report/pylint.txt.

If you're using Eclipse, I strongly suggest to configure PyDev following this tutorial.

3.c. Configuration file

You can have a look at the configuration file to see the conventions for maximum number of arguments, maximum length of line, etc...
Naming rules are done according to PEP8 (extended with camelcase and numbers).
Useful handouts.

Type checker is disabled, because pylint gets stuck when trying infer types. But these kind of errors should be caught by unit tests.

4. Javascript Unit tests

4.a. Program to install

We are using Google's js-test-driver. It allows to test javascript files on different browsers at the same time.

Great Video Tutorial to get a general idea of the thing.

Eclipse plugin: http://code.google.com/p/js-test-driver/wiki/UsingTheEclipsePlugin

For the code coverage, install LCOV to generate nice HTML files. You can install LCOV as a linux package:

sudo apt-get install lcov

java and genhtml have to be in your PATH. genhtml is an executable part of the lcov package.

4.b. Writing tests

First, before writing any line of code, make sure that your test file is in the tests folder (indicop/javascript/unit/tests/).

The structure of a js test file first declares the testcase, it is an identifier to run specific test cases, then a set up function, the actual tests and finally, if needed, a tear down function.

MoveTest = TestCase("MoveTest");

MoveTest.prototype.setUp = function() {
    //your set up stuff
};

MoveTest.prototype.testUpdateEntry = function() {
    //your tests
    assertEquals("MSG", expectedValue, actualValue);
};

Objects mocking and variables setting typically take place in setUp. See example file Timetable/Base?.js to see how to do objects and functions mocking. And here are the functions you need to write tests small tutorial

Javascript unit tests are pretty easy to write, but it can get quickly messy, so do not forget to comment well your tests to keep them maintainable!

4.c. Running tests

python setup.py test --jsunit

This will generate a report in indicop/report/jsunit.txt.

To run a specific test:

python setup.py test --jsspecify=TestCase.testname

The output will be displayed directly in the console.

4.d. Code coverage

To activate code coverage, simply add the flag --jscoverage:

python setup.py test --jsunit --jscoverage

Html files are going to be generated automatically, open file indicop/report/jscoverage/index.html.

5. Javascript source analysis

5.a. Program to install

Rhino is needed to run the javascript source analysis.
rhino needs to be in your PATH.

You can install Rhino simply on Linux as a package:

sudo apt-get install rhino

5.b. Running scan

python setup.py test --jslint

This will generate a report in indicop/report/jslint.txt.

Plugins folders are scanned, but also javascript files in indico/htdocs/js/indico. Some folders are blacklisted, typically the folder pack. If you need to blacklist a folder, append the name of the folder in the file indicop/Indicop.py, class Jsunit.

6. Selenium Grid

Selenium Grid allows you to run your functional tests on all sort of browsers and OS. If you have the a Selenium Grid set up, the only thing you need to do to run your tests is to set your configuration file tests.conf.

6.a. Configuration

A file configuration should contain the following:

#------------------------------------------------------------------------------
# Selenium Grid
#------------------------------------------------------------------------------
HubURL         = "macuds04.cern.ch"
HubPort        = 4444
HubEnv         = ['Firefox on Windows',
                  'IE on Windows', 
                  'Safari on OS X',
                  'Firefox on Linux']

The use HubURL and HubPort are straightforward. HubEnv contains the list of browsers and OS on which you want to run your functional tests. A list of the available environments can be found here http://macuds04.cern.ch:4444/console. At CERN, our hub is on a iMac, which also has 2 RCs (Firefox and Safari). The rest of the RCs are instantiated from VirtualBoxes on the iMac.

6.b. Running tests

python setup.py test --grid

This will generate a report in indicop/report/pygrid.txt. Now lean back and enjoy a cup of tea! But most important, do NOT press CTRL-C! This will jamm the hub and it will be unusable unless we restart it.

7. Databases Fake vs Production

The idea is to run the tests on the fake databases and temporary folders, so tests won't pollute anything with ouputs.

TODO

8. TODO

Bitten for trac Performance: Profiler using selenium and python: http://code.google.com/p/selenium-profiler/

Download in other formats:

Plain Text