* This page is under construction *

0. Before starting

Tests are in folder Indicop and they are organized according to their type.

To run all the tests in the test suite:

python setup.py tests

or with coverage:

python setup.py tests --coverage --jscoverage

Then reports will be generated in folder indicop/report.

1. Unit tests

1.a. Program to install

We are using the framework ​Nosetests. You can follow this ​Installation Guide.

For code coverage, install ​Figleaf, which is based on coverage.py.

1.b. Writing unit tests

You can either the standard ​UnitTest package provided in python or you can code in Nosetests style:

#Note that the 2 following functions will be called for each tests, very handy if we need to run a single test
#So we do not need to specify a setup and teardown for each class
def setup_module():
    DBMgr.getInstance().startRequest()

def teardown_module():
    DBMgr.getInstance().abort()
    DBMgr.getInstance().endRequest()

class TestCategories():
    
    def testBasicAddAndRemoveConferences(self):
        [...]
        c1._addConference(conf1)
        assert (croot.getNumConferences()==0)
        [...]

1.c. Files convention

A test file should correspond to a source file. For example: MaKaC_tests/conference_test.py must only have tests for MaKaC/conference.py. Classes and functions names should also match.

Of course at some point testing many things in different files is required. Therefore use explicit file names, like test_move_widget.py. TODO: What about folders?

1.d. Running tests

To run all the unit tests:

python setup.py tests --unit

It will run every class or function which has the keyword Test or test in its name.

If you need to run only a specific test, use the flag --specify and use nosetests convention:

file.py:class.function

For example:

python setup.py tests --specify=Selenium_tests/create_delete_lecture_test.py:Selenium_test.test_working

Note that you can also specify a set of tests like that:

python setup.py tests --specify=Selenium_tests/create_delete_lecture_test.py

and also only a folder like that:

python setup.py tests --specify=Selenium_tests

Note that the output will be displayed directly in the console.

1.e. Enable code coverage

To enable the code coverage, add the option --coverage:

python setup.py tests --unit --coverage

It will output a report in html files, you can find them in indicop/coverage/html_report. Open the file index.html to see the summary of the coverage. These files are going to be overwritten at each run.

Note that class and function definitions are executed on import, which is why def blabla(self): is green; it's just the contents of the functions themselves that aren't executed.

2. Functional tests

• Note: Selenium is going to be merged with Google's webdriver (​http://selenium.googlecode.com/) *

2.a. Program to install

You need ​Firefox, ​XPather, ​Selenium IDE and RC, ​Twill and ​Selenium egg.

java needs to be in your PATH!

2.b. Writing tests

Think of Selenium as the dumbest bot possibly existing :)

Handy functions are located in the file seleniumTestCase.py. This first thing to do is to modify the function getRootUrl from this file and set it according to your local installation of Indico. A dummy user is automatically created from this Class to carry the creation of conferences and so.

Here the template of a Selenium's test file:

import time
from seleniumTestCase import SeleniumTestCase

class ExampleTest(SeleniumTestCase):
    def setUp(self):
        SeleniumTestCase.setUp(self)
    
    [...]

    def tearDown(self):
        SeleniumTestCase.tearDown(self)

if __name__ == "__main__":
    unittest.main()

Have a look at the file example_test.py to see how functional tests are written.

If you create a new conference, you might want this conference to be deleted automatically if a test fails. Add this line just after the creation of the new conference or meeting:

SeleniumTestCase.setConfID(self, sel.get_location())

Since it is very time consuming and painful to debug Selenium's generated code, tests should be small and test specific part of Indico. Of course, a few overall tests are needed.

Use Firefox's Selenium IDE to record your tests and set the output format to python, but be rigorous with the following points:

1. Selenium does not recognize Javascript auto-complete when a text is entered.
   Solution: Enter the whole word by hand.
2. Javascript Calendar does not work either.
   Solution: Enter the date by hand.
3. Do not make clicks or assertions on element containing IDs.
   Solution: Use XPath instead. Xpaths are easy to find with Firefox's plugin XPather.
4. When replaying, Selenium can get confused with buttons' names and can end up clicking on the wrong button.
   Solution: Once again, use XPath instead. Xpaths are easy to find with Firefox's plugin XPather.
5. Popups and AJAX requests are very bothersome to test with Selenium. The latter would not wait for a page or a tab to load if we do not explicitly tell to do so.
   Solution: Use Selenium waitForText function whenever a window pops up, a tab is loading or an AJAX request is sent.
   

And since Selenium is so unreliable about AJAX, to ensure that the action is performed, we do some brute forcing. PROVIDE EXAMPLE

So this generated code from Selenium:

sel.click("addLink")
for i in range(60):
    try:
        if "Session" == sel.get_text("link=Session"): break
    except: pass
    time.sleep(1)
else: self.fail("time out")

Becomes:

sel.click("addLink")
for i in range(60):
    try:
        if "Session" == sel.get_text("link=Session"): break
        else: sel.click("addLink")
    except: pass
    time.sleep(1)
else: self.fail("time out")

Finally, a test might work several times and fails once because of the reasons mentioned earlier. Running a test several time is a good indication to tell if it is reliable or not. Use the script, it will run the test 20 times and see if it fails TODO: Where to put the script?

2.c. Naming conventions

Files are organized by topic. For example, if you want to test your new widget for the timetable, the test file has to be in the folder timetable. Overall tests are in folder general.
Names have to be explicit and conformed to nosetests conventions. For example: test_move_widget.py.

2.d. Running tests

python setup.py tests --functional

Or you can user the --specify flag as previously described for unit tests, but also the --coverage flag.

2.e. Summary

Watch this tutorial video TODO

3. Source analysis

3.a. Program to install

The only tool needed is ​Pylint and its associated libraries ​Python Abstract Syntax Tree New Generation and ​Logilab common.

pylint needs to be in your PATH!

3.b. Running analysis

python setup.py tests --pylint

3.c. Configuration file

You can have a look at the configuration file to see the conventions for maximum number of arguments, maximum length of line, etc...
Naming rules are done according to ​PEP8 (extended with underscores and numbers).
Useful ​handouts.

Type checker is disable because pylint gets stuck when trying infer types.

3.d. Exceptions

TODO: write an upper-layer to add exceptions of unwanted notifications from pylint

5. Javascript Unit tests

5.a. Program to install

We are using Google's ​js-test-driver. It allows to test javascript files on different browsers at the same time.
Great ​Video Tutorial to get a general idea of the thing.
Eclipse plugin: ​http://code.google.com/p/js-test-driver/wiki/UsingTheEclipsePlugin
For the code coverage, download the plugin and install ​LCOV to generate nice HTML files.

java and genhtml have to be in your PATH.

5.b. Writing tests

First, before writing any line of code, make sure that your test file is included in the configuration files. At the bottom of indicop/javascript_tests/jsTestDriver.conf and indicop/javascript_tests/jsTestDriverNoCoverage.conf:

#test files
- tests/dryRun.js
- tests/Timetable/*.js

The structure of a js test file first declares the testcase, it is an identifier to run specific test cases, then a set up function, the actual tests and finally, if needed, a tear down function.

MoveTest = TestCase("MoveTest");

MoveTest.prototype.setUp = function() {
    //your set up stuff
};

MoveTest.prototype.testUpdateEntry = function() {
    //your tests
    assertEquals("MSG", expectedValue, actualValue);
};

Objects mocking and variables setting typically take place in setUp. See example file Timetable/Base?.js to see how to do objects and functions mocking.

5.c. Running tests

python setup.py tests --jsunit

To run a specific test:

python setup.py tests --jsspecify TestCase.testname

The output will be displayed directly in the console.

5.d. Code coverage

To activate code coverage, simply add the flag --jscoverage:

python setup.py tests --jsunit --jscoverage

Html files are going to be generated automatically, open file indicop/javascript_tests/coverage/index.html

6. Javascript source analysis

6.a. Program to install

​Rhino is needed to run the javascript source analysis.

rhino needs to be in your PATH.

6.b. Running scan

python setup.py tests --jslint

A report will be created in indicop/report/jsLint.txt

6.c. Adding exceptions

COMING SOUNE

7. TODO

Selenium Grid?

Bitten for trac

possibility to chose DB which to run tests on?

Performance: Profiler using selenium and python: ​http://code.google.com/p/selenium-profiler/