User:ErsatzCulture/ExperimentalTestingFramework

Background

Important: These tests have at-time-of-writing only been tested against Python 2.7.15!

This isn't the approved/official version of Python for ISFDB, which means that:

• The tests might not work on Python 2.5 (the current "official version IIRC) - e.g. if the test code uses "except SomeError as err" which is a newer syntax form.
• Python version inconsistencies might mean the values I see from Python 2.7 are different from those in Python 2.5, which could mean that tests pass against one version of Python but not another.

I don't think either of these should be the case - especially the second one - but be warned...

... MUCH MORE TO ADD HERE ...

Prerequisites

Python libraries

The following Python libraries are required. Some of these may be in stdlib (I haven't checked), but all are pretty common and are probably available as packages in your Linux distro as well as from PyPI. I've not checked what the exact package names might be, but they should be fairly discoverable.

• nose aka nosetests - a test runner. (IIRC there are other Python test runners available, which should be equally usable - however I have no experience of them.)
• BeautifulSoup aka bs4 - (X)HTML parsing

Configured MySQL/MariaDB database

I have written a script that takes an ISFDB database backup and installs it into a running MySQL/MariaDB instance, including setting up both database user accounts and ISFDB application accounts. It can be downloaded from my GitHub repo of ISFDB related tools. It should be invoked by a *nix user with root or mysql permissions like this:

   ./setup_testing_database.sh -r /proj/3rdparty/isfdb-code-svn -u {MYSQL_ROOT_USER} -p {MYSQL_ROOT_PASSWORD} -a   /mnt/data2019/_isfdb_/cygdrive.20191019/c/ISFDB/Backups/backup-MySQL-55-2019-10-19

The -r argument specificies where your SVN checkout of the ISFDB website code - this is needed for the scripts/create_user.py script.

-u and -p are the username and password of your MySQL/MariaDB root/admin user

-a indicates to create the application/site user accounts - these are "TestEditor" and "TestModerator", with passwords the same as the username

The final argument is the location of the (unzipped) MySQL backup file.

The script has a couple of other options, but the ones above are all that are needed currently.

All the tests in this initial experimental phase use the 2019-10-19 backup, which was just an arbitary selection based on the most recent available backup when I started work on this. It's likely that any other fairly recent backup would also work with these tests, but keeping to a specific known backup avoids tests failing because the records have been updated as part of regular use of the real site. It's also preferable to run the tests against a separate database from your "real" development database, for similar reasons.

Installing the tests

There is a 12k tarball downloadable from my personal site. It should be unpacked in the /mod/ subdirectory of your ISFDB, creating a /tests/ subdirectory that should look like this:

   mod $ ls -l /proj/3rdparty/isfdb-code-svn/mod/tests
   total 64
   -rw-rw-r--. 1 john users     0 Oct 23 23:39 __init__.py
   -rw-rw-r--. 1 john users   453 Oct 31 14:20 authorupdate.xml
   -rw-rw-r--. 1 john users   781 Oct 31 14:20 newpub.xml
   -rw-rw-r--. 1 john users   363 Oct 28 15:11 pubupdate.xml
   -rw-rw-r--. 1 john users  2782 Oct 31 22:43 test_av_update.py
   -rw-rw-r--. 1 john users 18665 Oct 31 22:30 test_helpers.py
   -rw-rw-r--. 1 john users  3133 Oct 31 22:30 test_pv_new.py
   -rw-rw-r--. 1 john users  2221 Oct 31 23:20 test_pv_update.py
   -rw-rw-r--. 1 john users  4439 Oct 31 22:29 test_tv_merge.py
   -rw-rw-r--. 1 john users  3172 Oct 31 22:45 test_tv_update.py
   -rw-rw-r--. 1 john users   492 Oct 31 20:12 titlemerge.xml
   -rw-rw-r--. 1 john users   284 Oct 28 15:12 titleupdate.xml

It's recommended that you create a stub __init__.py in the /mod/ directory e.g. by doing

  touch {isfdb-checkout-dir}/mod/__init__.py

This isn't strictly necessary, but it's a helpful convenience as it means the test runner can discover tests from the root directory of your ISFDB checkout - without it, you have to run the test runner from the /mod/ subdirectory.

Running the tests

Using nose - specifically the Python 2.7 version that ships with my Fedora distro - you should get something like the following:

   isfdb-code-svn $ nosetests-2.7 
   .........................
   ----------------------------------------------------------------------
   Ran 25 tests in 2.939s
   
   OK

Pass in a --verbose argument to see details of the individual tests being run.

Note that the tests create a subdirectory in /tmp/ that is used for dynamically created configuration/environment data:

   ls -l /tmp/isfdb_2019_10_19_testing/localdefs.py
   -rw-rw-r--. 1 john john 456 Oct 31 23:55 /tmp/isfdb_2019_10_19_testing/localdefs.py

Currently this doesn't get cleaned up after the tests are run, but as it uses less than 1k, this seems a reasonably forgivable crime for the time being.

How the tests works

The test runner discovers the tests through some means probably better documented elsewhere on the web, but basically it trawls for directories and Python files with "test" in their names, and runs the methods in classes that inherit from unittest.TestCase.

The specific details for these ISFDB tests are:

• Five of the moderator CGI scripts get tested - these were a fairly random selection of the scripts that would need to be updated for FR1305, and quite possibly aren't a particularly good or representative sample. There's one test_xx_verb.py file for each CGI script in the /tests/ subdirectory.
• For each CGI script, five tests are run. 4 of the tests are generic and perfunctory permissions/validation tests e.g. verifying that a non-logged user or non-moderator cannot access the page.
• Each TestCase does a "golden path" test, setting up an entry in the submissions table using an appropriate XML "fixture". This calls the CGI script, parses the resultant HTML output using BeautifulSoup, and verifies that the page contains the expected elements/text etc.
• There is a library of helper functions and classes in the /tests/ subdirectory that contains some generic functionality e.g. running the CGI script against our test database (rather than the regular one you might use for development); turning the HTML tables into data structures more amenable to comparisons, etc. This helper code is much more complicated and fiddly than the actual tests, which are fundamentally a bunch of asserts comparing the returned content with the expected values.

In and of themselves, these initial experimental tests aren't particularly interesting or useful - the CGI scripts being tested are fairly basic moderator approval pages, and as I hadn't used these pages before writing tests for them, it's quite possible that I've missed some of the complexities or subtleties that might be usefully tested against.

Rather, these are just an initial toe-in-the-water to get something up-and-running that might prove the basis for wider and more useful test coverage of the ISFDB code.

Further technical details

This section mainly concerns the code in mod/tests/test_helpers.py.

The structure of the code being tested means a few hoops have to be jumped through in order to run the underlying code and get its output. Ideally the tests would just call the function being tested and get the returned value/content e.g.

   class TestSomeClass(unittest.TestCase):
   
       def test_some_functionality(self):
           ret = the_function_being_tested("foo", "bar")
           self.assertEqual(123, ret)
           ... etc ...

Unfortunately:

• All the code I've looked at so far is run from "main", and I don't know if/how that can be accessed as a function. (It wouldn't surprise me if there's some Python metamagic to do that, I just don't know what it is.)
• The CGI scripts output to stdout using print statements, rather than returning text content.
• Use of global variables and 'from somemodule import *' makes it harder/impossible to inject dependencies such as connections to a test database (rather than the "real" one).

As such, the current tests all execute the CGI scripts in a separate process, capturing the output to stdout, with further hackery to catch error cases that exit with non-zero status. This in itself isn't too bad, but having the code running in a separate process means that my initial idea to run tests against an InnoDB database, and not commit at the end (thus automagically rolling back any updates) isn't feasible. Instead, the code has to be careful that any data inserted specifically for the test to run has to be manually undone in the tearDown method.

The use of a test database is done in an (IMHO) hacky way, by dynamically creating a localdefs.py file, and running the CGI script with a PYTHONPATH that causes that file to be picked up ahead of the usual one.

The database definitions are controlled by mixin classes, currently there's just the one: ISFDB_2019_10_19_Mixin. If there's some schema change in the future, then any tests that need it would have to setup a similar mixin for a newer version of the database.

(This use of backups would I'm sure horrify some testing purists, who'd prefer proper fixtures, but IMHO that's too much of a leap at this point in time.)

The code that parses the HTML using BeautifulSoup is pretty mundane. Given my lack of familiarity with the pages being tested, I'm sure there are things that could be done better if I had more familiarity with the code being tested.