Background

Important: These tests have at-time-of-writing only been tested against Python 2.7.15!

This isn't the approved/official version of Python for ISFDB, which means that:

The tests might not work on Python 2.5 (the current "official version IIRC) - e.g. if the test code uses "except SomeError as err" which is a newer syntax form.
Python version inconsistencies might mean the values I see from Python 2.7 are different from those in Python 2.5, which could mean that tests pass against one version of Python but not another.

I don't think either of these should be the case - especially the second one - but be warned...

... MUCH MORE TO ADD HERE ...

Prerequisites

Python libraries

The following Python libraries are required. Some of these may be in stdlib (I haven't checked), but all are pretty common and are probably available as packages in your Linux distro as well as from PyPI. I've not checked what the exact package names might be, but they should be fairly discoverable.

nose aka nosetests - a test runner. (IIRC there are other Python test runners available, which should be equally usable - however I have no experience of them.)
BeautifulSoup aka bs4 - (X)HTML parsing

Configured MySQL/MariaDB database

I have written a script that takes an ISFDB database backup and installs it into a running MySQL/MariaDB instance, including setting up both database user accounts and ISFDB application accounts. It can be downloaded from my GitHub repo of ISFDB related tools. It should be invoked by a *nix user with root or mysql permissions like this:

   ./setup_testing_database.sh -r /proj/3rdparty/isfdb-code-svn -u {MYSQL_ROOT_USER} -p {MYSQL_ROOT_PASSWORD} -a   /mnt/data2019/_isfdb_/cygdrive.20191019/c/ISFDB/Backups/backup-MySQL-55-2019-10-19

The -r argument specificies where your SVN checkout of the ISFDB website code - this is needed for the scripts/create_user.py script.

-u and -p are the username and password of your MySQL/MariaDB root/admin user

-a indicates to create the application/site user accounts - these are "TestEditor" and "TestModerator", with passwords the same as the username

The final argument is the location of the (unzipped) MySQL backup file.

The script has a couple of other options, but the ones above are all that are needed currently.

All the tests in this initial experimental phase use the 2019-10-19 backup, which was just an arbitary selection based on the most recent available backup when I started work on this. It's likely that any other fairly recent backup would also work with these tests, but keeping to a specific known backup avoids tests failing because the records have been updated as part of regular use of the real site. It's also preferable to run the tests against a separate database from your "real" development database, for similar reasons.

Installing the tests

There is a 12k tarball downloadable from my personal site. It should be unpacked in the /mod/ subdirectory of your ISFDB, creating a /tests/ subdirectory that should look like this:

   mod $ ls -l /proj/3rdparty/isfdb-code-svn/mod/tests
   total 64
   -rw-rw-r--. 1 john users     0 Oct 23 23:39 __init__.py
   -rw-rw-r--. 1 john users   453 Oct 31 14:20 authorupdate.xml
   -rw-rw-r--. 1 john users   781 Oct 31 14:20 newpub.xml
   -rw-rw-r--. 1 john users   363 Oct 28 15:11 pubupdate.xml
   -rw-rw-r--. 1 john users  2782 Oct 31 22:43 test_av_update.py
   -rw-rw-r--. 1 john users 18665 Oct 31 22:30 test_helpers.py
   -rw-rw-r--. 1 john users  3133 Oct 31 22:30 test_pv_new.py
   -rw-rw-r--. 1 john users  2221 Oct 31 23:20 test_pv_update.py
   -rw-rw-r--. 1 john users  4439 Oct 31 22:29 test_tv_merge.py
   -rw-rw-r--. 1 john users  3172 Oct 31 22:45 test_tv_update.py
   -rw-rw-r--. 1 john users   492 Oct 31 20:12 titlemerge.xml
   -rw-rw-r--. 1 john users   284 Oct 28 15:12 titleupdate.xml

It's recommended that you create a stub __init__.py in the /mod/ directory e.g. by doing

  touch {isfdb-checkout-dir}/mod/__init__.py

This isn't strictly necessary, but it's a helpful convenience as it means the test runner can discover tests from the root directory of your ISFDB checkout - without it, you have to run the test runner from the /mod/ subdirectory.

Running the tests

Using nose - specifically the Python 2.7 version that ships with my Fedora distro - you should get something like the following:

   isfdb-code-svn $ nosetests-2.7 
   .........................
   ----------------------------------------------------------------------
   Ran 25 tests in 2.939s
   
   OK

Pass in a --verbose argument to see details of the individual tests being run.

Note that the tests create a subdirectory in /tmp/ that is used for dynamically created configuration/environment data:

   ls -l /tmp/isfdb_2019_10_19_testing/localdefs.py
   -rw-rw-r--. 1 john john 456 Oct 31 23:55 /tmp/isfdb_2019_10_19_testing/localdefs.py

Currently this doesn't get cleaned up after the tests are run, but as it uses less than 1k, this seems a reasonably forgivable crime for the time being.

How the tests works

The test runner discovers the tests through some means probably better documented elsewhere on the web, but basically it trawls for directories and Python files with "test" in their names, and runs the methods in classes that inherit from unittest.TestCase.

The specific details for these ISFDB tests are:

Five of the moderator CGI scripts get tested - these were a fairly random selection of the scripts that would need to be updated for FR1305, and quite possibly aren't a particularly good or representative sample. There's one test_xx_verb.py file for each CGI script in the /tests/ subdirectory.
For each CGI script, five tests are run. 4 of the tests are generic and perfunctory permissions/validation tests e.g. verifying that a non-logged user or non-moderator cannot access the page.
Each TestCase does a "golden path" test, setting up an entry in the submissions table using an appropriate XML "fixture". This calls the CGI script, parses the resultant HTML output using BeautifulSoup, and verifies that the page contains the expected elements/text etc.
There is a library of helper functions and classes in the /tests/ subdirectory that contains some generic functionality e.g. running the CGI script against our test database (rather than the regular one you might use for development); turning the HTML tables into data structures more amenable to comparisons, etc. This helper code is much more complicated and fiddly than the actual tests, which are fundamentally a bunch of asserts comparing the returned content with the expected values.

In and of themselves, these initial experimental tests aren't particularly interesting or useful - the CGI scripts being tested are fairly basic moderator approval pages, and as I hadn't used these pages before writing tests for them, it's quite possible that I've missed some of the complexities or subtleties that might be usefully tested against.

Rather, these are just an initial toe-in-the-water to get something up-and-running that might prove the basis for wider and more useful test coverage of the ISFDB code.

User:ErsatzCulture/ExperimentalTestingFramework

Contents