User/Alvonruff/Python3 Conversion

The primary difficulty with a python3 conversion project is trying to avoid a massive rewrite of the website, and then checkin all those changes with a single big bang integration. That said, the primary function of the scripts is to read data from MySQL, and then organize and print that information to the browser. But the two things that change with python3 is the MySQL connector (which requires a rewrite of all code interfacing with MySQL), and the way print statements work (which requires a rewrite of all code outputting information). So the first goal is to find a way for the ISFDB to exist simultaneously in Python2 and Python3 format. General outline of steps to move to Python3:

• Perform a blunt-force port to Python3 at isfdb2.org. This includes:
  • Introduce a python selection mechanism in the build system.
  • Integrate the formal tests in the new tests directory.
• Perform step-by-step integration onto the production server:
  • Fix the mixed tab/space issues.
  • Change usage of string objects to str.
  • Integrate the python selection mechanism in the build system
  • Introduce the connector class into SQLparsing, which hides the details of which MySQL connector is in use.
  • Modify all code to use the new connector class.
  • Make modifications to use python3 print syntax via futurize.
  • Make the use of encode/decode selectable via the python selection mechanism.
  • Make a has_key convenience function that can select the supported routines based on the python selection mechanism.

The "Quick" Port

The first step is to just do whatever it takes to get the scripts to run. This includes hand-editing files, as well as running through futurize. Futurize does (occasionally) make mistakes. The two most egregious so far are:

• Making spurious insertions of str() casts to things that should remain int.
• Treating the string 'next' as evidence of iterator use, and converting most (but not all!) of the appearances of 'next' to '__next__'.

As such, we can't simply check this port in when done and call it a day. The integration will need to proceed in a more disciplined manner. The order of the quick port will follow these directories:

• common - DONE
• biblio - DONE
• edit - In progress.
• mod
• rest
• nightly
• scripts

Since there are a few hundred files, this phase will likely complete sometime in July.

Python Selection Mechanism

There are two areas where we need to know the python version: during the installation of the scripts, and more generally during runtime.

Installation Mechanism

Currently the python location is hardcoded in the various local.mk files, under the cgi rule:

   local/%.cgi:    %.py
       python install.py $* local /usr/bin/python

First, the cgi rule in each local.mk is modified to pick up a version string. The version is string is found in .pythonver in the top-level directory:

   VERSION = $(shell cat ../.pythonver)

   local/%.cgi:    %.py
       python install.py $* local $(VERSION)

The VERSION variable must be assigned before the all target in the makefile. Next the known locations for the flavors of python are in the top-level Makefile. The defaults should work for most Linux installations:

   PYTHON2 = /usr/bin/python
   PYTHON3 = /usr/bin/python3

Finally two new targets are added to the top-level Makefile, such that the developer can deliberately change the target python version:

   python2:
       echo $(PYTHON2) > .pythonver
       echo "Now using Python2"

   python3:
       echo $(PYTHON3) > .pythonver
       echo "Now using Python3"

To change to python3, the developer would execute:

   make clean
   make python3

To change back to python2, the developer would execute:

   make clean
   make python2

Runtime Mechanism

There are code differences between python2 and python3 that cannot be handled with a futurize import (which connector to use, for instance). To handle runtime checks, the following new variable is placed in common/localdefs.py:

   PYTHONVER       = "python3"

This variable is automatically updated by common/setver.py, which is passed an integer number of the python version (this can be changed later if insufficient). The two new targets in the top-level Makefile are then changed to:

   python2:
       echo $(PYTHON2) > .pythonver
       cd common && python setver.py 2;
       echo "Now using Python2"

   python3:
       echo $(PYTHON3) > .pythonver
       cd common && python setver.py 3;
       echo "Now using Python3"

Now runtime checks can be performed with:

   if PYTHONVER == "python2":
       do_something()
   elif PYTHONVER == "python3":
       do_something_else()