User/Alvonruff/Python3 Conversion

The primary difficulty with a python3 conversion project is trying to avoid a massive rewrite of the website, and then checkin all those changes with a single big bang integration. That said, the primary function of the scripts is to read data from MySQL, and then organize and print that information to the browser. But the two things that change with python3 is the MySQL connector (which requires a rewrite of all code interfacing with MySQL), and the way print statements work (which requires a rewrite of all code outputting information). So the first goal is to find a way for the ISFDB to exist simultaneously in Python2 and Python3 format. General outline of steps to move to Python3:

• Perform a blunt-force port to Python3 at isfdb2.org. This includes:
  • Introduce a python selection mechanism in the build system.
  • Integrate the formal tests in the new tests directory.
• Perform step-by-step integration onto the production server:
  • Fix the mixed tab/space issues.
  • Change usage of string objects to str.
  • Integrate the python selection mechanism in the build system
  • Introduce the connector class into SQLparsing, which hides the details of which MySQL connector is in use.
  • Modify all code to use the new connector class.
  • Make modifications to use python3 print syntax via futurize.
  • Make the use of encode/decode selectable via the python selection mechanism.
  • Make a has_key convenience function that can select the supported routines based on the python selection mechanism.

The "Quick" Port

The first step is to just do whatever it takes to get the scripts to run. This includes hand-editing files, as well as running through futurize. Futurize does (occasionally) make mistakes. The two most egregious so far are:

• Making spurious insertions of str() casts to things that should remain int.
• Treating the string 'next' as evidence of iterator use, and converting most (but not all!) of the appearances of 'next' to '__next__'.

As such, we can't simply check this port in when done and call it a day. The integration will need to proceed in a more disciplined manner. The order of the quick port will follow these directories:

• common - DONE
• biblio - DONE
• edit - In progress. (not verified on python2 yet)
• mod - In progress.
• rest
• nightly
• scripts

Since there are a few hundred files, this phase will likely complete sometime in July.

Integration

The following section tracks the integration progress of the python3 conversion. The order is intended to minimize the potential number of issues on the production server.

Tabs/Spaces

These changes can be automated by setting do_tabs=1 in fixup.py, and issuing: ./fixup.py <target file>

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

Futurize/print()

In order to work on both python2 and python3, the following line needs to be the first line in the file:

   from __future__ import print_function

Unless the file is an executable one, in which case the #! line must be first to satisfy the operating system. Those files should look like this:

   #!_PYTHONLOC
   from __future__ import print_function

These changes can be automated by using: 2to3 -f print <target file> (except for the import line)

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

has_key() Transition

These changes can be automated by using: 2to3 -f has_key <target file>

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

string to str

These changes can be automated by setting do_str=1 in fixup.py, and issuing: ./fixup.py <target file>

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

New Connector Class

These changes can be partially automated by setting do_mysqldb=1 in fixup.py, and issuing: ./fixup.py <target file>

The are a few things that require human intervention are:

• fixup will replace db.store_result() with REMOVE_THIS_LINE. This is a reminder to do the next bullets.
• Any function that refers to the CNX variable needs to have a CNX = MYSQL_CONNECTOR() line added.
• The new connector makes a distinction between fetch one results (FETCHONE) and fetch many results (FETCHMANY). Fixup only issues FETCHMANY, so if there is no while loop it probably needs to be a FETCHONE.
• Nested SQL queries are beyond the scope of fixup, which can only perform lexical changes. If two SQL queries are going on simultaneously, then the second SQL instance will need a fresh connector instance by issuing something like CNX2 = MYSQL_CONNECTOR(), and patching up the subsequent DB calls to use the second connector instance.

Directories:

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

encode/decode Changes

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

FieldStorage

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

Leftovers

• common -
• biblio -
• edit -
• mod -
• rest -
• nightly -
• scripts -

Python Selection Mechanism

There are two areas where we need to know the python version: during the installation of the scripts, and more generally during runtime.

Installation Mechanism

Currently the python location is hardcoded in the various local.mk files, under the cgi rule:

   local/%.cgi:    %.py
       python install.py $* local /usr/bin/python

First, the cgi rule in each local.mk is modified to pick up a version string. The version is string is found in .pythonver in the top-level directory:

   VERSION = $(shell cat ../.pythonver)

   local/%.cgi:    %.py
       python install.py $* local $(VERSION)

The VERSION variable must be assigned before the all target in the makefile. Next the known locations for the flavors of python are in the top-level Makefile. The defaults should work for most Linux installations:

   PYTHON2 = /usr/bin/python
   PYTHON3 = /usr/bin/python3

Finally two new targets are added to the top-level Makefile, such that the developer can deliberately change the target python version:

   python2:
       echo $(PYTHON2) > .pythonver
       echo "Now using Python2"

   python3:
       echo $(PYTHON3) > .pythonver
       echo "Now using Python3"

To change to python3, the developer would execute:

   make clean
   make python3

To change back to python2, the developer would execute:

   make clean
   make python2

Runtime Mechanism

There are code differences between python2 and python3 that cannot be handled with a futurize import (which connector to use, for instance). To handle runtime checks, the following new variable is placed in common/localdefs.py:

   PYTHONVER       = "python3"

This variable is automatically updated by common/setver.py, which is passed an integer number of the python version (this can be changed later if insufficient). The two new targets in the top-level Makefile are then changed to:

   python2:
       echo $(PYTHON2) > .pythonver
       cd common && python setver.py 2;
       echo "Now using Python2"

   python3:
       echo $(PYTHON3) > .pythonver
       cd common && python setver.py 3;
       echo "Now using Python3"

Now runtime checks can be performed with:

   if PYTHONVER == "python2":
       do_something()
   elif PYTHONVER == "python3":
       do_something_else()