Difference between revisions of "User:Alvonruff"

From ISFDB
Jump to navigation Jump to search
 
(20 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
Founder of the ISFDB.  
 
Founder of the ISFDB.  
  
−
== Python3 Notes ==
+
== Large Projects ==
 +
* 2022: Update LAMP stack to latest version (except Python) and enable HTTPS. [https://isfdb.org/wiki/index.php/User:Alvonruff#System_Upgrade_Notes DONE]
 +
* 2023: Update site to Python3. [[User/Alvonruff/Python3_Conversion]]
 +
* 2024: Update charset to UTF-8.
  
−
The primary difficulty with a python3 conversion project is trying to avoid a massive rewrite of the website, and then checkin all those changes with a single big bang integration. That said, the primary function of the scripts is to read data from MySQL, and then organize and print that information to the browser. But the two things that change with python3 is the MySQL connector (which requires a rewrite of all code interfacing with MySQL), and the way print statements work (which requires a rewrite of all code outputting information). So the first goal is to find a way for the ISFDB to exist simultaneously in Python2 and Python3 format. General outline of steps to move to Python3:
+
== Python3 Status Trackers ==
−
 
+
* [[User:Alvonruff/Debugging_Remarks|Debugging Remarks]]
−
# Python3 does not tolerate mixed tabs and spaces. There needs to be a project to convert the tabs in all files to 8 spaces. This change works on either Python2 or Python3
+
* Per-File Status: [[User:Alvonruff/Python3_Files|Python3 File Status]]
−
# Introduce a MySQL connector class, which hides the differences between MySQLdb (which only works on Python2) and mysql.connector (which only works on Python3).
+
* ISFDB2 Status: [[User:Alvonruff/Python3_ISFDB2|ISFDB2 Per-Script Status]]
−
# Convert all MySQL code to use the new connector class. This is a large project in itself, with details listed [[User:Alvonruff/MySQLdb_Conversion|here]].
+
* Mod script test plans and status: [[User:Alvonruff/Python3 ModTests]]
−
# [[User:Alvonruff/Python2.7_Futurize|futurize]] all print statements. This can be done by running futurize, and then keeping only the print() syntax changes. These will run fine under python2.
+
* Formal Tests: [[User:Alvonruff/ISFDB_Tests|Formal ISFDB Test Status]]
−
# Perform the [[User/Alvonruff/2to3|Python3 Conversion]], and follow up on the numerous porting issues.
 
−
# Update all character sets. Final procedure still TBD.
 
−
# Change the default charset in MySQL
 
−
# Repair strings which have URL encodings in MySQL
 
−
 
 
−
=== Remarks on Debugging the Current Code Base ===
 
−
There are a number of issues slowing down the porting effort, which makes the debugging process slow.
 
−
 
 
−
* '''SESSION Arguments'''. Since we've introduced the SESSION variable, many ISFDB scripts are no longer executable from the command line. The Session class pulls the arguments from the environment variable QUERY_STRING, which is not set when executing from the command line. When command line execution is not possible, then the new script must be installed, and if a simple syntax error appears within the file, then the browser simply returns "Internal Server Error", with no clue as to where the issue is. These types of issues are easily observable when running from the command line. To re-enable command line execution, I added the following to the end of ParseParameters in the Session class:
 
−
 
 
−
    # Allow for command line invocation
 
−
    if (cgi_path == None) and (self.query_string == None):
 
−
        num_args = len(sys.argv)
 
−
        for i in range(1, num_args, 1):
 
−
            self.parameters.append(sys.argv[i])
 
−
 
 
−
* '''Try/Except Usage'''. We have a tendency to use try/except to cover many possible potential issues within a large code block. Here's an actual example from se.py, which contains one error when trying to run under Python3:
 
−
 
 
−
        try:
 
−
                type = form['type'].value
 
−
                # Save the double-quote-escaped version of the original search value
 
−
                # to be re-displayed in the search box
 
−
                search_value = form['arg'].value.replace('"','"')
 
−
                # Replace asterisks with % to facilitate wild cards
 
−
                arg = str.replace(normalizeInput(form['arg'].value), '*', '%')
 
−
                # Double escape backslashes, which is required by the SQL syntax
 
−
                arg = string.replace(arg, '\\', '\\\\')
 
−
                user = User()
 
−
                user.load()
 
−
                if not user.keep_spaces_in_searches:
 
−
                        arg = str.strip(arg)
 
−
                if not arg:
 
−
                        raise
 
−
        except:
 
−
                PrintHeader("ISFDB Search Error")
 
−
                PrintNavbar('search', %'%', 0, 'se.cgi', '')
 
−
                print("No search value specified")
 
−
                PrintTrailer('search', '', 0)
 
−
                sys.exit(0)
 
−
 
 
−
: When this runs, all we see is "ISFDB Search Error, No search value specified", with no clue as to which clause might have caused the error. The typical approach then is to copy all of the try code, copy it to a position above the try statement, and then reformat the lines. Rather than getting into a philosophical debate about the overusage of try/except, we can make these blocks more debuggable. For instance, if we change the except clause above to:
 
−
 
 
−
    import traceback
 
−
    except Exception as e:
 
−
        e = traceback.format_exc()
 
−
        PrintHeader("ISFDB Search Error")
 
−
        PrintNavbar('search', '', 0, 'se.cgi', '')
 
−
        print("No search value specified")
 
−
        print('Error: ', e)
 
−
        PrintTrailer('search', '', 0)
 
−
        sys.exit(0)
 
−
 
 
−
: We now see the following error message:
 
−
 
 
−
    Error: Traceback (most recent call last): File "/var/www/cgi-bin/se.cgi", line 263, in arg = string.replace(arg, '\\', '\\\\') AttributeError: module 'string' has no attribute 'replace'
 
−
 
 
−
: Showing that we forgot to replace 'string' with 'str'.
 
−
 
 
−
* '''SQL Debugging'''. As documented elsewhere, there is an issue with mysql.connector in extracting DATE values from MySQL when the month or day is zero. We have an SQL syntax fix for that, but there are many ''hundreds'' of SQL statements in the ISFDB, so only a small percentage have been addressed so far. When debugging a script (adv_search_results.py comes to mind, since that is my current problem area), one simply sees the following error:
 
−
 
 
−
    TypeError: must be str, not datetime.date
 
−
        args = ('must be str, not datetime.date',)
 
−
        with_traceback = <built-in method with_traceback of TypeError object>
 
−
 
 
−
: So we know where the final problem occurred, but not which SQL method needs to be altered. Sometimes it is trivial to find, but on other occasions it requires looking through library.py, common.py, and some number of *Class.py files, which takes a fair amount of time. Additionally, I would like to make an set of SQL unit tests, but don't know what a typical valid set of arguments look like.
 
−
 
 
−
: As such, I've added an SQL logging feature to the Session class, which can be enabled in SQLparsing.py by setting SQLlogging to 1. This outputs a bulleted list of the SQL function calls that were made by a particular script, outputting them in a new section added by PrintTrailer() [[User:Alvonruff|Alvonruff]] ([[User talk:Alvonruff|talk]]) 14:04, 8 May 2023 (EDT)
 
−
 
 
−
=== Status Trackers ===
 
−
* Per-File Status: [[User:Alvonruff/Python3_Files]]
 
−
* ISFDB2 Status: [[User:Alvonruff/Python3_ISFDB2]]
 
  
 
== System Upgrade Notes ==
 
== System Upgrade Notes ==
Line 98: Line 30:
 
* [[User:Alvonruff/Notes on MySQLdb]]
 
* [[User:Alvonruff/Notes on MySQLdb]]
 
* [[User:Alvonruff/Test_Pages]]
 
* [[User:Alvonruff/Test_Pages]]
 +
 +
== Articles Concerning Lee Mandelo ==
 +
* [[User:Alvonruff/A_Post-Mortem_on_the_Lee_Mandelo_Name_Change]]
 +
* [[User:Alvonruff/An_Open_Letter_To_The_SF_Community]]
  
 
== Obituary Sources ==
 
== Obituary Sources ==

Latest revision as of 11:02, 16 September 2023

Founder of the ISFDB.

Large Projects

Python3 Status Trackers

System Upgrade Notes

Details on how to bring up a LAMP stack (on two different OSs), and how to setup https:

User:Alvonruff/Test Page

Other Loose Notes

Articles Concerning Lee Mandelo

Obituary Sources

Reading List

Retrieved from "https://isfdb.org/wiki/index.php?title=User:Alvonruff&oldid=671004"