Difference between revisions of "User:Alvonruff/Debugging Remarks"

From ISFDB
Jump to navigation Jump to search
Line 64: Line 64:
 
:* SQLlog(arg) - Logs an SQL function call.
 
:* SQLlog(arg) - Logs an SQL function call.
 
:* SQLoutputLog() - Will output the log stored in the SESSION class. Each line is prepended with an HTML LI to print a bullet list in the browser.
 
:* SQLoutputLog() - Will output the log stored in the SESSION class. Each line is prepended with an HTML LI to print a bullet list in the browser.
 
: This can also find aberrant behavior. For instance, something peculiar is going on in  adv_search_results, causing it to reissue all previous SQL queries when moving to each next author in the list:
 
 
    AdvancedSearchResults::search, query=select distinct authors.* from authors where ((authors.author_lastname like 'as%') ) order by author_lastname, author_canonical limit 0,101
 
    SQLauthorIsPseudo, au_id=159305
 
    SQLUpdateQueries
 
    SQLLoadAllLanguages
 
    SQLgetDatabaseStatus
 
    SQLgetSchemaVersion
 
    AdvancedSearchResults::search, query=select distinct authors.* from authors where ((authors.author_lastname like 'as%') ) order by author_lastname, author_canonical limit 0,101
 
    SQLauthorIsPseudo, au_id=159305
 
    SQLauthorIsPseudo, au_id=304174
 
    SQLUpdateQueries
 
    SQLLoadAllLanguages
 
    SQLgetDatabaseStatus
 
    SQLgetSchemaVersion
 
    AdvancedSearchResults::search, query=select distinct authors.* from authors where ((authors.author_lastname like 'as%') ) order by author_lastname, author_canonical limit 0,101
 
    SQLauthorIsPseudo, au_id=159305
 
    SQLauthorIsPseudo, au_id=304174
 
    SQLauthorIsPseudo, au_id=255055
 
    SQLUpdateQueries
 
    SQLLoadAllLanguages
 
    SQLgetDatabaseStatus
 
    SQLgetSchemaVersion
 
    AdvancedSearchResults::search, query=select distinct authors.* from authors where ((authors.author_lastname like 'as%') ) order by author_lastname, author_canonical limit 0,101
 
    SQLauthorIsPseudo, au_id=159305
 
    SQLauthorIsPseudo, au_id=304174
 
    SQLauthorIsPseudo, au_id=255055
 
    SQLauthorIsPseudo, au_id=245794
 
    ...and so on...
 

Revision as of 21:59, 8 May 2023

Remarks on Debugging the Current Code Base

There are a number of issues slowing down the porting effort, which makes the debugging process slow.

  • SESSION Arguments. Since we've introduced the SESSION variable, many ISFDB scripts are no longer executable from the command line. The Session class pulls the arguments from the environment variable QUERY_STRING, which is not set when executing from the command line. When command line execution is not possible, then the new script must be installed, and if a simple syntax error appears within the file, then the browser simply returns "Internal Server Error", with no clue as to where the issue is. These types of issues are easily observable when running from the command line. To re-enable command line execution, I added the following to the end of ParseParameters in the Session class:
   # Allow for command line invocation
   if (cgi_path == None) and (self.query_string == None):
       num_args = len(sys.argv)
       for i in range(1, num_args, 1):
           self.parameters.append(sys.argv[i])
  • Try/Except Usage. We have a tendency to use try/except to cover many possible potential issues within a large code block. Here's an actual example from se.py, which contains one error when trying to run under Python3:
       try:
               type = form['type'].value
               # Save the double-quote-escaped version of the original search value
               # to be re-displayed in the search box
               search_value = form['arg'].value.replace('"','"')
               # Replace asterisks with % to facilitate wild cards
               arg = str.replace(normalizeInput(form['arg'].value), '*', '%')
               # Double escape backslashes, which is required by the SQL syntax
               arg = string.replace(arg, '\\', '\\\\')
               user = User()
               user.load()
               if not user.keep_spaces_in_searches:
                       arg = str.strip(arg)
               if not arg:
                       raise
       except:
               PrintHeader("ISFDB Search Error")
               PrintNavbar('search', %'%', 0, 'se.cgi', )
               print("No search value specified")
               PrintTrailer('search', , 0)
               sys.exit(0)
When this runs, all we see is "ISFDB Search Error, No search value specified", with no clue as to which clause might have caused the error. The typical approach then is to copy all of the try code, copy it to a position above the try statement, and then reformat the lines. Rather than getting into a philosophical debate about the overusage of try/except, we can make these blocks more debuggable. For instance, if we change the except clause above to:
   import traceback
   except Exception as e:
       e = traceback.format_exc()
       PrintHeader("ISFDB Search Error")
       PrintNavbar('search', , 0, 'se.cgi', )
       print("No search value specified")
       print('Error: ', e)
       PrintTrailer('search', , 0)
       sys.exit(0)
We now see the following error message:
   Error: Traceback (most recent call last): File "/var/www/cgi-bin/se.cgi", line 263, in arg = string.replace(arg, '\\', '\\\\') AttributeError: module 'string' has no attribute 'replace'
Showing that we forgot to replace 'string' with 'str'.
  • SQL Debugging. As documented elsewhere, there is an issue with mysql.connector in extracting DATE values from MySQL when the month or day is zero. We have an SQL syntax fix for that, but there are many hundreds of SQL statements in the ISFDB, so only a small percentage have been addressed so far. When debugging a script (adv_search_results.py comes to mind, since that is my current problem area), one simply sees the following error:
   TypeError: must be str, not datetime.date
       args = ('must be str, not datetime.date',)
       with_traceback = <built-in method with_traceback of TypeError object>
So we know where the final problem occurred, but not which SQL method needs to be altered. Sometimes it is trivial to find, but on other occasions it requires looking through library.py, common.py, and some number of *Class.py files, which takes a fair amount of time. Additionally, I would like to make a set of SQL unit tests, but don't know what typical valid arguments look like.
As such, I've added an SQL logging feature to the Session class, which can be enabled in SQLparsing.py by setting SQLlogging to 1. This records a running log in SESSION, and can output a bulleted list of the SQL function calls that were made by a particular script, outputting them in a new section added by PrintTrailer(). For scripts that are generating faults, a call to SQLoutputLog() can be temporarily inserted just above the fault point in the code. Relevant variables/methods:
  • SQLlogging (SQLparsing.py) - Turns the feature on/off.
  • SQLlog(arg) - Logs an SQL function call.
  • SQLoutputLog() - Will output the log stored in the SESSION class. Each line is prepended with an HTML LI to print a bullet list in the browser.