Development

ISFDB Discussion Pages and Noticeboards
Before posting to this page, consider whether one of the other discussion pages or noticeboards might suit your needs better. If you're looking for help remembering a book title, check out the resources in our FAQ. Please also see our Help pages.
Help desk Questions about doing a specific task, or how to correct information when the solution is not immediately obvious. • New post • Archives	Research Assistance Help with bibliographic projects. • New post • Archives	Rules and standards Discussions about the rules and standards, as well as questions about interpretation and application of those rules. • New post • Rules changelog • Archives	Community Portal General discussion about anything not covered by the more specialized noticeboards to the left. • New post • Archives	Moderator noticeboard Get the attention of moderators regarding submission questions.   • New post • Archives • Cancel submission
Roadmap: For the original discussion of Roadmap 2017 see this archived section. For the current implementation status, see What's New#Roadmap 2017.

Related resources:

• SourceForge: ISFDB Bibliographic Tools
  • repo viewer: source code
  • discussion
  • tickets
    • Bugs
    • Support Requests
    • Patches
    • Feature Requests
• Obsolete historic issues
  • ISFDB Bug List
  • ISFDB Feature List
• ISFDB:Proposed Design Changes‎
• ISFDB:Proposed Interface Changes
• Recent Patch list (archive)
• Fixer's Talk page (for Fixer-related issues)
• ISFDB Design Documentation (some parts are still being brought up-to-date)
• Database Schema
• ISFDB Downloads (contains links to the installation instructions)
• ISFDB Build Process (explanation of the build/install process)
• ISFDB Page Map — a list of which pages link to which other pages

This page serves as the hub for discussing ISFDB-related software development.

Developers and Testers

• MartyD — developer
• Kpulliam — developer (of meager ability) / Tester
• Roglo — developer (inactive since mid-2009)
• Ahasuerus — developer, tester and installer/administrator
• Alvonruff — currently unavailable, but has full development and administrative access
• Marc Kupper — has full administrative access and a partial development setup
• JesseW — developer
• Uzume — developer, tester, etc.

The following editors are currently in the process of setting up a local copy of the application:

• Swfritter

Software Development Process

At this time the development process works as follows:

• A Bug or Feature Request ("FR") is created in SourceForge. Anyone can do it, even "anonymous".
• FRs should be discussed on the Community Portal and approved by the community before the work begins. Obvious bugs can be fixed and submitted without a prior discussion.
• Once a developer decides to work on a Bug/FR, s/he identifies the scripts that are affected and lists the Bug/FR along with the scripts on the Development page under "Outstanding Changes". This helps avoid effort duplication.
• The developer makes changes on his or her development server and tests them. S/he then sends to the ISFDB administrator (Ahasuerus since 2009), who tests them on his local development server. In some cases the administrator may ask another developer to test the changes.
• If everything works fine, the administrator commits the changes to the SourceForge repository using the SVN "commit" command, e.g.:
  • svn commit -m "FR 1099 Add Last Activity date to 'Top Contributor/Moderator/etc' lists - Part 1"
• The administrator gets the latest revision number from the SourceForge repository:
  • svn info --show-item=revision -r HEAD
• The administrator creates a deployable tarball file from the modified source files, e.g:
  • tar -cvf ~/patches/3.tar biblio/toptaggers.py, where '3' is the revision number retrieved in the previous step
• The administrator copies the tarball to the ISFDB server and moves it to /home/avonruff/isfdb2/, where the main copy of the source code currently resides
• The administrator untars the tarball and types "make install" (or "make -B install" if the patch added new scripts or otherwise requires a complete rebuild)
• The administrator gzips the tarball and moves it to the archive area
• The administrator marks the Bug(s)/FR(s) as "Fixed/Completed" in SourceForge
• The administrator moves the completed change(s) from the list of Outstanding Changes on the Development page to the "Recent Patches" sub-page. "Recent Changes" covers all builds for the current year, so it's a useful page for developers to review.

Changes and Patches

Recent Patches

Patch Archive for 2009-2017

See Development/Recent Patches for the list of changes implemented recently.

Outstanding changes

Bug or Feature	Description	Modules and versions	Developer	Tester	Date Passed	Patch

Planned changes

Bug or Feature	Description	Modules and versions	Developer	Tester	Date Passed	Patch
Bug 3183863	Static and dynamic content cannot be "rehomed" correctly	TBD	Uzume

Current Activity

JesseW

Rename the three PrintNavBar's to distinguish them

See FR 3115118. It'd be very easy, but I want some feedback before I do it. JesseW 05:58, 22 November 2010 (UTC)

Clarify the license

See Bug 3115153; it's not good that the actual files make no mention of the license the code is released under. JesseW 07:09, 22 November 2010 (UTC)

MartyD

Concentrating on bugs, clean-up/consolidation, and small features. If anything ever comes of the tagging discussion at ISFDB:Community Portal#Title type tags in bibliography display, I will revisit the tagging used, too.

Development Process

General Principles

The ISFDB code base is over 5MB in size and has a number of dependencies, which are not always obvious. For this reason, developers should follow the "if it ain't broken, don't fix it" principle. Developers should also try to address one problem at a time. If a feature or a bug fix requires mass changes, discuss it with other developers and the administrator first.

Coding style

Among the developers, there is a wide variety of backgrounds and levels of Python experience. Due to this, it's important to take care to keep the code understandable without a high level of Python-specific knowledge. Using basic structured programming constructs (i.e., def, if, while, etc.) and simple classes rather than heavy object-orientation or more esoteric functional programming tricks is recommended. If you need to create a complex class, method or functions, you may want to discuss it with other developers and the administrator first and then document the code thoroughly.

Where is db defined?

Whenever SQLparsing.py is imported, a database connection is created, and assigned to the variable db. SQLparsing.py is copied by the makefiles from common/ into all the other directories, and imported by more or less everything, often multiple times. This is may be suboptimal.

Code Format

• The code appears to use 'TAB' instead of 'SPACE SPACE SPACE SPACE' to indent the code. (Some online python tutorials indicate a preference for one or the other.) You should use TABS for this project to indent because Python uses the indent level to define code blocks and sub steps. Mixing tabs and spaces in a File/project can cause the compiler to misunderstand where the functions and loops start and stop.
• In other ways the code should probably adhere to PEP 8: Style Guide for Python Code

How to Add a New Cleanup Report

ISFDB has two background jobs. The first one runs nightly and regenerates certain computationally intensive Statistics and Top Lists. It also regenerates most of our 240+ Cleanup Reports.

The second background job runs monthly and is meant to generate cleanup reports which take a very long time (hours) to run. At this time it is only used to regenerate the "Suspected Duplicate Authors" report.

If you want to add another cleanup report, use the following steps:

1. Develop a SQL query to find potentially problematic records. Your query should return only one column, which should contain the record ID of a suspicious record. Any subsequent columns will be ignored by the report generator. Note that you should only retrieve one record type (titles, pubs, series, etc) per report. If you need to return multiple record types, create one report per record type.
2. Add a new report to the bottom of module "nightly/nightly_update.py". Set the "query" variable to the body of the SQL statement that you created in the previous step. Invoke "standardReport(query, NN)" where NN is the report number of your report. To determine the value of NN, take the last report number used and add 1. When this module is run by the nightly job, it will extract the specified record IDs and add them to the "cleanup" table in the database.
3. In module mod/common.py, add the name of the new report to dictionary "reports" in function reportsDict. Increment the last use key to get the key for your report -- it should be the same as the report number that you created in the previous step. In the same function, add this report number to the appropriate tuple of the "sections" list. For example, if you are developing a cleanup report that affects magazines, add the report number to the "Magazines" tuple.
4. In module mod/cleanup_report.py, create a new function called "functionNN" where NN is the report number describe in the last two steps. The function should take no arguments, e.g. "def function99()". Defined a SQl query that contains the same logic as the query describe in the first step except that it also limits the search to the records stored in table cleanup for this report number. To do this, add the following logic to WHERE clause of the SQL statement: "and TABLE_NAME.RECORD_ID=cleanup.record_id and cleanup.report_type=NN and cleanup.resolved IS NULL", where TABLE_NAME is the table name of the record that you are extracting, RECORD_ID is the name of the column of the record ID and NN is the report number. Display the retrieved data using other reports as an example. Use function PrintTableColumns to print table columns. Use functions PrintSeriesRecord, PrintPublicationRecord, PrintTitleRecord, PrintAuthorRecord, PrintPublisherRecord and PrintPubSeriesRecord to print simple rows consisting of two columns: row number and a link to the problematic database record. These functions also support an optional third column for "Ignore this record" links. See the function definitions for the parameters that they take.

How to Add a new Field to Title records

1. Create a SQL script to add the new field to the titles tables
2. Modify the load and cgi2obj methods of the Title class
3. Add the new field to the Edit Title modules (edit/edittitle.py, edit/submittitle.py, mod/ta_update.py)
4. Add the new field to the DisplayTitleEdit function in biblio/viewers.py
5. Add the new field to biblio/title.py
6. Consider whether the field needs to be displayed in the Content section of the Publication display page, similar to non-genre or graphic. If so, modify biblio/pl.py
7. Consider whether the field should appear on Summary/Series pages, similar to non-genre or graphic
8. Consider whether the new field should be added to the New Pub data entry form, similar to non-genre or graphic
9. Add mouse-over help in edit/isfdblib.py. If the field has also been added to NewPub, make sure to add it to two places in edit/isfdblib.py, one for Title Help and one for Pub Help.
10. Account for the new field in the following module groups, including the data entry forms, the relevant viewers.py functions and the filers:
    • Title Merge
    • Title Unmerge
    • Add Variant
    • Variant Title
11. Consider adding the new field to the Title section of the Advanced Search

How to Add a New Language

1. Note that only ISO 639-2-recognized languages are supposed to be added to the ISFDB.
2. Add the requested language to the MySQL table "languages". For example, here is the body of scripts/add_middle_high_german_language.sql which added Middle High German:
   • INSERT INTO languages (lang_id, lang_code, lang_name) VALUES (102, 'gmh','Middle High German')
     • "102" is the primary key and "gmh" is the ISO 639-2 code for this language. (ISO 639-2 codes are not displayed by the ISFDB software at this time.)
3. Add the name of the newly added language to the end of the tuple "LANGUAGES" in common/isfdb.py. This tuple must be kept in sync with the "languages" table at all times. Even though it was originally added to improve performance, if this tuple ever gets out of sync with the "languages" table, the ISFDB software will stop working.

Duplicate Functions and Duplicate Filenames

WARNING — There are duplicate file names and duplicate functions. The function PrintNavBar for instance, appears in /biblio/common.py (with 5 arguments) and in /edit/isfdblib.py (with 2 arguments) and again in /mod/isfdblib.py (with no arguments). Be sure to watch your directory of the file you are editing, and you cannot count on a function in one directory behaving the same when working in another directory.

Indeed. When looking for "SQLwikiLinkExists" it turns out we have five of them, and rather too many "SQLparsing.py" files. We could do with some comparisons and centralisation, although this obviously has possibly far-ranging effects and would need lots of regression testing. BLongley

Files living in common are master files and are copied to the other parts of the tree during the build process. The good news is, CVS won't let you commit the others, as they don't actually exist in those directories in CVS. TortoiseCVS distinguishes them as local files with a different icon.... —MartyD 20:48, 14 June 2009 (UTC)

This isn't a case of common files. This is duplicate file and function naming. Kevin 03:25, 15 June 2009 (UTC)

I have begun the first step in the journey to fixing the PrintNavBar duplication problem. I've created a new common/navbar.py and made all of the other directories share it. We can slowly move NavBar things into it. —MartyD 21:14, 5 July 2009 (UTC)

Related Projects

• Solr Powered ISFDB