|ISFDB Discussion Pages and Noticeboards|
|Before posting to this page, consider whether one of the other discussion pages or noticeboards might suit your needs better.|
If you're looking for help remembering a book title, check out the resources in our FAQ.
Please also see our Help pages.
Questions about doing a specific task, or how to correct information when the solution is not immediately obvious.
• New post • Archives
Help with bibliographic, image credit, and other questions which require a physical check of the work in question.
• New post • Archives
|Rules and standards
Discussions about the rules and standards, as well as questions about interpretation and application of those rules.
• New post • Rules changelog • Archives
General discussion about anything not covered by the more specialized noticeboards to the left.
• New post • Archives
Get the attention of moderators regarding submission questions.
• New post • Archives • Cancel submission
|Roadmap: For the original discussion of Roadmap 2017 see this archived section. For the current implementation status, see What's New#Roadmap 2017.|
This page serves as the hub for discussing ISFDB-related software development.
Developers and Testers
- MartyD — developer
- Ahasuerus — developer, tester and installer/administrator
- Alvonruff — developer and administrator
Software Development Process
At this time the development process works as follows:
- A Bug, Support Request ("SR") or Feature Request ("FR") is created in SourceForge. Anyone can do it, even "anonymous".
- FRs should be discussed on the ISFDB:Community Portal and approved by the community before the work begins. Obvious bugs can be fixed and submitted without a prior discussion.
- Once a developer decides to work on a Bug/SR/FR, s/he identifies the scripts that are affected and lists the Bug/SR/FR along with the scripts on the Development page under "Outstanding Changes". This helps avoid effort duplication.
- The developer makes changes on his or her development server and tests them. S/he then sends to the ISFDB administrator (Ahasuerus since 2009), who tests them on his local development server. In some cases the administrator may ask another developer to test the changes.
- If everything works fine, the administrator commits the changes to the SourceForge repository using the SVN "commit" command, e.g.:
svn commit -m "FR 1099 Add Last Activity date to 'Top Contributor/Moderator/etc' lists - Part 1"
- The administrator gets the latest revision number from the SourceForge repository:
svn info --show-item=revision -r HEAD
- The administrator creates a deployable tarball file from the modified source files, e.g:
- where '3' is the revision number retrieved in the previous step
tar -cvf ~/patches/3.tar biblio/toptaggers.py
- The administrator copies the tarball to the ISFDB server and moves it to
/home/avonruff/isfdb2/, where the main copy of the source code currently resides
- The administrator untars the tarball and types
make -B installif the patch added new scripts or otherwise requires a complete rebuild)
- The administrator gzips the tarball and moves it to the archive area
- The administrator marks the Bug(s)/SR(s)/FR(s) as "Fixed/Completed" in SourceForge
- The administrator moves the completed change(s) from the list of Outstanding Changes on the Development page to the "Recent Patches" sub-page. "Recent Changes" covers all builds for the current year, so it's a useful page for developers to review.
Changes and Patches
See Development/Recent Patches for the list of changes implemented recently.
|Bug or Feature||Description||Modules and versions||Developer||Tester||Date Passed||Patch|
|Bug or Feature||Description||Modules and versions||Developer||Tester||Date Passed||Patch|
|SR 24||Static and dynamic content cannot be "rehomed" correctly||
See Development/HTTPS, User:Alvonruff/HTTPS Notes, and User:Alvonruff/ISFDB Move. The auto-renewal of certs is still an open item, and I am generally unsatisfied with the cookie-cutter, copy-and-paste methodology I used to get the current HTTPS solution running. Some deeper expertise needs to be cultivated. --Alvonruff (talk) 11:01, 16 September 2022 (EDT)
Concentrating on bugs, clean-up/consolidation, and small features. If anything ever comes of the tagging discussion at ISFDB:Community Portal/Archive/Archive14#Title type tags in bibliography display, I will revisit the tagging used, too.
The ISFDB code base is over 5MB in size and has a number of dependencies, which are not always obvious. For this reason, developers should follow the "if it ain't broken, don't fix it" principle. Developers should also try to address one problem at a time. If a feature or a bug fix requires mass changes, discuss it with other developers and the administrator first.
Among the developers, there is a wide variety of backgrounds and levels of Python experience. Due to this, it's important to take care to keep the code understandable without a high level of Python-specific knowledge. Using basic structured programming constructs (i.e.,
while, etc.) and simple classes rather than heavy object-orientation or more esoteric functional programming tricks is recommended. If you need to create a complex class, method or functions, you may want to discuss it with other developers and the administrator first and then document the code thoroughly.
Where Are the Globally Scoped Variables defined?
Almost all CGI scripts which comprise the ISFDB system have a
from isfdb import * statement at the top of the script. Module
isfdb defines many all-uppercase variables like
EURO_SIGN, offset values for tuples retrieved from the SQL database, etc. It also sets up
SESSION, an object which contains session-specific information like the name of the directory that the current CGI script resides in. In addition, module
from localdefs import *, which imports installation-specific global variables like
WIKILOC. Upon import, all of these variable become available to the invoking CGI script.
ISFDB scripts which interact with the SQL database, i.e. the vast majority of them, import functions from SQLparsing. When the import occurs, the SQLparsing module creates a database connection and assigns it to the global variable
db, which is then used by individual CGI scripts and other modules to interact with the database. SQLparsing also retrieves a list of supported languages and assigns it to the globally scoped variable
Older ISFDB modules tend to use tabs. Newer ISFDB modules tend to use 8 spaces. A few ISFDB modules use 4 spaces. Some modules use a mix of spaces and tabs, which can cause problems, especially if and when we migrate from Python 2 to Python 3.
- For new development, use 8 spaces.
- When modifying existing modules which use either spaces OR tabs, use the current indentation method.
- When modifying existing modules which use a mix of spaces and tabs, change the module to use 8 spaces. This may require additional testing to ensure that nothing got broken.
- When modifying existing code, use the naming conventions and other standards found within the module.
- For new development, use PEP 8: Style Guide for Python Code unless it goes ISFDB-wide coding conventions.
How to Add a New Cleanup Report
ISFDB has three background jobs, not counting the backups:
- Nightly job. It regenerates less computationally intensive Cleanup Reports and takes over 5 minutes to run.
- Weekly job. It regenerates all Statistics and Top Lists as well as all Cleanup Reports. It takes over 20 minutes to run.
- Monthly job. It regenerates cleanup reports which take a very long time to run. At this time it is only used to regenerate the "Suspected Duplicate Authors" report. The job is currently suspended because the algorithm doesn't scale well and it takes many days to complete.
If you want to add another cleanup report, use the following steps:
- Develop a SQL query to find potentially problematic records. Your query should return only one column, which should contain the record ID of a suspicious record. Any subsequent columns will be ignored by the report generator. Note that you should only retrieve one record type (titles, pubs, series, etc) per report. If you need to return multiple record types, create one report per record type.
- Add a new report to the bottom of module
nightly/nightly_update.py. Set the
queryvariable to the body of the SQL statement that you created in the previous step. Invoke
standardReport(query, NNN)where NNN is the report number of your report. To determine the value of NNN, take the last report number used and add 1. When this module is run by the nightly job, it will extract the specified record IDs and add them to the "cleanup" table in the database.
- In module
edit/cleanup_lib.py, add the name of the new report to the dictionary
reportsDict. Increment the last use key to get the key for your report -- it should be the same as the report number that you created in the previous step. In the same function, add this report number to the appropriate tuple of the
sectionslist. For example, if you are developing a cleanup report that affects magazines, add the report number to the
Magazinestuple. If the report is supposed to be accessed by editors without moderator privileges, add its number to the of the
non_moderatortuple in the same function.
- In module
edit/cleanup_report.py, create a new function called
functionNNNwhere NNN is the report number describe in the last two steps. The function should take no arguments, e.g.
def function555(). Define a SQL query that contains the same logic as the query describe in the first step except that it also limits the search to the records stored in the table "cleanup" for this report number. To do this, add the following logic to the
WHEREclause of the SQL statement:
and TABLE_NAME.RECORD_ID=cleanup.record_id and cleanup.report_type=NNN and cleanup.resolved IS NULL, where TABLE_NAME is the table name of the record that you are extracting, RECORD_ID is the name of the column of the record ID and NNN is the report number. Display the retrieved data using other reports as an example. Use function
PrintTableColumnsto print table columns. Use functions
PrintPubSeriesRecordto print simple rows consisting of two columns: row number and a link to the problematic database record. These functions also support an optional third column for "Ignore this record" links. See the function definitions for the parameters that they take.
How to Add a new Field to Title records
- Create a SQL script to add the new field to the titles tables
- Modify the
cgi2objmethods of the
- Add the new field to the Edit Title modules (
- Add the new field to the
- Add the new field to
- Consider whether the field needs to be displayed in the "Content" section of the "Publication" display page, similar to non-genre or graphic. If so, modify
- Consider whether the field should appear on "Summary/Series" pages, similar to non-genre or graphic
- Consider whether the new field should be added to the "New Pub" data entry form, similar to non-genre or graphic
- Add mouse-over help in
edit/isfdblib.py. If the field has also been added to "NewPub", make sure to add it to two places in
edit/isfdblib.py, one for "Title Help" and one for "Pub Help".
- Account for the new field in the following module groups, including the data entry forms, the relevant viewers.py functions and the filers:
- Title Merge
- Title Unmerge
- Add Variant
- Variant Title
- Consider adding the new field to the "Title" section of the "Advanced Search"
How to Add a New External ID Type
- Add the requested External ID Type to the MySQL table
identifier_types. For example, here is the command that added KBR as a new External ID Type:
INSERT INTO identifier_types (identifier_type_name, identifier_type_full_name) VALUES ('KBR', 'De Belgische Bibliografie/La Bibliographie de Belgique');
- If the new External ID Type links to one or more third party URL, add that URL (or URLs) to the MySQL table
identifier_sites, one URL per command. Use "%s" for the part of the URL which will be replaced with the actual External ID at run time. Use the value of
identifier_type_idgenerated by the previous command. For example, here is the command that added the URL of De Belgische Bibliografie:
INSERT INTO identifier_sites (identifier_type_id, site_position, site_url) VALUES (19, 1, 'https://opac.kbr.be/Library/doc/SYRACUSE/%s/');
- If applicable, add a matching linking template to the function
common/library.py. See the comments section at the beginning of the function for details. See an existing linking template like "ASIN" for an example.
How to Add a New Banner Image
- Make sure that the new image file contains a valid JPG image
- Put a copy of the new image file in the
- Change the name of the new JPG file to
IsfdbBannerNNwhere "NN" is 1 higher than the highest currently existing file name
common/isfdb.py, find the variable which contains the number of the last IsfdbBanner file and increment it by one to match NN
Duplicate Functions and Duplicate Filenames
WARNING — There are duplicate file names and duplicate functions. The function
PrintNavBar for instance, appears in
/biblio/common.py (with 5 arguments) and in
/edit/isfdblib.py (with 2 arguments) and again in
/mod/isfdblib.py (with no arguments). Be sure to watch your directory of the file you are editing, and you cannot count on a function in one directory behaving the same when working in another directory.
- Indeed. When looking for
SQLwikiLinkExistsit turns out we have five of them, and rather too many
SQLparsing.pyfiles. We could do with some comparisons and centralisation, although this obviously has possibly far-ranging effects and would need lots of regression testing. BLongley
- Files living in common are master files and are copied to the other parts of the tree during the build process. The good news is, CVS won't let you commit the others, as they don't actually exist in those directories in CVS. TortoiseCVS distinguishes them as local files with a different icon.... —MartyD 20:48, 14 June 2009 (UTC)
- This isn't a case of common files. This is duplicate file and function naming. Kevin 03:25, 15 June 2009 (UTC)
- I have begun the first step in the journey to fixing the
PrintNavBarduplication problem. I've created a new
common/navbar.pyand made all of the other directories share it. We can slowly move NavBar things into it. —MartyD 21:14, 5 July 2009 (UTC)