User:Fixer/Python MySQL Migration

From ISFDB
< User:Fixer
Revision as of 12:00, 8 August 2023 by Ahasuerus (talk | contribs) (+Pymarc)
Jump to navigation Jump to search
  • 2023-08-01: Since the current version of the Fixer software depends on Windows 10 to function and Windows 10 will no longer be getting security patches after 2025-10-25, we need to migrate all of Fixer's processes to Linux, Python and MySQL. At this time, the ISBN/ASIN identification process and the Fixer-Amazon API use Python.
  • 2023-08-02: Tentative migration plan is as follows:
    1. Move Fixer's data from "data stores" to MySQL tables on Linux. At that point the Fixer software should be talking to MySQL using ODBC.
    2. Rewrite the current version of the Fixer software in Python 3, which will eliminate Windows dependencies and make it possible for other developers to maintain/upgrade Fixer's code if something happens to me. The UI will still be "roll-and-scroll" at that point.
    3. Upgrade Fixer's Amazon interface to support multiple concurrent users, which will require a single-threaded request queue due to Amazon throttling.
    4. Upgrade the UI from "roll-and-scroll" to regular HTML.
    5. Integrate the new HTML interface with the ISFDB code; leverage ISFDB/Wiki account authentication.
    6. Deploy the new Fixer on isfdb2.org and make it available to moderators.
  • 2023-08-03. Fixer-MySQL communications via ODBC appear to be more problematic than expected due to 32 bit/64 bit incompatibilities with the development server's version of MySQL (5.5 32-bit). Upgrading MySQL to 8.0 may be needed for it to work.
  • 2023-08-04. Fixer-Amazon API upgraded to be compatible with Python 3.
  • 2023-08-05. Started designing SQL tables for the MySQL-based version of Fixer.
  • 2023-08-06. Created a new MySQL database where new Fixer tables and copies of publicly available ISFDB tables will coexist.
  • 2023-08-07. Backed up, deleted and consolidated unused Fixer data stores. Moved remaining Queue 4 and 5 ISBNs/ASINs to queues 1-3; deleted Queues 4 and 5.
  • 2023-08-05 through 2023-08-08. Download of MARC21 archives made publicly available by major libraries like LOC, BL and DNB. We may be able to leverage them once Fixer is fully migrated.
    • Note that MARC21 files can be read and parsed using the Python library Pymarc -- see documentation. Versions 5+ of Pymarc require Python 3.7+.
Retrieved from "https://isfdb.org/wiki/index.php?title=User:Fixer/Python_MySQL_Migration&oldid=668761"