User:Fixer/Python MySQL Migration
Jump to navigation
Jump to search
- 2023-08-01: Since the current version of the Fixer software depends on Windows 10 to function and Windows 10 will no longer be getting security patches after 2025-10-25, we need to migrate all of Fixer's processes to Linux, Python and MySQL. At this time, the ISBN/ASIN identification process and the Fixer-Amazon API use Python.
- 2023-08-02: Tentative migration plan is as follows:
- Move Fixer's data from "data stores" to MySQL tables on Linux. At that point the Fixer software should be talking to MySQL using ODBC.
- Rewrite the current version of the Fixer software in Python 3, which will eliminate Windows dependencies and make it possible for other developers to maintain/upgrade Fixer's code if something happens to me. The UI will still be "roll-and-scroll" at that point.
- Upgrade Fixer's Amazon interface to support multiple concurrent users, which will require a single-threaded request queue due to Amazon throttling.
- Upgrade the UI from "roll-and-scroll" to regular HTML.
- Integrate the new HTML interface with the ISFDB code; leverage ISFDB/Wiki account authentication.
- Deploy the new Fixer on isfdb2.org and make it available to moderators.
- 2023-08-03. Fixer-MySQL communications via ODBC appear to be more problematic than expected due to 32 bit/64 bit incompatibilities with the development server's version of MySQL (5.5 32-bit). Upgrading MySQL to 8.0 may be needed for it to work.
- 2023-08-04. Fixer-Amazon API upgraded to be compatible with Python 3.
- 2023-08-05. Started designing SQL tables for the MySQL-based version of Fixer.
- 2023-08-05/09. Download of MARC21 archives made publicly available by major libraries like LOC, BL and DNB. We may be able to leverage them once Fixer is fully migrated.
- Note that MARC21 files can be read and parsed using the Python library Pymarc -- see documentation. Versions 5+ of Pymarc require Python 3.7+.
- 2023-08-06. Created a new MySQL database where new Fixer tables and copies of publicly available ISFDB tables will coexist.
- 2023-08-07. Backed up, deleted and consolidated unused Fixer data stores. Moved remaining Queue 4 and 5 ISBNs/ASINs to Queues 1-3; deleted Queues 4 and 5.
- 2023-08-09. Finished the first iteration of the MySQL table design.
- 2023-08-09/10. Upgraded the process of exporting Fixer's data stores to XML.
- 2023-08-10. Started work on importing Fixer-generated XML into MySQL.
- 2023-08-12. Upgraded XML dumps made public as part of the 2023-08-12 backups.