Difference between revisions of "User:Alvonruff/ISFDB Move"

From ISFDB
Jump to navigation Jump to search
(→‎Wiki Content: Max upload file size)
(→‎MySQL: Added binlog-expire-logs-seconds instructions)
Line 40: Line 40:
 
* GRANT ALL PRIVILEGES ON isfdb.* TO 'isfdb1'@'localhost';
 
* GRANT ALL PRIVILEGES ON isfdb.* TO 'isfdb1'@'localhost';
 
* SET GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION';"
 
* SET GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION';"
 +
* Add the following line to the end of /etc/my.cnf.d/community-mysql-server.cnf:
 +
** binlog-expire-logs-seconds=259200
 +
* and restart MySQL (systemctl restart mysqld); this will ensure that MySQL's binary log files are automatically purged after 3 days as opposed to the default value (30 days)
  
 
==Python 2.7.18==
 
==Python 2.7.18==

Revision as of 11:34, 25 September 2022

Items While System Only Has an IP Address

The new ISFDB location has the static IP address: 67.225.129.86 (Link: http://67.225.129.86/cgi-bin/index.cgi)

Prerequisites

The system is a minimum configuration of Fedora Core. As such, many packages need to be installed with dnf.

  • dnf install gcc
  • dnf install make
  • dnf install tar
  • dnf install zip.x86_64
  • dnf install bzip2.x86_64
  • dnf install wget
  • dnf install mod_dav_svn subversion
  • dnf install telnet
  • dnf install ImageMagick

Apache

  • dnf install httpd
  • firewall-cmd --add-service=http --add-service=https --permanent
  • Add to etc/httpd/httpd.conf: LoadModule dir_module modules/mod_dir.so
  • Change etc/httpd/httpd.conf: DirectoryIndex /cgi-bin/index.cgi index.html index.html.var
  • systemctl enable httpd
  • systemctl start httpd
  • copy /var/www/html/robots.txt from the old server to the new server

MySQL

  • dnf update
  • dnf module enable mysql:8.0
  • dnf install @mysql
  • systemctl enable mysqld
  • systemctl start mysqld
  • Issue: mysql
  • While in mysql, issue the command: create database isfdb;
  • While in mysql, issue the command: use isfdb;
  • While in mysql, issue the command: alter database isfdb character set latin1 collate latin1_swedish_ci;
  • While in mysql, issue the command: source <<backupfile>>;'
  • create user isfdb1@localhost identified by 'PASSWORD';
  • GRANT ALL PRIVILEGES ON isfdb.* TO 'isfdb1'@'localhost';
  • SET GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION';"
  • Add the following line to the end of /etc/my.cnf.d/community-mysql-server.cnf:
    • binlog-expire-logs-seconds=259200
  • and restart MySQL (systemctl restart mysqld); this will ensure that MySQL's binary log files are automatically purged after 3 days as opposed to the default value (30 days)

Python 2.7.18

  • dnf install python2
  • dnf install python2-devel
  • dnf install mysql-devel
  • python2 -m ensurepip --no-default-pip
  • pip install --upgrade pip
  • pip2 install mysqlclient
  • alternatives --install /usr/bin/python python /usr/bin/python3.9 2
  • alternatives --install /usr/bin/python python /usr/bin/python2.7 1
  • alternatives --config python
  • python --version

ISFDB

  • In home directory: svn checkout https://svn.code.sf.net/p/isfdb/code-svn/trunk isfdb-code-svn
  • Update common/localdefs.py:
    • COOKIEHOST = "67.225.129.86"
    • HTFAKE = "/67.225.129.86/cgi-bin"
    • HTMLHOST = "67.225.129.86"
    • HTMLLOC = "67.225.129.86"
    • PASSWORD = "PASSWORD"
    • USERNAME = "isfdb1"
    • WIKILOC = "67.225.129.86/wiki"
  • make

PHP

  • dnf module reset php
  • dnf module enable php:7.4
  • dnf install php php-common php-opcache php-cli php-gd php-curl php-mysqlnd
  • systemctl start php-fpm
  • systemctl enable php-fpm
  • php -v
  • systemctl stop httpd
  • systemctl start httpd

Add Users

  • Add the necessary users to the new system
  • Give them sudo access by adding them to the wheel group

MediaWiki

  • cd to /var/www/html
  • Fetch the current stable long-term support release of MediaWiki:
  • unzip mediawiki-1.35.6.zip
  • mv mediawiki-1.35.6 wiki
  • cd wiki && place the old isfdb version of LocalSettings.php here
  • Edit LocalSettings.php and make the follow changes:
    • Change wgServer
    • Change wgDBuser
    • Change wgDBpassword
    • Comment out the ConfirmEdit extension
    • Comment out the SyntaxHighlight extension
    • Comment out the SVGtag extension
  • mkdir wiki/images
  • chown apache images
  • chgrp apache images
  • Copy all image subdirectory content from isfdb.org:/var/www/html/wiki/images to the new server.
  • php maintenance/update.php
    • Wait a very long time for the update to finish (about 3.5 hours). This will perform hundreds of thousands of revision updates, but it issues a constant stream of progress lines. There is a web-based alternative, but I recommend against that, as the long processing time will generate a 504 error from Apache.
    • It helps to ssh into the system with the "-o ServerAliveInterval=600" option, which will generate a null keep alive packet every 10 minutes, keeping the session alive.
  • cp isfdb.gif to wiki/skins/common/images
  • Add the following lines to the end of LocalSettings.php and remove the old commented out versions:
    • wfLoadSkin( 'Vector' );
    • wfLoadExtension( 'ConfirmEdit' );
    • wfLoadExtension( 'SyntaxHighlight_GeSHi' );
    • wfLoadExtension( 'WikiEditor' );
    • $wgExtraSignatureNamespaces = [ NS_MAIN, NS_USER, NS_TALK, NS_PROJECT ];
    • Uncomment and change to: $wgImageMagickConvertCommand = "/usr/bin/convert";
    • Uncomment and change to: $wgLogo = "$wgStylePath/common/images/isfdb.gif";

New Password Support

  • pip2 install backports.pbkdf2
  • Copy new submitlogin.py into biblio
  • set legacy_mode = 0 in submitlogin.py
  • reinstall (make) isfdb source

Move Servers (Commence 9/6)

  • At old server:
    • DONE: Turn off editing at isfdb.org
    • DONE: Create final backups of database and images.
  • At new server:
    • DONE: Commit the new password code in submitlogin.py to the SourceForge SVN repository. The value of the "legacy_mode" flag should be set to 0. (Previously the file was manually placed, but not committed to SVN -- see above.)
    • DONE: Ahasuerus should do a source install to ensure he has the ability to install source updates.
    • DONE: Drop the isfdb tables from mysql using the commands listed here. (Dropping the entire database may cause goofyness with the isfdb1 user, and the privileges tables)
    • DONE: Remove the wiki images from the filesystem (Everything under /var/www/html/wiki/images)
    • DONE: Re-execute the operations to get the full database backup installed.
    • DONE: Install the latest wiki image backups
    • DONE: Put the original LocalSettings.php in place.
    • DONE: Run: php maintenance/update.php. (Note: This took about 24 hours to execute on the new server last time. If the ssh connection drops during the update, it does keep track of where it is in the process)
    • Disable scripts/wikitrim.py until it has been updated to the new version of MediaWiki

Move Domain Name

  • DONE: Update the domain name servers to point to the new server
  • DONE: Wait for domain name to propagate (up to 48 hours).
  • DONE: Update the following fields in localdefs.py:
    • COOKIEHOST
    • HTFAKE
    • HTMLHOST
    • HTMLLOC
    • WIKILOC
  • DONE: Update the following fields in LocalSettings.php
    • wgServer

Mail

  • DONE: dnf install postfix
  • DONE: Edit /etc/postfix/main.cf:
    • Set myhostname = isfdb.org
    • Set mydomain = isfdb.org
    • Set mail_spool_directory = /var/mail
    • Set mynetworks_style = host
    • Set mynetworks = 127.0.0.0/8 67.225.129.86
    • Uncomment relay_domains = $mydestination
    • Uncomment inet_interfaces = $myhostname, localhost
  • DONE: systemctl start postfix
  • DONE: systemctl enable postfix
  • DONE: firewall-cmd --add-service=smtp --permanent

Certs

Apache

  • DONE: dnf install mod_ssl
  • DONE: systemctl restart httpd
  • DONE: cd /etc/httpd/conf.d
  • DONE: Create file isfdb.org.conf
  • DONE: Add the following contents:
<VirtualHost *:80>
    ServerName server.isfdb.org
    DocumentRoot /var/www/html
    ServerAlias isfdb.org
    ErrorLog /var/www/error.log
    CustomLog /var/www/requests.log combined
    Redirect permanent / https://isfdb.org/
</VirtualHost>
  • systemctl restart httpd

Certificate Support Software

  • SKIP: dnf install epel-release (not a valid package any longer)
  • DONE: dnf install snapd
  • DONE: dnf install certbot
  • DONE: dnf install python3-certbot-apache

Certificates

  • DONE: certbot --apache
    • Select isfdb.org
  • DONE: systemctl restart httpd
  • DONE: certbot -d isfdb.org,www.isfdb.org --expand
    • Select isfdb.org.conf
    • This generates the error: The selected vhost would conflict with other HTTPS VirtualHosts within Apache. Please select another vhost or add ServerNames to your configuration. Could not install certificate
    • But links to https://www.isfdb.org now work.
  • Since the last Certbot operation failed (see above), it didn't install a cron job to auto-renew the certificate before it expires after 90 days. We will need to either get Certbot to work (probably best tested on isfdb2.org first) or set up a manual process to renew the certificate once every 3 months.

Settings

  • DONE: Change PROTOCOL in localdefs.py to "https"
  • DONE: Change $wgServer in /var/www/html/wiki/LocalSettings.php to use https
  • DONE: Go to the wiki and type MediaWiki:Sidebar in the search window. Edit that page and change http to https

Interlude

While everything at isfdb.org seems to be working, and external links at Wikipedia follow okay, the domain www.isfdb.org is still busted, and points to a GoDaddy parking page. Will debug that tomorrow. Alvonruff (talk) 22:14, 6 September 2022 (EDT)

Just in case you aren't aware, as at 12:05 BST Add Publication (and likely other pages/scripts) is currently broken - browser dev tools implies it's failing to load a bunch of static images, .js files etc from 67.225.129.86 on a http:// URL, but as this is coming from an https:// page, my browser (Vivaldi) is blocking this. I guess whatever setting is used to generate the URLs will have to be switched to use https and a proper domain name ErsatzCulture (talk) 07:05, 7 September 2022 (EDT)
EDIT: I see one of the steps below is "Enable editing at new server", so I guess this isn't expected to be working yet - however I assume that's related to the config flag that causes the "Editing facilities are currently offline" message, and the URLs are a different issue?
We setup the new site such that Ahasuerus could do the official updates (much like the old site), but a side-effect of that during the domain move was me doing hand edits of files, without the generation that occurs during make, leading to some stray files with the IP address. So the next order of business will be for Ahasuerus to pick up the latest localdefs.py and do a make from the official branch. Alvonruff (talk) 07:21, 7 September 2022 (EDT)
OK, thanks ErsatzCulture (talk) 07:28, 7 September 2022 (EDT)
This update happened at 07:52 (Eastern Time), and a quick check shows the editing apps working as expected (without references to the IP address), so we can start looking for the next level of issues. Alvonruff (talk) 08:16, 7 September 2022 (EDT)

Final Steps

  • DONE: Uncomment the daily backup job in root's crontab (vi /etc/crontab)
    • DONE: Confirm that the daily backup job ran successfully the following morning
  • DONE: Uncomment 2 weekly image backup jobs in root's crontab
    • DONE: Confirm that the weekly image backup jobs ran successfully on Sunday/Monday morning
  • DONE: Update Ahasuerus's development server to download daily backups and weekly image backups automatically
    • DONE: Create public backups on the development server and confirm that they contain the correct data
  • DONE: Uncomment the nightly job in root's crontab
    • DONE: Confirm that the nightly job ran successfully the following night
  • DONE: Uncomment the weekly job in root's crontab
    • DONE: Confirm that the weekly job ran successfully the following Sunday morning
  • DONE: Keep the monthly job in root's crontab commented out
  • DONE: Enable editing for moderators at new server
  • SKIP: Update scripts/create_user.py to generate a pbkdf2 hash for the password. Since create_user.py is present specifically for users setting up a copy of the ISFDB without a wiki, submitlogin.py can be configured to use the old MD5 hashing scheme, which will match what's used in create_user.py.
  • DONE for Alvonruff: Folks that have logins and data in their home directories on the current isfdb.org will need to move it to the new server, as it will be decommissioned. IP address of the current (old) isfdb.org is: 72.44.93.41
  • DONE: Consider updating the old site to use the IP address, so that we can locate the stray content that made it into the wiki after editing was turned off, and the backups made. (Turning off editing only affects the ISFDB side, not the wiki side). Nexcess will likely decommission the old site pretty quickly.
  • DONE: On the old server, download, zip up and back up all files in /var/www/html. They include a variety of superseded PDF, HTML and SVG files generated during the 2000s and 2010s. Although they are no longer relevant, we should keep a copy just in case.
  • DONE: Enable public editing at new server
  • DONE: Change Fixer's settings to use HTTPS
  • DONE: Notify The Encyclopedia of Science Fiction about the switch to HTTPS
    • All SFE links to ISFDB pages have been modified to use HTTPS

Wiki Content

Wikipedia Templates

The following templates need to be converted from http to https:

  • Template:Isfdb series - Updated
  • Template:Isfdb contents - Updated
  • Template:Isfdb name - Updated
  • Template:Isfdb title - Updated

Versions

  • Linux: 5.14.18-100.fc33.x86_64
  • Apache: Apache/2.4.37
  • MySQL: 8.0.27
  • Python: 2.7.18
  • PHP: 7.4.26
  • MediaWiki: 1.35.6
Retrieved from "https://isfdb.org/wiki/index.php?title=User:Alvonruff/ISFDB_Move&oldid=645563"