Items While System Only Has an IP Address
The new ISFDB location has the static IP address: 188.8.131.52 (Link: http://184.108.40.206/cgi-bin/index.cgi)
The system is a minimum configuration of Fedora Core. As such, many packages need to be installed with dnf.
- dnf install gcc
- dnf install make
- dnf install tar
- dnf install zip.x86_64
- dnf install bzip2.x86_64
- dnf install wget
- dnf install mod_dav_svn subversion
- dnf install telnet
- dnf install ImageMagick
- dnf install httpd
- firewall-cmd --add-service=http --add-service=https --permanent
- Add to etc/httpd/httpd.conf: LoadModule dir_module modules/mod_dir.so
- Change etc/httpd/httpd.conf: DirectoryIndex /cgi-bin/index.cgi index.html index.html.var
- systemctl enable httpd
- systemctl start httpd
- copy /var/www/html/robots.txt from the old server to the new server
- dnf update
- dnf module enable mysql:8.0
- dnf install @mysql
- systemctl enable mysqld
- systemctl start mysqld
- Issue: mysql
- While in mysql, issue the command: create database isfdb;
- While in mysql, issue the command: use isfdb;
- While in mysql, issue the command: alter database isfdb character set latin1 collate latin1_swedish_ci;
- While in mysql, issue the command: source <<backupfile>>;'
- create user isfdb1@localhost identified by 'PASSWORD';
- GRANT ALL PRIVILEGES ON isfdb.* TO 'isfdb1'@'localhost';
- SET GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION';"
- Add the following line to the end of /etc/my.cnf.d/community-mysql-server.cnf:
- and restart MySQL (systemctl restart mysqld); this will ensure that MySQL's binary log files are automatically purged after 3 days as opposed to the default value (30 days)
- dnf install python2
- dnf install python2-devel
- dnf install mysql-devel
- python2 -m ensurepip --no-default-pip
- pip install --upgrade pip
- pip2 install mysqlclient
- alternatives --install /usr/bin/python python /usr/bin/python3.9 2
- alternatives --install /usr/bin/python python /usr/bin/python2.7 1
- alternatives --config python
- python --version
- In home directory: svn checkout https://svn.code.sf.net/p/isfdb/code-svn/trunk isfdb-code-svn
- Update common/localdefs.py:
- COOKIEHOST = "220.127.116.11"
- HTFAKE = "/18.104.22.168/cgi-bin"
- HTMLHOST = "22.214.171.124"
- HTMLLOC = "126.96.36.199"
- PASSWORD = "PASSWORD"
- USERNAME = "isfdb1"
- WIKILOC = "188.8.131.52/wiki"
- dnf module reset php
- dnf module enable php:7.4
- dnf install php php-common php-opcache php-cli php-gd php-curl php-mysqlnd
- systemctl start php-fpm
- systemctl enable php-fpm
- php -v
- systemctl stop httpd
- systemctl start httpd
- Add the necessary users to the new system
- Give them sudo access by adding them to the wheel group
- cd to /var/www/html
- Fetch the current stable long-term support release of MediaWiki:
- unzip mediawiki-1.35.6.zip
- mv mediawiki-1.35.6 wiki
- cd wiki && place the old isfdb version of LocalSettings.php here
- Edit LocalSettings.php and make the follow changes:
- Change wgServer
- Change wgDBuser
- Change wgDBpassword
- Comment out the ConfirmEdit extension
- Comment out the SyntaxHighlight extension
- Comment out the SVGtag extension
- mkdir wiki/images
- chown apache images
- chgrp apache images
- Copy all image subdirectory content from isfdb.org:/var/www/html/wiki/images to the new server.
- php maintenance/update.php
- Wait a very long time for the update to finish (about 3.5 hours). This will perform hundreds of thousands of revision updates, but it issues a constant stream of progress lines. There is a web-based alternative, but I recommend against that, as the long processing time will generate a 504 error from Apache.
- It helps to ssh into the system with the "-o ServerAliveInterval=600" option, which will generate a null keep alive packet every 10 minutes, keeping the session alive.
- cp isfdb.gif to wiki/skins/common/images
- Add the following lines to the end of LocalSettings.php and remove the old commented out versions:
- wfLoadSkin( 'Vector' );
- wfLoadExtension( 'ConfirmEdit' );
- wfLoadExtension( 'SyntaxHighlight_GeSHi' );
- wfLoadExtension( 'WikiEditor' );
- $wgExtraSignatureNamespaces = [ NS_MAIN, NS_USER, NS_TALK, NS_PROJECT ];
- Uncomment and change to: $wgImageMagickConvertCommand = "/usr/bin/convert";
- Uncomment and change to: $wgLogo = "$wgStylePath/common/images/isfdb.gif";
New Password Support
- pip2 install backports.pbkdf2
- Copy new submitlogin.py into biblio
- set legacy_mode = 0 in submitlogin.py
- reinstall (make) isfdb source
Move Servers (Commence 9/6)
- At old server:
- DONE: Turn off editing at isfdb.org
- DONE: Create final backups of database and images.
- At new server:
- DONE: Commit the new password code in submitlogin.py to the SourceForge SVN repository. The value of the "legacy_mode" flag should be set to 0. (Previously the file was manually placed, but not committed to SVN -- see above.)
- DONE: Ahasuerus should do a source install to ensure he has the ability to install source updates.
- DONE: Drop the isfdb tables from mysql using the commands listed here. (Dropping the entire database may cause goofyness with the isfdb1 user, and the privileges tables)
- DONE: Remove the wiki images from the filesystem (Everything under /var/www/html/wiki/images)
- DONE: Re-execute the operations to get the full database backup installed.
- DONE: Install the latest wiki image backups
- DONE: Put the original LocalSettings.php in place.
- DONE: Run: php maintenance/update.php. (Note: This took about 24 hours to execute on the new server last time. If the ssh connection drops during the update, it does keep track of where it is in the process)
- Disable scripts/wikitrim.py until it has been updated to the new version of MediaWiki
Move Domain Name
- DONE: Update the domain name servers to point to the new server
- DONE: Wait for domain name to propagate (up to 48 hours).
- DONE: Update the following fields in localdefs.py:
- DONE: Update the following fields in LocalSettings.php
- DONE: dnf install postfix
- DONE: Edit /etc/postfix/main.cf:
- Set myhostname = isfdb.org
- Set mydomain = isfdb.org
- Set mail_spool_directory = /var/mail
- Set mynetworks_style = host
- Set mynetworks = 127.0.0.0/8 184.108.40.206
- Uncomment relay_domains = $mydestination
- Uncomment inet_interfaces = $myhostname, localhost
- DONE: systemctl start postfix
- DONE: systemctl enable postfix
- DONE: firewall-cmd --add-service=smtp --permanent
- DONE: dnf install mod_ssl
- DONE: systemctl restart httpd
- DONE: cd /etc/httpd/conf.d
- DONE: Create file isfdb.org.conf
- DONE: Add the following contents:
<VirtualHost *:80> ServerName server.isfdb.org DocumentRoot /var/www/html ServerAlias isfdb.org ErrorLog /var/www/error.log CustomLog /var/www/requests.log combined Redirect permanent / https://isfdb.org/ </VirtualHost>
- systemctl restart httpd
Certificate Support Software
- SKIP: dnf install epel-release (not a valid package any longer)
- DONE: dnf install snapd
- DONE: dnf install certbot
- DONE: dnf install python3-certbot-apache
- DONE: certbot --apache
- Select isfdb.org
- DONE: systemctl restart httpd
- DONE: certbot -d isfdb.org,www.isfdb.org --expand
- Select isfdb.org.conf
- This generates the error: The selected vhost would conflict with other HTTPS VirtualHosts within Apache. Please select another vhost or add ServerNames to your configuration. Could not install certificate
- But links to https://www.isfdb.org now work.
- Since the last Certbot operation failed (see above), it didn't install a cron job to auto-renew the certificate before it expires after 90 days. We will need to either get Certbot to work (probably best tested on isfdb2.org first) or set up a manual process to renew the certificate once every 3 months.
- DONE: Change PROTOCOL in localdefs.py to "https"
- DONE: Change $wgServer in /var/www/html/wiki/LocalSettings.php to use https
- DONE: Go to the wiki and type MediaWiki:Sidebar in the search window. Edit that page and change http to https
While everything at isfdb.org seems to be working, and external links at Wikipedia follow okay, the domain www.isfdb.org is still busted, and points to a GoDaddy parking page. Will debug that tomorrow. Alvonruff (talk) 22:14, 6 September 2022 (EDT)
- Just in case you aren't aware, as at 12:05 BST Add Publication (and likely other pages/scripts) is currently broken - browser dev tools implies it's failing to load a bunch of static images, .js files etc from 220.127.116.11 on a http:// URL, but as this is coming from an https:// page, my browser (Vivaldi) is blocking this. I guess whatever setting is used to generate the URLs will have to be switched to use https and a proper domain name ErsatzCulture (talk) 07:05, 7 September 2022 (EDT)
- EDIT: I see one of the steps below is "Enable editing at new server", so I guess this isn't expected to be working yet - however I assume that's related to the config flag that causes the "Editing facilities are currently offline" message, and the URLs are a different issue?
- We setup the new site such that Ahasuerus could do the official updates (much like the old site), but a side-effect of that during the domain move was me doing hand edits of files, without the generation that occurs during make, leading to some stray files with the IP address. So the next order of business will be for Ahasuerus to pick up the latest localdefs.py and do a make from the official branch. Alvonruff (talk) 07:21, 7 September 2022 (EDT)
- DONE: Uncomment the daily backup job in root's crontab (vi /etc/crontab)
- DONE: Confirm that the daily backup job ran successfully the following morning
- DONE: Uncomment 2 weekly image backup jobs in root's crontab
- DONE: Confirm that the weekly image backup jobs ran successfully on Sunday/Monday morning
- DONE: Update Ahasuerus's development server to download daily backups and weekly image backups automatically
- DONE: Create public backups on the development server and confirm that they contain the correct data
- DONE: Uncomment the nightly job in root's crontab
- DONE: Confirm that the nightly job ran successfully the following night
- DONE: Uncomment the weekly job in root's crontab
- DONE: Confirm that the weekly job ran successfully the following Sunday morning
- DONE: Keep the monthly job in root's crontab commented out
- DONE: Enable editing for moderators at new server
- SKIP: Update scripts/create_user.py to generate a pbkdf2 hash for the password. Since create_user.py is present specifically for users setting up a copy of the ISFDB without a wiki, submitlogin.py can be configured to use the old MD5 hashing scheme, which will match what's used in create_user.py.
- DONE for Alvonruff: Folks that have logins and data in their home directories on the current isfdb.org will need to move it to the new server, as it will be decommissioned. IP address of the current (old) isfdb.org is: 18.104.22.168
- DONE: Consider updating the old site to use the IP address, so that we can locate the stray content that made it into the wiki after editing was turned off, and the backups made. (Turning off editing only affects the ISFDB side, not the wiki side). Nexcess will likely decommission the old site pretty quickly.
- The URL for the old wiki is: http://22.214.171.124/wiki/index.php/Main_Page [Defunct as of 12/1/2022]
- DONE: On the old server, download, zip up and back up all files in /var/www/html. They include a variety of superseded PDF, HTML and SVG files generated during the 2000s and 2010s. Although they are no longer relevant, we should keep a copy just in case.
- DONE: Enable public editing at new server
- DONE: Change Fixer's settings to use HTTPS
- DONE: Notify The Encyclopedia of Science Fiction about the switch to HTTPS
- All SFE links to ISFDB pages have been modified to use HTTPS
- DONE: The TablePager class was deprecated. Instead of using class="TablePager" use class="wikitable" in all Wiki tables. Update the most visible Wiki pages manually and post on the Community Portal to explain the change.
- Update ISFDB_Downloads#Installation_and_Setup to reflect new Python/MySQL/etc versions and new installation steps
- DONE: Update ISFDB:FAQ#Why_do_browsers_say_that_the_connection_to_ISFDB_is_not_secure.3F to explain browser warnings about "mixed content" due to some displayed images coming from HTTP sites
- DONE: Update ISFDB:How To Create a Public Backup File with the MediaWiki 1.35+-compliant instructions currently displayed in a separate section at the bottom of the page
- DONE: In MediaWiki:Sidebar:
- change "http://www.isfdb.org/cgi-bin/index.cgi" to "https://www.isfdb.org/cgi-bin/index.cgi"
- replace "helppage|help", which links to MediaWiki Help, with https://isfdb.org/wiki/index.php/Help:Contents%7CHelp, which links to ISFDB-specific Help
- remove sitesupport-url|sitesupport
- DONE: Update Help:Using Templates and HTML in Note Fieldsand Help:Linking templates to use HTTPS instead of HTTPS
- DONE: Edit https://isfdb.org/wiki/index.php/MediaWiki:Ipb-default-expiry (which is blank out of the box) and enter the world "infinite" in the body of the page. This make "indefinite" the default value when blocking users.
- DONE: Activate ConfirmEdit and set up captcha challenges as follows:
- $wgMainCacheType = CACHE_ANYTHING;
- $wgCaptchaTriggers['edit'] = false;
- $wgCaptchaTriggers['create'] = false;
- $wgCaptchaTriggers['createtalk'] = true;
- $wgCaptchaTriggers['addurl'] = true;
- $wgCaptchaTriggers['createaccount'] = true;
- $wgCaptchaTriggers['badlogin'] = false;
- DONE: Add recognized third party domains which shouldn't trigger captcha challenges to MediaWiki:Captcha-addurl-whitelist
- DONE: Change the value of upload_max_filesize in /etc/php.ini from "2MB" to "200KB"
The following templates need to be converted from http to https:
- Template:Isfdb series - Updated
- Template:Isfdb contents - Updated
- Template:Isfdb name - Updated
- Template:Isfdb title - Updated
- Linux: 5.14.18-100.fc33.x86_64
- Apache: Apache/2.4.37
- MySQL: 8.0.27
- Python: 2.7.18
- PHP: 7.4.26
- MediaWiki: 1.35.6