Difference between revisions of "User:Alvonruff/ISFDB2 Notes"

From ISFDB
Jump to navigation Jump to search
 
(48 intermediate revisions by the same user not shown)
Line 10: Line 10:
 
* dnf install bzip2.x86_64
 
* dnf install bzip2.x86_64
 
* dnf install wget
 
* dnf install wget
 +
* dnf install mod_dav_svn subversion
 +
* dnf install telnet
 +
* dnf install ImageMagick
  
 
==Apache==
 
==Apache==
Line 15: Line 18:
 
* dnf install httpd
 
* dnf install httpd
 
* firewall-cmd --add-service=http --add-service=https --permanent
 
* firewall-cmd --add-service=http --add-service=https --permanent
−
* service httpd start
+
* Add to etc/httpd/httpd.conf: LoadModule dir_module modules/mod_dir.so
 +
* Change etc/httpd/httpd.conf: DirectoryIndex /cgi-bin/index.cgi index.html index.html.var
 +
* systemctl enable httpd
 +
* systemctl start httpd
  
 
==MySQL==
 
==MySQL==
Line 35: Line 41:
 
* dnf install python2-devel.x86_64
 
* dnf install python2-devel.x86_64
 
* dnf install mysql-devel.x86_64  
 
* dnf install mysql-devel.x86_64  
−
* pip2 install mysqlclient''
+
* pip2 install mysqlclient
 +
* alternatives --set python /usr/bin/python2
  
−
==Versions==
+
==PHP==
−
* Linux: 4.18.0-240.15.1.el8_3.x86_64 x86_64
+
* dnf module reset php
−
* Apache: Apache/2.4.37 (AlmaLinux)
+
* dnf module enable php:7.4
−
* MySQL: 8.0.26
+
* dnf install php php-common php-opcache php-cli php-gd php-curl php-mysqlnd
−
* Python: 2.7.18
+
* systemctl start php-fpm
 +
* systemctl enable php-fpm
 +
* php -v
 +
* systemctl stop httpd
 +
* systemctl start httpd
  
−
==Charset Experiments==
+
==Mail==
 +
* dnf install postfix
 +
* Edit ''/etc/postfix/main.cf'':
 +
** Set myhostname = isfdb2.org
 +
** Set mydomain = isfdb2.org
 +
** Set mail_spool_directory = /var/mail
 +
** Set mynetworks_style = host
 +
** Set mynetworks = 127.0.0.0/8 104.225.208.158
 +
** Uncomment relay_domains = $mydestination
 +
** Uncomment inet_interfaces = $myhostname, localhost
 +
* systemctl start postfix
 +
* systemctl enable postfix
 +
* firewall-cmd --add-service=smtp --permanent
  
−
I have a python script for generating Wikipedia article stubs from the ISFDB tables in MySQL. It was run on both isfdb.org and isfdb2.org. Running diff on the outputs shows:
+
==MediaWiki==
 +
* cd to /var/www/html
 +
* Fetch the current stable long-term support release of MediaWiki:
 +
** wget https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.6.zip
 +
* unzip mediawiki-1.35.6.zip
 +
* mv mediawiki-1.35.6 wiki
 +
* cd wiki && place the old isfdb version of LocalSettings.php here
 +
* Edit LocalSettings.php and make the follow changes:
 +
** Change wgServer
 +
** Change wgDBuser
 +
** Change wgDBpassword
 +
** Comment out the ConfirmEdit extension
 +
** Comment out the SyntaxHighlight extension
 +
** Comment out the SVGtag extension
 +
* mkdir wiki/images
 +
* chown apache images
 +
* chgrp apache images
 +
* Copy all image subdirectory content from isfdb.org:/var/www/html/wiki/images to isfdb2.org
 +
* php maintenance/update.php
 +
** Wait a very long time for the update to finish (about 3.5 hours). This will perform hundreds of thousands of revision updates, but it issues a constant stream of progress lines. There is a web-based alternative, but I recommend against that, as the long processing time will generate a 504 error from Apache.
 +
** It helps to ssh into the system with the "-o ServerAliveInterval=600" option, which will generate a null keep alive packet every 10 minutes, keeping the session alive.
 +
* cp isfdb.gif to wiki/skins/common/images
 +
* Add the following lines to the end of LocalSettings.php and remove the old commented out versions:
 +
** wfLoadSkin( 'Vector' );
 +
** wfLoadExtension( 'ConfirmEdit' );
 +
** wfLoadExtension( 'SyntaxHighlight_GeSHi' );
 +
** Uncomment and change to: $wgImageMagickConvertCommand = "/usr/bin/convert";
 +
** Uncomment and change to: $wgLogo = "$wgStylePath/common/images/isfdb.gif";
  
−
<pre>
 
−
4c4
 
−
< | name        = Philip José Farmer
 
−
---
 
−
> | name        = Philip José Farmer
 
−
</pre>
 
−
 
−
If, however, the files are brought up in the vim text editor, they both appear to be correct. If I pull the name string out of each file and run ''od -X --endian=big STRING_FILE'', the results are (with hand annotation):
 
  
 +
=== Isfdb2-Specific Changes===
 +
* Edit the navbar by searching for MediaWiki:Sidebar, and change entries to:
 
<pre>
 
<pre>
−
0000000 5068696c 6970204a 6f73e920 4661726d        Phil  ip J  osé  Farm
+
* navigation
−
0000020 65720a00
+
** http://isfdb2.org/cgi-bin/index.cgi|ISFDB Home Page
−
 
+
** mainpage|Main Wiki Page
−
0000000 5068696c 6970204a 6f73c3a9 20466172        Phil  ip J  osé  Far
+
** https://isfdb2.org/wiki/index.php/ISFDB:Community_Portal|portal
−
0000020 6d65720a
+
** currentevents-url|currentevents
 +
** recentchanges-url|recentchanges
 +
** https://isfdb2.org/wiki/index.php/Help:Contents|help
 +
** sitesupport-url|sitesupport
 
</pre>
 
</pre>
  
−
This generates 2 questions:  
+
=== To Do ===
−
# Why is the output different on the two systems (python is clearly generating more and different characters), and
+
* Post isfdb.org update: The TablePager class was deprecated, so instead of using '''class="TablePager"''' use '''class="wikitable"'''
−
# Why does the file appear correct when viewed inside a vim session?
 
  
−
The answer to the first question is highlighted by the answer to the second question: vim uses utf-8 as it's default charset. I made the following changes to ''common/isfdb.py'':
+
=== Performing a Subsequent Update ===
 +
If after installation, you desire to update the database to the latest available from isfdb.org, you'll find that the mediaWiki update script isn't entirely successful. When overwriting the database with a new version of dbdump, it only drops tables that are being updated, so any new tables (like those from the mediaWiki upgrade) are left in place. This leaves the wiki state in an indeterminate state. In order to return the database to it's original isfdb state, execute the following sql commands before sourcing the dbdump backup:
  
−
[1] Altered the html content type output from:
 
 
<pre>
 
<pre>
−
print 'Content-type: text/html; charset=%s\n' % UNICODE
+
use isfdb;
−
</pre>
+
drop table mw_actor;
−
to
+
drop table mw_archive;
−
<pre>
+
drop table mw_blobs;
−
print 'Content-type: text/html; charset=%s\n' % "UTF-8"
+
drop table mw_bot_passwords;
 +
drop table mw_brokenlinks;
 +
drop table mw_category;
 +
drop table mw_categorylinks;
 +
drop table mw_change_tag;
 +
drop table mw_change_tag_def;
 +
drop table mw_comment;
 +
drop table mw_content;
 +
drop table mw_content_models;
 +
drop table mw_cur;
 +
drop table mw_externallinks;
 +
drop table mw_filearchive;
 +
drop table mw_image;
 +
drop table mw_imagelinks;
 +
drop table mw_interwiki;
 +
drop table mw_ip_changes;
 +
drop table mw_ipblocks;
 +
drop table mw_ipblocks_old;
 +
drop table mw_ipblocks_restrictions;
 +
drop table mw_iwlinks;
 +
drop table mw_job;
 +
drop table mw_l10n_cache;
 +
drop table mw_langlinks;
 +
drop table mw_links;
 +
drop table mw_linkscc;
 +
drop table mw_log_search;
 +
drop table mw_logging;
 +
drop table mw_math;
 +
drop table mw_module_deps;
 +
drop table mw_objectcache;
 +
drop table mw_oldimage;
 +
drop table mw_page;
 +
drop table mw_page_props;
 +
drop table mw_page_restrictions;
 +
drop table mw_pagelinks;
 +
drop table mw_protected_titles;
 +
drop table mw_querycache;
 +
drop table mw_querycache_info;
 +
drop table mw_querycachetwo;
 +
drop table mw_recentchanges;
 +
drop table mw_redirect;
 +
drop table mw_revision;
 +
drop table mw_revision_actor_temp;
 +
drop table mw_revision_comment_temp;
 +
drop table mw_searchindex;
 +
drop table mw_site_identifiers;
 +
drop table mw_site_stats;
 +
drop table mw_sites;
 +
drop table mw_slot_roles;
 +
drop table mw_slots;
 +
drop table mw_templatelinks;
 +
drop table mw_text;
 +
drop table mw_trackbacks;
 +
drop table mw_transcache;
 +
drop table mw_updatelog;
 +
drop table mw_uploadstash;
 +
drop table mw_user;
 +
drop table mw_user_former_groups;
 +
drop table mw_user_groups;
 +
drop table mw_user_newtalk;
 +
drop table mw_user_properties;
 +
drop table mw_user_rights;
 +
drop table mw_validate;
 +
drop table mw_watchlist;
 +
drop table mw_watchlist_expiry;
 
</pre>
 
</pre>
  
−
[2] Altered the meta tag string from:
+
==Versions==
−
<pre>
+
* Linux: 4.18.0-240.15.1.el8_3.x86_64 x86_64
−
print '<meta http-equiv="content-type" content="text/html; charset=%s" >' % UNICODE
+
* Apache: Apache/2.4.37 (AlmaLinux)
−
</pre>
+
* MySQL: 8.0.26
−
to:
+
* Python: 2.7.18
−
<pre>
+
* PHP: 7.4.19
−
print '<meta charset="UTF-8"/>'
+
* MediaWiki: 1.35.6
−
</pre>
 
−
 
 
−
The output now appears normal. Need to run with this for a while to see if there are any untoward side effects. This also does not answer the question as to why this runs fine on the original isfdb.org
 

Latest revision as of 06:32, 19 June 2022

The isfdb2 staging system is a minimal system, with few packages installed, which uses dnf instead of apt-get.

Prerequisites

The staging system a minimum configuration AlmaLinux system, which is a variant of Fedora Core. It's really intended for tight cloud installations, so almost everything is missing, and installation of packages is done with yum/dnf.

  • dnf install gcc
  • dnf install make
  • dnf install tar
  • dnf install zip.x86_64
  • dnf install bzip2.x86_64
  • dnf install wget
  • dnf install mod_dav_svn subversion
  • dnf install telnet
  • dnf install ImageMagick

Apache

  • dnf install httpd
  • firewall-cmd --add-service=http --add-service=https --permanent
  • Add to etc/httpd/httpd.conf: LoadModule dir_module modules/mod_dir.so
  • Change etc/httpd/httpd.conf: DirectoryIndex /cgi-bin/index.cgi index.html index.html.var
  • systemctl enable httpd
  • systemctl start httpd

MySQL

  • dnf update
  • dnf module enable mysql:8.0
  • dnf install @mysql
  • systemctl enable mysqld
  • systemctl start mysqld
  • Issue: mysql
  • While in mysql, issue the command: create database isfdb;
  • While in mysql, issue the command: use isfdb;
  • While in mysql, issue the command: alter database isfdb character set latin1 collate latin1_swedish_ci;
  • While in mysql, issue the command: source <<backupfile>>;'
  • GRANT ALL PRIVILEGES ON isfdb.* TO 'isfdb1'@'localhost';

Python 2.7.18

  • dnf install python2.x86_64
  • dnf install python2-devel.x86_64
  • dnf install mysql-devel.x86_64
  • pip2 install mysqlclient
  • alternatives --set python /usr/bin/python2

PHP

  • dnf module reset php
  • dnf module enable php:7.4
  • dnf install php php-common php-opcache php-cli php-gd php-curl php-mysqlnd
  • systemctl start php-fpm
  • systemctl enable php-fpm
  • php -v
  • systemctl stop httpd
  • systemctl start httpd

Mail

  • dnf install postfix
  • Edit /etc/postfix/main.cf:
    • Set myhostname = isfdb2.org
    • Set mydomain = isfdb2.org
    • Set mail_spool_directory = /var/mail
    • Set mynetworks_style = host
    • Set mynetworks = 127.0.0.0/8 104.225.208.158
    • Uncomment relay_domains = $mydestination
    • Uncomment inet_interfaces = $myhostname, localhost
  • systemctl start postfix
  • systemctl enable postfix
  • firewall-cmd --add-service=smtp --permanent

MediaWiki

  • cd to /var/www/html
  • Fetch the current stable long-term support release of MediaWiki:
  • unzip mediawiki-1.35.6.zip
  • mv mediawiki-1.35.6 wiki
  • cd wiki && place the old isfdb version of LocalSettings.php here
  • Edit LocalSettings.php and make the follow changes:
    • Change wgServer
    • Change wgDBuser
    • Change wgDBpassword
    • Comment out the ConfirmEdit extension
    • Comment out the SyntaxHighlight extension
    • Comment out the SVGtag extension
  • mkdir wiki/images
  • chown apache images
  • chgrp apache images
  • Copy all image subdirectory content from isfdb.org:/var/www/html/wiki/images to isfdb2.org
  • php maintenance/update.php
    • Wait a very long time for the update to finish (about 3.5 hours). This will perform hundreds of thousands of revision updates, but it issues a constant stream of progress lines. There is a web-based alternative, but I recommend against that, as the long processing time will generate a 504 error from Apache.
    • It helps to ssh into the system with the "-o ServerAliveInterval=600" option, which will generate a null keep alive packet every 10 minutes, keeping the session alive.
  • cp isfdb.gif to wiki/skins/common/images
  • Add the following lines to the end of LocalSettings.php and remove the old commented out versions:
    • wfLoadSkin( 'Vector' );
    • wfLoadExtension( 'ConfirmEdit' );
    • wfLoadExtension( 'SyntaxHighlight_GeSHi' );
    • Uncomment and change to: $wgImageMagickConvertCommand = "/usr/bin/convert";
    • Uncomment and change to: $wgLogo = "$wgStylePath/common/images/isfdb.gif";


Isfdb2-Specific Changes

  • Edit the navbar by searching for MediaWiki:Sidebar, and change entries to:
* navigation
** http://isfdb2.org/cgi-bin/index.cgi|ISFDB Home Page
** mainpage|Main Wiki Page
** https://isfdb2.org/wiki/index.php/ISFDB:Community_Portal|portal
** currentevents-url|currentevents
** recentchanges-url|recentchanges
** https://isfdb2.org/wiki/index.php/Help:Contents|help
** sitesupport-url|sitesupport

To Do

  • Post isfdb.org update: The TablePager class was deprecated, so instead of using class="TablePager" use class="wikitable"

Performing a Subsequent Update

If after installation, you desire to update the database to the latest available from isfdb.org, you'll find that the mediaWiki update script isn't entirely successful. When overwriting the database with a new version of dbdump, it only drops tables that are being updated, so any new tables (like those from the mediaWiki upgrade) are left in place. This leaves the wiki state in an indeterminate state. In order to return the database to it's original isfdb state, execute the following sql commands before sourcing the dbdump backup: 

use isfdb;
drop table mw_actor;
drop table mw_archive;
drop table mw_blobs;
drop table mw_bot_passwords;
drop table mw_brokenlinks;
drop table mw_category;
drop table mw_categorylinks;
drop table mw_change_tag;
drop table mw_change_tag_def;
drop table mw_comment;
drop table mw_content;
drop table mw_content_models;
drop table mw_cur;
drop table mw_externallinks;
drop table mw_filearchive;
drop table mw_image;
drop table mw_imagelinks;
drop table mw_interwiki;
drop table mw_ip_changes;
drop table mw_ipblocks;
drop table mw_ipblocks_old;
drop table mw_ipblocks_restrictions;
drop table mw_iwlinks;
drop table mw_job;
drop table mw_l10n_cache;
drop table mw_langlinks;
drop table mw_links;
drop table mw_linkscc;
drop table mw_log_search;
drop table mw_logging;
drop table mw_math;
drop table mw_module_deps;
drop table mw_objectcache;
drop table mw_oldimage;
drop table mw_page;
drop table mw_page_props;
drop table mw_page_restrictions;
drop table mw_pagelinks;
drop table mw_protected_titles;
drop table mw_querycache;
drop table mw_querycache_info;
drop table mw_querycachetwo;
drop table mw_recentchanges;
drop table mw_redirect;
drop table mw_revision;
drop table mw_revision_actor_temp;
drop table mw_revision_comment_temp;
drop table mw_searchindex;
drop table mw_site_identifiers;
drop table mw_site_stats;
drop table mw_sites;
drop table mw_slot_roles;
drop table mw_slots;
drop table mw_templatelinks;
drop table mw_text;
drop table mw_trackbacks;
drop table mw_transcache;
drop table mw_updatelog;
drop table mw_uploadstash;
drop table mw_user;
drop table mw_user_former_groups;
drop table mw_user_groups;
drop table mw_user_newtalk;
drop table mw_user_properties;
drop table mw_user_rights;
drop table mw_validate;
drop table mw_watchlist;
drop table mw_watchlist_expiry;

Versions

  • Linux: 4.18.0-240.15.1.el8_3.x86_64 x86_64
  • Apache: Apache/2.4.37 (AlmaLinux)
  • MySQL: 8.0.26
  • Python: 2.7.18
  • PHP: 7.4.19
  • MediaWiki: 1.35.6
Retrieved from "https://isfdb.org/wiki/index.php?title=User:Alvonruff/ISFDB2_Notes&oldid=636318"