Skip to content

Restore WP1 from a backup

Jump to bottom
Travis Briggs edited this page Oct 17, 2024 · 17 revisions

Aside from a Zimfarm instance which is not covered in this documentation, WP1 relies on a compute instance (mwcurator) and a Trove DB.

This documentation details the procedure to restore both components.

Recreate mwcurator cloud VPS instance and database (if necessary)

You may need to request additional quota for these steps if the original machines are still running.

  1. Go to https://horizon.wikimedia.org/
  2. Select mwoffliner as your project (you must be an admin of mwoffliner)
  3. (Optional) Re-create the application server
    1. Under Compute -> Instances
    2. Select “Launch Instance”
    3. Under “Source” use debian-11-bullseye
    4. Under “Flavor” choose an appropriate number of vCPUs, RAM and disk
  4. Re-create the database server
    1. Under Database -> Instances
    2. Select “Launch Instance”
    3. Under “Volume Size”, choose an appropriate number of GB to handle the size of the database (60 GB at the time of writing)
    4. Under “Datastore”, choose mariadb
    5. Under “Flavor” choose an appropriate number of vCPUs, RAM and disk (the original database server was g4.cores2.ram4.disk20 | 4GB RAM).
    6. Under “Initialize Databases”
      1. Initial Databases: enwp10_prod
      2. Initial Admin User: wp1
      3. Password: generate a secure password externally and paste in. Write it down, you'll need it later of course.
  5. Under “Advanced”
    1. Configuration Group: wp1-db-import (will make the db import faster)

Import the database from a backup (if necessary)

Follow these directions to access the database borg backups. You will need BitWarden credentials. The name of the backup store is wp1db.

NOTE: This step should be performed on a local machine. The application server will not have enough storage to hold the database backup files.

Find the hostname of your new Trove (database) instance that you created above. This is in Horizon under Databases -> Instances. Click the instance name and you should see something like this:

image

Set up an SSH tunnel to that database host, through your toolforge account. You can use the command:

ssh -L 4711:ofi3zurkdgo.svc.trove.eqiad1.wikimedia.cloud:3306 login.toolforge.org

NOTE: See the Wikimedia Cloud docs for more information on setting up the tunnel. You will need to have Tools or Toolforge credentials set up in your ~/.ssh/config file for this to work. See this help file for details on setting up SSH.

Find your backup file. Mine was in /data/restore/root/.borgmatic/mysql_databases/tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud/ but if the production trove hostname changes, yours could be different.

Use the following command, entering the password you chose above, to start restoring the database:

mysql -h 127.0.0.1 -P 4711 -u wp1 -p enwp10_prod < data/restore/root/.borgmatic/mysql_databases/tdlqt33y3nt.svc.trove.eqiad1.wikimedia.cloud/enwp10_prod

In testing, this took about 7 hours.

Set up the replacement application server (if necessary)

  1. Install docker using these directions.
  2. Create the following directories:
sudo mkdir -p /data/wp1bot /data/code/ /data/wp1bot/db/ /srv/log/wp1bot/ /srv/data/wp1bot/

Note that the /srv directory is an NFS mount. The /data directory, on the original server, was an attached Cinder volume that was needed for other operations on that server, but is not needed for WP1. These paths are hardcoded in docker-compose.yml but could be updated there if you're having trouble creating the directories.

  1. cd /data/code and checkout the wp1 repository: sudo git clone https://github.com/openzim/wp1.git
  2. Copy the example credentials: sudo cp /data/code/wp1/wp1/credentials.py.example /data/wp1bot/credentials.py
  3. Edit the file (sudo nano /data/wp1bot/credentials.py), providing the necessary values (commented out) and deleting the keys: Environment.DEVELOPMENT, Environment.TEST, and the existing empty Environment.PRODUCTION key.
    1. WIKIDB is the Wikipedia replica db, also known as enwiki_p. The credentials are your project toolforge credentials, which can be found buy logging into ssh login.toolforge.org and reading the file replica.my.cnf.
    2. WP10DB is the application database that you restored to Trove. User should be wp1, password is the password you set when you restored, host is the Trove host (ofi3zurkdgo.svc.trove.eqiad1.wikimedia.cloud in our example). You can leave out the port (it defaults to 3306 which is where Trove is running). 3: 'REDIS': { 'host': 'redis', 'port': 6379 }
    3. 'API': { 'user': 'WP 1.0 bot@WP_1.0_Bot', 'pass': ??? }, TODO: figure out how we would find/reset this password.
    4. 'MWOAUTH': If you've lost this credential, you will need to register a new OAuth application. The client secret cannot be recovered from any Wikimedia web interface.
    5. 'SESSION': { 'secret_key': 'any sufficiently long string of random characters, like a password' }
    6. TODO: Finish this

Mapping the IP/DNS

The application server is mapped to a floating IP address, which allows it to be mapped to its domain, wp1.openzim.org. If you are restoring to a new application server, you should go to the floating IP management screen and detach the IP from the existing (crashed) server and reattach it to your new server.

Clone this wiki locally