The production whimsy.apache.org server is managed by Puppet, and is automatically updated whenever commits are made to the master branch of this repository. Thus code changes here are reflected in the production server within a few minutes. In the event of a major server crash, the infra team simply re-deploys the whole VM from Puppet.
Committers: please test changes to end-user critical scripts before committing to master!
To deploy a completely new whimsy VM, see Manual Steps.
Application developers may need to know where different things are configured:
rake update to update various gem or ruby settingsThus, in less than 5 minutes from any git push, the server is running the new code!
The Whimsy VM runs Ubuntu 20.04 and is fully managed by Puppet using the normal methods Apache infra uses for managing servers. Note however that management of Whimsy code and tools is a PMC responsibility.
The puppet definition is contained in the following files (these are private files and need a login):
https://github.com/apache/infrastructure-p6/blob/production/data/nodes/whimsy-vmN.apache.org.yaml (Includes modules, software, vhosts, ldap realms, and httpd.conf)
https://github.com/apache/infrastructure-p6/tree/production/modules/vhosts_whimsy/lib/puppet/functions (macro functions used above)
https://github.com/apache/infrastructure-p6/blob/production/modules/whimsy_server/manifests/init.pp (Defines various tools and directories used in some tools)
https://github.com/apache/infrastructure-p6/blob/production/modules/whimsy_server/manifests/cronjobs.pp (Cronjobs control when /public/*.json is built and code and mail updates)
Before pushing any changes here, understand the Apache Infra puppet workflow and test:
https://cwiki.apache.org/confluence/display/INFRA/Git+workflow+for+infrastructure-puppet+repo To understand the high-level workflow for puppet changes.
https://github.com/apache/infrastructure-puppet-kitchen#readme
addition to Make modules useable step:
rm -rf zmanda_asf
mkdir -p zmanda_asf/manifests
echo "class zmanda_asf::client (){}" > zmanda_asf/manifests/client.pp
https://github.com/apache/infrastructure-p6/blob/production/modules/vhosts_whimsy/README.md This details the changes to default puppet we use for Whimsy.
The following additional steps are required to get a new Whimsy VM up and running - these are only needed for a new deployment.
Ensure that the IP address is static, and has been added to the list of allowed mail relays
check that the Puppet node yaml file has the appropriate version settings for ruby, gems and passenger. This may require trial and error.
Optionally set up an initial SSL certificate just for the new node, i.e. excluding whimsy.apache.org. This is to allow for initial testing.
certbot certonly from root, select option (2) - standalone.Set up a new SSL cert (also works if the individual node cert has already been set up): this can be done using some files that should be set up by Puppet. You will need root access to whimsy.apache.org as well in order to set up the challenge.
The SVN settings should now be set up in whimsy-vm5 and later (Puppet 6)
check reporter.apache.org is reachable (depends on which datacenter it is in; see INFRA-25403):
curl -I https://reporter.apache.orgcheck that board-agenda-websocket.service is running:
sudo systemctl status board-agenda-websocket.service - this should show the service is running and has been up for some whilecurl -N localhost:34234 - should produce ‘curl: (52) Empty reply from server’ or similarsudo systemctl stop/start board-agenda-websocket.serviceUpdate the following cron scripts now under https://github.com/apache/infrastructure-p6/tree/production/modules/qmail_asf/files/apmail/bin (but may revert here) https://svn.apache.org/repos/infra/infrastructure/apmail/trunk/bin:
Add the following mail subscriptions (see apmail/trunk/bin/whimsy_subscribe.sh):
board@whimsy-vmN.apache.org to board@apache.org.members@whimsy-vmN.apache.org to members@apache.org.secretary@whimsy-vmN.apache.org to the secretary@apache.org alias.www-data@whimsy-vmN.apache.org to private-allow@whimsical.apache.org. (Cron daemon)www-data@whimsy-vmN.apache.org to notifications-allow@whimsical.apache.org. (Cron daemon) [Note that the allow subscriptions must use the physical host name; the others can use a DNS equivalent.]Verify that email can be sent to non-apache.org email addresses.
check that mail is being delivered to the /srv/mail directory tree
check that mail subscriptions are being updated under /srv/subscriptions
if pip3 show img2pdf doesn't show version 0.3.1 or later:
pip3 install --upgrade img2pdfEnsure that gitpubsub is working. Check that updates to whimsy git are pulled within 5 minutes or less
Synchronising data: Whimsy keeps some data locally on the server. This needs to be copied across prior to switchover. Using the www-data user, copy over the following directories from the previous whimsy-vm server:
/srv/agenda/srv/gpg - this contains the public key ring used to check ICLA signatures/srv/mail - Note that the /srv/mail/* directory entries may be different between hosts. This is because the full email content is used if no Message-Id is found (happens occasionally), and final delivery routes in the mail headers will vary. However, rather than try and merge the files, it is simpler to do a full copy. Using the whimsysvn user, copy the following file:/srv/svn/committee-info_last_revision.txt - needed by pubsub-ci-email to track last reported change Note: if the copies are done by root, generally file ownership will be preserved.The mail server is unlikely to change, but if it does, rsync auth will need to be set up.
sudo -Hiu apmailssh-keygen -t ecdsa -b 521.ssh/id_ecdsa.pub to the Puppet file data/nodes/whimsy-vmN.apache.org.yaml under the whimsy_server::procmail::apmail_keycontent key.