• [email protected]
  • email address for sending email from TOD (e.g. for Jenkins)
  • ask [email protected] if you would like the password
• email list: [email protected]
  • email list for receiving email (e.g. if a backup fails, it will email this group, which will result in emails being sent to one or more individuals responsible for system admin)
  • follow the instructions here to request being added to this list: https://groups.google.com/forum/?hl=en#!forum/bradnerlab-systemadminemail [email protected] if you would like to be added to the group

• http://tod.dfci.harvard.edu/jenkins
  • install documented at misc-2
• tip: use for remote system admin, e.g.
  • ssh -L 9001:tod.dfci.harvard.edu:80 [email protected]
  • you can now access jenkins in a local web browser at http://localhost:9001/jenkins/
• only the bradneradmin user currently has permission to create/configure jobs
• the jenkins user has sudo permission to run some backup scripts in /root/backup-scripts/
  • the jenkins user should still be safe to use for other jobs, and it would be fine to allow any authenticated user access to configure/create jenkins jobs
    • configuration is carefully setup to NOT allow the jenkins user to run arbitrary backup commands (which would allow a user to backup files they don't have access to to arbitrary locations, or restore files they wouldn't normally have access to)
• [email protected] email used for sending messages

• summary: roughly every 48 hours rsync copies to /mnt/bradner/data
  • starts 1 AM every Tuesday, Thursday, and Saturday, run by Jenkins: http://tod.dfci.harvard.edu/jenkins/job/Backup_TOD/ (only viewable by the bradneradmin Jenkins user)
  • error emails sent to [email protected]
  • /mnt/bradner/data is independently incrementally backed up by IT

• backups used to use rdiff-backup, and John was concerned that renaming it might cause a huge incremental backup, so it is left misleadingly named rdiff-backup instead of rsync-backup
• Backup & Recovery provides an excellent concise section "Using rsync" but you can also reference the man page
• note that all backups used to be on crusader, so many older examples reference that
• mysql backup with automysqlbackup
  • backups up all mysql databases (as listed by "SHOW DATABASES") to /mnt/bradner/data/backup/automysqlbackup/
  • example restore commands
    1. copy backup to server that we want to restore to (can be TOD, but note that it will overwrite any tables being restored)

jdm@tod:/crusader/backup/automysqlbackup/daily$ sudo scp meta_analysis/meta_analysis_2013-11-07_03h01m.Thursday.sql.gz [email protected]:/Users/jdimatteo/DanaFarber/copied_from_tod/

    2. unzip, and [load in via mysql command](http://www.howtoforge.com/faq/6_4_en.html)

jd-mba:copied_from_tod jdimatteo$ gunzip meta_analysis_2013-11-07_03h01m.Thursday.sql.gz
jd-mba:copied_from_tod jdimatteo$ mysql -uroot meta_analysis < meta_analysis_2013-11-07_03h01m.Thursday.sql
jd-mba:copied_from_tod jdimatteo$

* background on automysqlbackup:
    * http://sourceforge.net/projects/automysqlbackup/
    * http://www.howtoforge.com/creating-mysql-backups-with-automysqlbackup
* installation notes (note that I disabled the cron job, and later removed the performance_schema table to resolve an error as described [here](http://askubuntu.com/questions/134670/how-do-i-stop-automysqlbackup-throwing-lock-tables-error))

root@tod:~# apt-get install automysqlbackup
root@tod:~# cp /etc/default/automysqlbackup gunk/
root@tod:~# vim /etc/default/automysqlbackup
root@tod:~# diff gunk/automysqlbackup /etc/default/automysqlbackup 
31c31
< BACKUPDIR="/var/lib/automysqlbackup"
---
> BACKUPDIR="/crusader/backup/automysqlbackup"
39c39
< MAILCONTENT="quiet"
---
> MAILCONTENT="stdout"
root@tod:~# mv /etc/cron.daily/automysqlbackup gunk/

• the actual commands to do the backups are in scripts in /root/backup-scripts, which the Jenkins user has sudo access to

The rsync backups to /mnt/bradner/data/backup/rdiff-backup/ are simple snapshots. Files can be restored using (cp, cp -R, cp -a)[http://unix.stackexchange.com/questions/44967/difference-between-cp-r-and-cp-a], etc. You might consider using rsync to restore as well.

Incremental backups will need to go through IT.

Note that this example shows how to add an additional filesystem to be backed up. The current scheme is to have an individual job for each filesystem, so that the backups may occur simultaneously. If you are adding just an additional directory to be backed up on a file system already being backed up, you can just edit the already existing backup job for that filesystem instead of adding a new job.

1. Log into Jenkins as the bradneradmin user
2. Click the "New Job" button in the top left corner of the Jenkins Dashboard
3. Select "Copy existing Job", enter an individual backup job in the "Copy existing Job" field, and specify a name (ideally following the same convention of the other jobs, starting with "Backup_TOD_" followed by the file system), and click "OK"
4. The configuration page will open. Scroll down to the "Build Triggers" section, select "Build after other projects are built" and enter "Backup_TOD". (The Backup_TOD job just triggers the individual backup jobs and this allows your new job to run automatically on the full backup TOD schedule, currently roughly every 48 hours.)
5. Scroll to the "Build" section, and replace the command to run the actual backup script (e.g. changing "sudo /root/backup-scripts/backup-ark.sh" to "sudo /root/backup-scripts/backup-grail.sh". This script needs to be manually created on TOD and the jenkins user needs to be given explicit permission to run them as root, e.g.

jdm@tod:~$ sudo su
[sudo] password for jdm: 
root@tod:/ark/home/jdm# cd
root@tod:~# cat /root/backup-scripts/backup-ark.sh
#!/bin/bash

rdiff-backup --backup-mode --exclude-other-filesystems /ark/ /crusader/backup/rdiff-backup/ark      
root@tod:~# cp /root/backup-scripts/backup-ark.sh /root/backup-scripts/backup-grail.sh
root@tod:~# vim /root/backup-scripts/backup-grail.sh
root@tod:~# cat /root/backup-scripts/backup-grail.sh
#!/bin/bash

rdiff-backup --backup-mode --exclude-other-filesystems /grail/ /crusader/backup/rdiff-backup/grail
root@tod:~# cp /etc/sudoers Gunk/
root@tod:~# visudo # adding backup-grail.sh to BACKUPS list
root@tod:~# diff --context=6 Gunk/sudoers /etc/sudoers
*** Gunk/sudoers        2013-12-10 21:19:20.195641184 -0500
--- /etc/sudoers        2013-12-10 21:19:46.519642748 -0500
***************
*** 17,28 ****
--- 17,29 ----
  # Cmnd alias specification
  Cmnd_Alias    BACKUPS = /root/backup-scripts/backup-ark.sh,             \
                          /root/backup-scripts/backup-raider.sh,          \
                          /root/backup-scripts/backup-d0-0-bradnerlab.sh, \
                          /root/backup-scripts/backup-ifs-bradner.sh,     \
                          /root/backup-scripts/backup-root-directory.sh,  \
+                         /root/backup-scripts/backup-grail.sh,           \
                          /usr/sbin/automysqlbackup ""
  
  # User privilege specification
  root  ALL=(ALL:ALL) ALL
  jdm     ALL=(ALL) ALL
  jenkins ALL=NOPASSWD: BACKUPS 
root@tod:~# 

6. Note that the "Copy existing Job" already filled in a "Post-build Action" of emailing system administrators if the backup job script returns a non-zero error code
7. Click the "Save" button on the bottom of the page to save the new backup job
8. Click the "Build Now" button on Jenkins to manually run the backup job to verify it runs without error.
9. Review the Jenkins "Console Output" and check the backup directory to verify things look right
10. Note that the job will now automatically run in the future whenever "Backup_TOD" runs
11. Add a row to the Backups - Overview table on this wiki to document the backup

• How do I disable or delete a backup job?
  • To disable, log into Jenkins as the bradneradmin user, select the backup job you'd like to disable, and click the "Disable Project" button on the right, e.g.
  • To delete, click the "Delete Project" button on the left. You can't undelete a Jenkins job, so you may want to just disable a job instead of deleting it. If you do delete the job, you might want to also delete any associated backup scripts on TOD in /root/backup-scripts/ and remove the special sudo privileges given to Jenkins to run any deleted scripts.

https://github.com/BradnerLab/SystemAdmin/issues?labels=backup&page=1&state=open

• some apt-get installed software on TOD (this isn't intended to be a complete list)
  • automysqlbackup
  • realpath
  • iotop
• some python packages installed with easy_install:
  • redis
  • sphinx
  • bokeh
  • requests
• nice bash checker:
  • http://www.shellcheck.net/#
• TOD hardware
  • Supermicro H8QG6 Motherboard

jdm@tod:~/Gunk/speedtest-cli$ sudo dmidecode -t 2
[sudo] password for jdm: 
# dmidecode 2.11
SMBIOS 2.6 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Supermicro
        Product Name: H8QG6
        Version: 1234567890
        Serial Number: WM135S600672
        Asset Tag: 1234567890
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: 1234567890
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0

jdm@tod:~/Gunk/speedtest-cli$ 

* 32 physical CPU cores running at 2.1 GHz (64 hyperthreading cores)
    * AMD Opteron(TM) Processor 6272
    * 4 CPUs, each with 8 cores, each with two threads
    * cpu scales speed down when low load

jdm@tod:~$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                64
On-line CPU(s) list:   0-63
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             4
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            21
Model:                 1
Stepping:              2
CPU MHz:               1400.000
BogoMIPS:              4200.16
Virtualization:        AMD-V
L1d cache:             16K
L1i cache:             64K
L2 cache:              2048K
L3 cache:              6144K
NUMA node0 CPU(s):     0-15
dm@tod:~$ cat /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 1
model name	: AMD Opteron(TM) Processor 6272                 
stepping	: 2
microcode	: 0x600063d
cpu MHz		: 1400.000
cache size	: 2048 KB
physical id	: 0
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 32
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips	: 4199.82
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb
...

* about 128 GB of RAM:

jdm@tod:~$ free -g
             total       used       free     shared    buffers     cached
Mem:           125        123          2          0          0        121
-/+ buffers/cache:          2        123
Swap:          127          0        127
jdm@tod:~$