Diferencia entre revisiones de «Rsync»
(→Limiting tranfer rate) |
(→script (ssh)) |
||
(No se muestran 73 ediciones intermedias de 2 usuarios) | |||
Línea 1: | Línea 1: | ||
− | = | + | =short selection of rsync parameters= |
− | -a, --archive archive mode | + | ==-a, --archive== |
+ | -a, --archive archive mode | ||
+ | '''"-a" is the same as -rlptgoD (no -H)''' | ||
-r, --recursive recurse into directories | -r, --recursive recurse into directories | ||
-l, --links copy symlinks as symlinks | -l, --links copy symlinks as symlinks | ||
Línea 8: | Línea 10: | ||
-o, --owner preserve owner (super-user only) | -o, --owner preserve owner (super-user only) | ||
-D same as --devices --specials | -D same as --devices --specials | ||
− | + | ||
+ | ==commons== | ||
-v, --verbose increase verbosity | -v, --verbose increase verbosity | ||
-u, --update skip files that are newer on the receiver | -u, --update skip files that are newer on the receiver | ||
-z, --compress compress file data during the transfer | -z, --compress compress file data during the transfer | ||
-e, --rsh=COMMAND specify the remote shell to use | -e, --rsh=COMMAND specify the remote shell to use | ||
− | -h, --human-readable output numbers in a human-readable format | + | -h, --human-readable output numbers in a human-readable format '''Only for Etch version''' |
--progress show progress during transfer | --progress show progress during transfer | ||
+ | -f, --filter=RULE add a file-filtering RULE | ||
+ | |||
+ | ==for incrementals== | ||
+ | -b, --backup destination files are renamed | ||
+ | --backup-dir=DIR for incremental backups | ||
+ | |||
+ | ==limits== | ||
+ | --max-size=SIZE don't transfer any file larger than SIZE | ||
--bwlimit=KBPS limit I/O bandwidth; KBytes per second | --bwlimit=KBPS limit I/O bandwidth; KBytes per second | ||
− | |||
− | = | + | = Entire host excluding some directories (ssh without using root) RECOMENDED = |
− | ==gacq.com-get== | + | ==At host to backup (as root)== |
+ | mkdir /var/backups/.ssh | ||
+ | chmod 750 /var/backups/.ssh | ||
+ | chown backup:root /var/backups/.ssh | ||
+ | echo "backup ALL = NOPASSWD: /usr/bin/rsync" >> /etc/sudoers | ||
+ | |||
+ | ==At backup server (as root)== | ||
+ | ssh-keygen -f id_dsa | ||
+ | scp id_dsa.pub server_to_backup:/var/backups/.ssh/authorized_keys | ||
+ | |||
+ | ==script (ssh)== | ||
+ | <pre><nowiki> | ||
+ | HOST=debian | ||
+ | |||
+ | dir=full_backup | ||
+ | date=`date +%F_%T | tr \: _` | ||
+ | incremental=incremental/incremental_$date | ||
+ | |||
+ | mkdir -p $dir | ||
+ | mkdir -p $incremental | ||
+ | |||
+ | rsync -auzv \ | ||
+ | -e "ssh -l backup -i id_rsa" \ | ||
+ | --rsync-path="sudo /usr/bin/rsync" \ | ||
+ | --delete \ | ||
+ | --filter='- /sys/' \ | ||
+ | --filter='- /tmp/' \ | ||
+ | --filter='- /dev/' \ | ||
+ | --filter='- /mnt/' \ | ||
+ | --filter='- /media/' \ | ||
+ | --filter='- /proc/' \ | ||
+ | --filter='- /var/log/mysql/' \ | ||
+ | --filter='- /var/lib/mysql/' \ | ||
+ | --filter='- /var/cache/apt/archives/' \ | ||
+ | --filter='- /var/tmp/' \ | ||
+ | --filter='- /nobackup/' \ | ||
+ | --filter='- /opt/lampp/var/mysql/' \ | ||
+ | --backup --backup-dir=../$incremental \ | ||
+ | $HOST:/ $dir | ||
+ | </nowiki></pre> | ||
+ | |||
+ | = Backing up host configuration and selected files (ssh with using root user) = | ||
+ | ==Commands to run== | ||
+ | ===At host to backup (as root)=== | ||
+ | '''If root ssh access is enable''' (/etc/ssh/sshd_config with "PermitRootLogin yes") you can create the user with UID=0, and no problems with permisions: | ||
+ | useradd -g 0 -o --uid 0 rbackup | ||
+ | |||
+ | '''If not,''' for other servers with no ssh root access enabled, add user to system groups '''or use rsyncd''' | ||
+ | useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup | ||
+ | |||
+ | passwd rbackup | ||
+ | su - rbackup | ||
+ | mkdir -p /home/rbackup/.ssh | ||
+ | chmod 700 /home/rbackup/.ssh | ||
+ | chown -R rbackup.root /home/rbackup | ||
+ | |||
+ | ===At backup server (as root)=== | ||
+ | ssh-keygen -t dsa -f diskarray.gacq.com-id_dsa | ||
+ | scp diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com:.ssh/authorized_keys | ||
+ | |||
+ | '''Test the key:''' | ||
+ | ssh -i diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com | ||
+ | |||
+ | ==script (ssh)== | ||
+ | <pre><nowiki> | ||
+ | #!/bin/sh | ||
+ | # | ||
+ | # Configuration: | ||
+ | HOST=mynotebook.gacq.com | ||
+ | USER=gacq | ||
+ | MYSQLPASS=MYSQL_ROOT_PASS | ||
+ | |||
+ | # | ||
+ | dir=full_backup | ||
+ | date=`date +%F_%T | tr \: _` | ||
+ | incremental=incremental/incremental_$date | ||
+ | dumps=dumps/dump_$date | ||
+ | |||
+ | mkdir -p $dir | ||
+ | mkdir -p $incremental | ||
+ | mkdir -p $dumps | ||
+ | |||
+ | rsync -auzv \ | ||
+ | -e "ssh -l $USER -i $HOST-id_dsa" \ | ||
+ | --delete \ | ||
+ | --filter='+ /etc/' \ | ||
+ | --filter='+ /etc/**' \ | ||
+ | --filter='+ /root/' \ | ||
+ | --filter='+ /root/**' \ | ||
+ | --filter='+ /usr/' \ | ||
+ | --filter='+ /usr/local/' \ | ||
+ | --filter='+ /usr/local/scripts/' \ | ||
+ | --filter='+ /usr/local/scripts/**' \ | ||
+ | --filter='- /**' \ | ||
+ | --backup --backup-dir=../$incremental \ | ||
+ | $HOST:/ $dir | ||
+ | |||
+ | ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt | ||
+ | |||
+ | ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz | ||
+ | </nowiki></pre> | ||
+ | |||
+ | =Backup remote server (rsyncd - no root ssh)= | ||
+ | '''(Without root ssh access available - /etc/ssh/sshd_config with "PermitRootLogin no")''' | ||
+ | ==/etc/rsyncd.conf== | ||
<pre><nowiki> | <pre><nowiki> | ||
− | + | [all] | |
− | + | comment = from root | |
+ | path = / | ||
+ | read only = yes | ||
+ | list = yes | ||
+ | uid = root | ||
+ | gid = root | ||
+ | strict modes = yes | ||
+ | hosts allow = gacq.com | ||
+ | ignore errors = no | ||
+ | ignore nonreadable = yes | ||
+ | transfer logging = no | ||
+ | timeout = 600 | ||
+ | refuse options = checksum dry-run | ||
+ | dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz | ||
</nowiki></pre> | </nowiki></pre> | ||
− | == | + | ==script (rsyncd)== |
<pre><nowiki> | <pre><nowiki> | ||
− | + | #!/bin/sh | |
− | rsync - | + | # |
+ | # Configuration: | ||
+ | HOST=mycustomer.gacq.com | ||
+ | USER=gacq | ||
+ | MYSQLPASS=MYSQL_ROOT_PASS | ||
+ | |||
+ | # | ||
+ | dir=full_backup | ||
+ | date=`date +%F_%T | tr \: _` | ||
+ | incremental=incremental/incremental_$date | ||
+ | dumps=dumps/dump_$date | ||
+ | |||
+ | mkdir -p $dir | ||
+ | mkdir -p $incremental | ||
+ | mkdir -p $dumps | ||
+ | |||
+ | rsync -azv \ | ||
+ | --delete \ | ||
+ | --max-size=50M \ | ||
+ | --bwlimit=200 \ | ||
+ | --progress \ | ||
+ | --filter='+ /etc/' \ | ||
+ | --filter='+ /etc/**' \ | ||
+ | --filter='+ /root/' \ | ||
+ | --filter='+ /root/**' \ | ||
+ | --filter='+ /home/' \ | ||
+ | --filter='+ /home/**' \ | ||
+ | --filter='+ /usr/' \ | ||
+ | --filter='+ /usr/local/' \ | ||
+ | --filter='+ /usr/local/scripts/' \ | ||
+ | --filter='+ /usr/local/scripts/**' \ | ||
+ | --filter='- /**' \ | ||
+ | --backup --backup-dir=../$incremental \ | ||
+ | $HOST::all $dir | ||
+ | |||
+ | ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt | ||
+ | |||
+ | ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz | ||
</nowiki></pre> | </nowiki></pre> | ||
− | = | + | =NO root ssh access (and) NO rsyncd installed on remote host= |
+ | Use non root user and put it in the necesaries systems groups, for example daemon,dip,lpadmin,shadow,sasl | ||
+ | |||
+ | =Diferences between ryncd and ssh= | ||
+ | In the above scripts: | ||
+ | ==add/remove this== | ||
+ | -e "ssh -l $USER -i $HOST-id_dsa" | ||
+ | |||
+ | ==and change the source== | ||
+ | for rsyncd use: | ||
+ | $HOST::all $dir | ||
+ | for ssh use: | ||
+ | $HOST:/ $dir | ||
+ | |||
+ | =duplicating systems with sync= | ||
+ | '''Not tested''' | ||
<pre><nowiki> | <pre><nowiki> | ||
rsync -v -r -p -o -g -D -t -S -l -H \ | rsync -v -r -p -o -g -D -t -S -l -H \ | ||
Línea 39: | Línea 218: | ||
--exclude /home/ / /mnt/fireball/ | --exclude /home/ / /mnt/fireball/ | ||
</nowiki></pre> | </nowiki></pre> | ||
+ | |||
Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line: | Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line: | ||
+ | |||
<pre><nowiki> | <pre><nowiki> | ||
--exclude /etc/fstab \ | --exclude /etc/fstab \ | ||
Línea 46: | Línea 227: | ||
</nowiki></pre> | </nowiki></pre> | ||
− | = | + | =Working files syncronization / desktop <--> notebook= |
+ | *IMPORTANT: If target not exist will be deleted, use with caution. | ||
+ | *Remember to syncronize always after a change. | ||
+ | *Use these scripts from your home directory | ||
+ | |||
+ | ==gacq.com-get== | ||
+ | <pre><nowiki> | ||
+ | # Run this at the notebook | ||
+ | # Download new files from server, also remove files not present on server | ||
+ | # | ||
+ | HOST=gacq.com | ||
+ | USER=anysyncuser | ||
+ | SSH_PORT=22 | ||
− | ssh- | + | rsync -avuze 'ssh -p $SSH_PORT' --delete $USER@$HOST:/srv/gacq/rsync/ rsync |
+ | </nowiki></pre> | ||
+ | |||
+ | ==gacq.com-put== | ||
+ | <pre><nowiki> | ||
+ | # Run this at the notebook | ||
+ | # Upload new files to server, also remove files deleted here | ||
+ | # | ||
+ | HOST=gacq.com | ||
+ | USER=anysyncuser | ||
+ | SSH_PORT=22 | ||
+ | |||
+ | rsync -avuze 'ssh -p $SSH_PORT' --delete rsync $USER@$HOST:/srv/gacq/ | ||
+ | </nowiki></pre> |
Revisión actual del 17:47 22 jul 2010
Contenido
- 1 short selection of rsync parameters
- 2 Entire host excluding some directories (ssh without using root) RECOMENDED
- 3 Backing up host configuration and selected files (ssh with using root user)
- 4 Backup remote server (rsyncd - no root ssh)
- 5 NO root ssh access (and) NO rsyncd installed on remote host
- 6 Diferences between ryncd and ssh
- 7 duplicating systems with sync
- 8 Working files syncronization / desktop <--> notebook
short selection of rsync parameters
-a, --archive
-a, --archive archive mode
"-a" is the same as -rlptgoD (no -H)
-r, --recursive recurse into directories -l, --links copy symlinks as symlinks -p, --perms preserve permissions -t, --times preserve times -g, --group preserve group -o, --owner preserve owner (super-user only) -D same as --devices --specials
commons
-v, --verbose increase verbosity -u, --update skip files that are newer on the receiver -z, --compress compress file data during the transfer -e, --rsh=COMMAND specify the remote shell to use -h, --human-readable output numbers in a human-readable format Only for Etch version --progress show progress during transfer -f, --filter=RULE add a file-filtering RULE
for incrementals
-b, --backup destination files are renamed --backup-dir=DIR for incremental backups
limits
--max-size=SIZE don't transfer any file larger than SIZE --bwlimit=KBPS limit I/O bandwidth; KBytes per second
Entire host excluding some directories (ssh without using root) RECOMENDED
At host to backup (as root)
mkdir /var/backups/.ssh chmod 750 /var/backups/.ssh chown backup:root /var/backups/.ssh echo "backup ALL = NOPASSWD: /usr/bin/rsync" >> /etc/sudoers
At backup server (as root)
ssh-keygen -f id_dsa scp id_dsa.pub server_to_backup:/var/backups/.ssh/authorized_keys
script (ssh)
HOST=debian dir=full_backup date=`date +%F_%T | tr \: _` incremental=incremental/incremental_$date mkdir -p $dir mkdir -p $incremental rsync -auzv \ -e "ssh -l backup -i id_rsa" \ --rsync-path="sudo /usr/bin/rsync" \ --delete \ --filter='- /sys/' \ --filter='- /tmp/' \ --filter='- /dev/' \ --filter='- /mnt/' \ --filter='- /media/' \ --filter='- /proc/' \ --filter='- /var/log/mysql/' \ --filter='- /var/lib/mysql/' \ --filter='- /var/cache/apt/archives/' \ --filter='- /var/tmp/' \ --filter='- /nobackup/' \ --filter='- /opt/lampp/var/mysql/' \ --backup --backup-dir=../$incremental \ $HOST:/ $dir
Backing up host configuration and selected files (ssh with using root user)
Commands to run
At host to backup (as root)
If root ssh access is enable (/etc/ssh/sshd_config with "PermitRootLogin yes") you can create the user with UID=0, and no problems with permisions:
useradd -g 0 -o --uid 0 rbackup
If not, for other servers with no ssh root access enabled, add user to system groups or use rsyncd
useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup
passwd rbackup su - rbackup mkdir -p /home/rbackup/.ssh chmod 700 /home/rbackup/.ssh chown -R rbackup.root /home/rbackup
At backup server (as root)
ssh-keygen -t dsa -f diskarray.gacq.com-id_dsa scp diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com:.ssh/authorized_keys
Test the key:
ssh -i diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com
script (ssh)
#!/bin/sh # # Configuration: HOST=mynotebook.gacq.com USER=gacq MYSQLPASS=MYSQL_ROOT_PASS # dir=full_backup date=`date +%F_%T | tr \: _` incremental=incremental/incremental_$date dumps=dumps/dump_$date mkdir -p $dir mkdir -p $incremental mkdir -p $dumps rsync -auzv \ -e "ssh -l $USER -i $HOST-id_dsa" \ --delete \ --filter='+ /etc/' \ --filter='+ /etc/**' \ --filter='+ /root/' \ --filter='+ /root/**' \ --filter='+ /usr/' \ --filter='+ /usr/local/' \ --filter='+ /usr/local/scripts/' \ --filter='+ /usr/local/scripts/**' \ --filter='- /**' \ --backup --backup-dir=../$incremental \ $HOST:/ $dir ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz
Backup remote server (rsyncd - no root ssh)
(Without root ssh access available - /etc/ssh/sshd_config with "PermitRootLogin no")
/etc/rsyncd.conf
[all] comment = from root path = / read only = yes list = yes uid = root gid = root strict modes = yes hosts allow = gacq.com ignore errors = no ignore nonreadable = yes transfer logging = no timeout = 600 refuse options = checksum dry-run dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz
script (rsyncd)
#!/bin/sh # # Configuration: HOST=mycustomer.gacq.com USER=gacq MYSQLPASS=MYSQL_ROOT_PASS # dir=full_backup date=`date +%F_%T | tr \: _` incremental=incremental/incremental_$date dumps=dumps/dump_$date mkdir -p $dir mkdir -p $incremental mkdir -p $dumps rsync -azv \ --delete \ --max-size=50M \ --bwlimit=200 \ --progress \ --filter='+ /etc/' \ --filter='+ /etc/**' \ --filter='+ /root/' \ --filter='+ /root/**' \ --filter='+ /home/' \ --filter='+ /home/**' \ --filter='+ /usr/' \ --filter='+ /usr/local/' \ --filter='+ /usr/local/scripts/' \ --filter='+ /usr/local/scripts/**' \ --filter='- /**' \ --backup --backup-dir=../$incremental \ $HOST::all $dir ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz
NO root ssh access (and) NO rsyncd installed on remote host
Use non root user and put it in the necesaries systems groups, for example daemon,dip,lpadmin,shadow,sasl
Diferences between ryncd and ssh
In the above scripts:
add/remove this
-e "ssh -l $USER -i $HOST-id_dsa"
and change the source
for rsyncd use:
$HOST::all $dir
for ssh use:
$HOST:/ $dir
duplicating systems with sync
Not tested
rsync -v -r -p -o -g -D -t -S -l -H \ --exclude /mnt/ \ --exclude /proc/ \ --exclude /tmp/ \ --exclude /home/ / /mnt/fireball/
Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line:
--exclude /etc/fstab \ --exclude /etc/lilo.conf \ --delete
Working files syncronization / desktop <--> notebook
- IMPORTANT: If target not exist will be deleted, use with caution.
- Remember to syncronize always after a change.
- Use these scripts from your home directory
gacq.com-get
# Run this at the notebook # Download new files from server, also remove files not present on server # HOST=gacq.com USER=anysyncuser SSH_PORT=22 rsync -avuze 'ssh -p $SSH_PORT' --delete $USER@$HOST:/srv/gacq/rsync/ rsync
gacq.com-put
# Run this at the notebook # Upload new files to server, also remove files deleted here # HOST=gacq.com USER=anysyncuser SSH_PORT=22 rsync -avuze 'ssh -p $SSH_PORT' --delete rsync $USER@$HOST:/srv/gacq/