Diferencia entre revisiones de «Rsync»

De gacq wiki
Saltar a: navegación, buscar
(Limiting tranfer rate)
(script (ssh))
 
(No se muestran 73 ediciones intermedias de 2 usuarios)
Línea 1: Línea 1:
=Most used options=
+
=short selection of rsync parameters=
  -a, --archive              archive mode; same as -rlptgoD (no -H)
+
==-a, --archive==
 +
  -a, --archive              archive mode
 +
'''"-a" is the same as -rlptgoD (no -H)'''
 
   -r, --recursive            recurse into directories
 
   -r, --recursive            recurse into directories
 
   -l, --links                copy symlinks as symlinks
 
   -l, --links                copy symlinks as symlinks
Línea 8: Línea 10:
 
   -o, --owner                preserve owner (super-user only)
 
   -o, --owner                preserve owner (super-user only)
 
   -D                          same as --devices --specials
 
   -D                          same as --devices --specials
+
 
 +
==commons==
 
  -v, --verbose              increase verbosity
 
  -v, --verbose              increase verbosity
 
  -u, --update                skip files that are newer on the receiver
 
  -u, --update                skip files that are newer on the receiver
 
  -z, --compress              compress file data during the transfer
 
  -z, --compress              compress file data during the transfer
 
  -e, --rsh=COMMAND          specify the remote shell to use
 
  -e, --rsh=COMMAND          specify the remote shell to use
  -h, --human-readable        output numbers in a human-readable format
+
  -h, --human-readable        output numbers in a human-readable format '''Only for Etch version'''
 
     --progress              show progress during transfer
 
     --progress              show progress during transfer
 +
-f, --filter=RULE          add a file-filtering RULE
 +
 +
==for incrementals==
 +
-b, --backup                destination files are renamed
 +
  --backup-dir=DIR          for incremental backups
 +
 +
==limits==
 +
--max-size=SIZE        don't transfer any file larger than SIZE
 
  --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
 
  --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
--exclude-from=FILE    read exclude patterns from FILE
 
  
=My notebook syncronization=
+
= Entire host excluding some directories (ssh without using root) RECOMENDED =
==gacq.com-get==
+
==At host to backup (as root)==
 +
mkdir /var/backups/.ssh
 +
chmod 750 /var/backups/.ssh
 +
chown backup:root /var/backups/.ssh
 +
echo "backup ALL = NOPASSWD: /usr/bin/rsync" >> /etc/sudoers
 +
 
 +
==At backup server (as root)==
 +
ssh-keygen -f id_dsa
 +
scp id_dsa.pub server_to_backup:/var/backups/.ssh/authorized_keys
 +
 
 +
==script (ssh)==
 +
<pre><nowiki>
 +
HOST=debian
 +
 
 +
dir=full_backup
 +
date=`date +%F_%T | tr \: _`
 +
incremental=incremental/incremental_$date
 +
 
 +
mkdir -p $dir
 +
mkdir -p $incremental
 +
 
 +
rsync -auzv \
 +
  -e "ssh -l backup -i id_rsa" \
 +
  --rsync-path="sudo /usr/bin/rsync"  \
 +
  --delete \
 +
  --filter='- /sys/' \
 +
  --filter='- /tmp/' \
 +
  --filter='- /dev/' \
 +
  --filter='- /mnt/' \
 +
  --filter='- /media/' \
 +
  --filter='- /proc/' \
 +
  --filter='- /var/log/mysql/' \
 +
  --filter='- /var/lib/mysql/' \
 +
  --filter='- /var/cache/apt/archives/' \
 +
  --filter='- /var/tmp/' \
 +
  --filter='- /nobackup/' \
 +
  --filter='- /opt/lampp/var/mysql/' \
 +
  --backup --backup-dir=../$incremental \
 +
  $HOST:/ $dir
 +
</nowiki></pre>
 +
 
 +
= Backing up host configuration and selected files (ssh with using root user) =
 +
==Commands to run==
 +
===At host to backup (as root)===
 +
'''If root ssh access is enable''' (/etc/ssh/sshd_config with "PermitRootLogin yes") you can create the user with UID=0, and no problems with permisions:
 +
useradd -g 0 -o --uid 0 rbackup
 +
 
 +
'''If not,''' for other servers with no ssh root access enabled, add user to system groups '''or use rsyncd'''
 +
useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup
 +
 
 +
passwd rbackup
 +
su - rbackup
 +
mkdir -p /home/rbackup/.ssh
 +
chmod 700 /home/rbackup/.ssh
 +
chown -R rbackup.root /home/rbackup
 +
 
 +
===At backup server (as root)===
 +
ssh-keygen -t dsa -f diskarray.gacq.com-id_dsa
 +
scp diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com:.ssh/authorized_keys
 +
 
 +
'''Test the key:'''
 +
ssh -i diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com
 +
 
 +
==script (ssh)==
 +
<pre><nowiki>
 +
#!/bin/sh
 +
#
 +
# Configuration:
 +
HOST=mynotebook.gacq.com
 +
USER=gacq
 +
MYSQLPASS=MYSQL_ROOT_PASS
 +
 
 +
#
 +
dir=full_backup
 +
date=`date +%F_%T | tr \: _`
 +
incremental=incremental/incremental_$date
 +
dumps=dumps/dump_$date
 +
 
 +
mkdir -p $dir
 +
mkdir -p $incremental
 +
mkdir -p $dumps
 +
 
 +
rsync -auzv \
 +
  -e "ssh -l $USER -i $HOST-id_dsa" \
 +
  --delete \
 +
  --filter='+ /etc/' \
 +
  --filter='+ /etc/**' \
 +
  --filter='+ /root/' \
 +
  --filter='+ /root/**' \
 +
  --filter='+ /usr/' \
 +
  --filter='+ /usr/local/' \
 +
  --filter='+ /usr/local/scripts/' \
 +
  --filter='+ /usr/local/scripts/**' \
 +
  --filter='- /**' \
 +
  --backup --backup-dir=../$incremental \
 +
  $HOST:/ $dir
 +
 
 +
ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt
 +
 
 +
ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz
 +
</nowiki></pre>
 +
 
 +
=Backup remote server (rsyncd - no root ssh)=
 +
'''(Without root ssh access available - /etc/ssh/sshd_config with "PermitRootLogin no")'''
 +
==/etc/rsyncd.conf==
 
<pre><nowiki>
 
<pre><nowiki>
echo Receiving.....
+
[all]
rsync -avuze 'ssh -p 22' --delete anysyncuser@myserver:/srv/gacq/rsync/ rsync
+
        comment = from root
 +
        path = /
 +
        read only = yes
 +
        list = yes
 +
        uid = root
 +
        gid = root
 +
        strict modes = yes
 +
        hosts allow = gacq.com
 +
        ignore errors = no
 +
        ignore nonreadable = yes
 +
        transfer logging = no
 +
        timeout = 600
 +
        refuse options = checksum dry-run
 +
        dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz
 
</nowiki></pre>
 
</nowiki></pre>
  
==gacq.com-put==
+
==script (rsyncd)==
 
<pre><nowiki>
 
<pre><nowiki>
echo Sending.....
+
#!/bin/sh
rsync -avuze 'ssh -p 22' --delete rsync anysyncuser@myserver:/srv/gacq/
+
#
 +
# Configuration:
 +
HOST=mycustomer.gacq.com
 +
USER=gacq
 +
MYSQLPASS=MYSQL_ROOT_PASS
 +
 
 +
#
 +
dir=full_backup
 +
date=`date +%F_%T | tr \: _`
 +
incremental=incremental/incremental_$date
 +
dumps=dumps/dump_$date
 +
 
 +
mkdir -p $dir
 +
mkdir -p $incremental
 +
mkdir -p $dumps
 +
 
 +
rsync -azv \
 +
  --delete \
 +
  --max-size=50M \
 +
  --bwlimit=200 \
 +
  --progress \
 +
  --filter='+ /etc/' \
 +
  --filter='+ /etc/**' \
 +
  --filter='+ /root/' \
 +
  --filter='+ /root/**' \
 +
  --filter='+ /home/' \
 +
  --filter='+ /home/**' \
 +
  --filter='+ /usr/' \
 +
  --filter='+ /usr/local/' \
 +
  --filter='+ /usr/local/scripts/' \
 +
  --filter='+ /usr/local/scripts/**' \
 +
  --filter='- /**' \
 +
  --backup --backup-dir=../$incremental \
 +
  $HOST::all $dir
 +
 
 +
ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt
 +
 
 +
ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz
 
</nowiki></pre>
 
</nowiki></pre>
  
= Duplicando una maquina con rsync =
+
=NO root ssh access (and) NO rsyncd installed on remote host=
 +
Use non root user and put it in the necesaries systems groups, for example daemon,dip,lpadmin,shadow,sasl
 +
 
 +
=Diferences between ryncd and ssh=
 +
In the above scripts:
 +
==add/remove this==
 +
-e "ssh -l $USER -i $HOST-id_dsa"
 +
 
 +
==and change the source==
 +
for rsyncd use:
 +
$HOST::all $dir
 +
for ssh use:
 +
$HOST:/ $dir
 +
 
 +
=duplicating systems with sync=
 +
'''Not tested'''
 
<pre><nowiki>
 
<pre><nowiki>
 
rsync -v -r -p -o -g -D -t -S -l -H \
 
rsync -v -r -p -o -g -D -t -S -l -H \
Línea 39: Línea 218:
 
--exclude /home/ / /mnt/fireball/
 
--exclude /home/ / /mnt/fireball/
 
</nowiki></pre>
 
</nowiki></pre>
 +
 
Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line:
 
Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line:
 +
 
<pre><nowiki>
 
<pre><nowiki>
 
--exclude /etc/fstab \
 
--exclude /etc/fstab \
Línea 46: Línea 227:
 
</nowiki></pre>
 
</nowiki></pre>
  
=Backup using rsync and ssh=
+
=Working files syncronization / desktop <--> notebook=
 +
*IMPORTANT: If target not exist will be deleted, use with caution.
 +
*Remember to syncronize always after a change.
 +
*Use these scripts from your home directory
 +
 
 +
==gacq.com-get==
 +
<pre><nowiki>
 +
# Run this at the notebook
 +
# Download new files from server, also remove files not present on server
 +
#
 +
HOST=gacq.com
 +
USER=anysyncuser
 +
SSH_PORT=22
  
ssh-keygen -t dsa -f fabre.homelinux.net-id_dsa
+
rsync -avuze 'ssh -p $SSH_PORT' --delete $USER@$HOST:/srv/gacq/rsync/ rsync
 +
</nowiki></pre>
 +
 
 +
==gacq.com-put==
 +
<pre><nowiki>
 +
# Run this at the notebook
 +
# Upload new files to server, also remove files deleted here
 +
#
 +
HOST=gacq.com
 +
USER=anysyncuser
 +
SSH_PORT=22
 +
 
 +
rsync -avuze 'ssh -p $SSH_PORT' --delete rsync $USER@$HOST:/srv/gacq/
 +
</nowiki></pre>

Revisión actual del 17:47 22 jul 2010

short selection of rsync parameters

-a, --archive

-a, --archive               archive mode

"-a" is the same as -rlptgoD (no -H)

 -r, --recursive             recurse into directories
 -l, --links                 copy symlinks as symlinks
 -p, --perms                 preserve permissions
 -t, --times                 preserve times
 -g, --group                 preserve group
 -o, --owner                 preserve owner (super-user only)
 -D                          same as --devices --specials

commons

-v, --verbose               increase verbosity
-u, --update                skip files that are newer on the receiver
-z, --compress              compress file data during the transfer
-e, --rsh=COMMAND           specify the remote shell to use
-h, --human-readable        output numbers in a human-readable format Only for Etch version
    --progress              show progress during transfer
-f, --filter=RULE           add a file-filtering RULE

for incrementals

-b, --backup                destination files are renamed
  --backup-dir=DIR          for incremental backups

limits

--max-size=SIZE         don't transfer any file larger than SIZE
--bwlimit=KBPS          limit I/O bandwidth; KBytes per second

Entire host excluding some directories (ssh without using root) RECOMENDED

At host to backup (as root)

mkdir /var/backups/.ssh
chmod 750 /var/backups/.ssh
chown backup:root /var/backups/.ssh
echo "backup ALL = NOPASSWD: /usr/bin/rsync" >> /etc/sudoers

At backup server (as root)

ssh-keygen -f id_dsa
scp id_dsa.pub server_to_backup:/var/backups/.ssh/authorized_keys

script (ssh)

HOST=debian

dir=full_backup
date=`date +%F_%T | tr \: _`
incremental=incremental/incremental_$date

mkdir -p $dir
mkdir -p $incremental

rsync -auzv \
  -e "ssh -l backup -i id_rsa" \
  --rsync-path="sudo /usr/bin/rsync"  \
  --delete \
  --filter='- /sys/' \
  --filter='- /tmp/' \
  --filter='- /dev/' \
  --filter='- /mnt/' \
  --filter='- /media/' \
  --filter='- /proc/' \
  --filter='- /var/log/mysql/' \
  --filter='- /var/lib/mysql/' \
  --filter='- /var/cache/apt/archives/' \
  --filter='- /var/tmp/' \
  --filter='- /nobackup/' \
  --filter='- /opt/lampp/var/mysql/' \
  --backup --backup-dir=../$incremental \
  $HOST:/ $dir

Backing up host configuration and selected files (ssh with using root user)

Commands to run

At host to backup (as root)

If root ssh access is enable (/etc/ssh/sshd_config with "PermitRootLogin yes") you can create the user with UID=0, and no problems with permisions:

useradd -g 0 -o --uid 0 rbackup

If not, for other servers with no ssh root access enabled, add user to system groups or use rsyncd

useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup
passwd rbackup
su - rbackup
mkdir -p /home/rbackup/.ssh
chmod 700 /home/rbackup/.ssh
chown -R rbackup.root /home/rbackup

At backup server (as root)

ssh-keygen -t dsa -f diskarray.gacq.com-id_dsa
scp diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com:.ssh/authorized_keys

Test the key:

ssh -i diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com

script (ssh)

#!/bin/sh
#
# Configuration:
HOST=mynotebook.gacq.com
USER=gacq
MYSQLPASS=MYSQL_ROOT_PASS

#
dir=full_backup
date=`date +%F_%T | tr \: _`
incremental=incremental/incremental_$date
dumps=dumps/dump_$date

mkdir -p $dir
mkdir -p $incremental
mkdir -p $dumps

rsync -auzv \
  -e "ssh -l $USER -i $HOST-id_dsa" \
  --delete \
  --filter='+ /etc/' \
  --filter='+ /etc/**' \
  --filter='+ /root/' \
  --filter='+ /root/**' \
  --filter='+ /usr/' \
  --filter='+ /usr/local/' \
  --filter='+ /usr/local/scripts/' \
  --filter='+ /usr/local/scripts/**' \
  --filter='- /**' \
  --backup --backup-dir=../$incremental \
  $HOST:/ $dir

ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt

ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz

Backup remote server (rsyncd - no root ssh)

(Without root ssh access available - /etc/ssh/sshd_config with "PermitRootLogin no")

/etc/rsyncd.conf

[all]
        comment = from root
        path = /
        read only = yes
        list = yes
        uid = root
        gid = root
        strict modes = yes
        hosts allow = gacq.com
        ignore errors = no
        ignore nonreadable = yes
        transfer logging = no
        timeout = 600
        refuse options = checksum dry-run
        dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

script (rsyncd)

#!/bin/sh
#
# Configuration:
HOST=mycustomer.gacq.com
USER=gacq
MYSQLPASS=MYSQL_ROOT_PASS

#
dir=full_backup
date=`date +%F_%T | tr \: _`
incremental=incremental/incremental_$date
dumps=dumps/dump_$date

mkdir -p $dir
mkdir -p $incremental
mkdir -p $dumps

rsync -azv \
  --delete \
  --max-size=50M \
  --bwlimit=200 \
  --progress \
  --filter='+ /etc/' \
  --filter='+ /etc/**' \
  --filter='+ /root/' \
  --filter='+ /root/**' \
  --filter='+ /home/' \
  --filter='+ /home/**' \
  --filter='+ /usr/' \
  --filter='+ /usr/local/' \
  --filter='+ /usr/local/scripts/' \
  --filter='+ /usr/local/scripts/**' \
  --filter='- /**' \
  --backup --backup-dir=../$incremental \
  $HOST::all $dir

ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt

ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz

NO root ssh access (and) NO rsyncd installed on remote host

Use non root user and put it in the necesaries systems groups, for example daemon,dip,lpadmin,shadow,sasl

Diferences between ryncd and ssh

In the above scripts:

add/remove this

-e "ssh -l $USER -i $HOST-id_dsa"

and change the source

for rsyncd use:

$HOST::all $dir

for ssh use:

$HOST:/ $dir

duplicating systems with sync

Not tested

rsync -v -r -p -o -g -D -t -S -l -H \
--exclude /mnt/   \
--exclude /proc/ \
--exclude /tmp/ \
--exclude /home/ / /mnt/fireball/

Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line:

--exclude /etc/fstab \
--exclude /etc/lilo.conf \
--delete 

Working files syncronization / desktop <--> notebook

  • IMPORTANT: If target not exist will be deleted, use with caution.
  • Remember to syncronize always after a change.
  • Use these scripts from your home directory

gacq.com-get

# Run this at the notebook
# Download new files from server, also remove files not present on server
#
HOST=gacq.com
USER=anysyncuser
SSH_PORT=22

rsync -avuze 'ssh -p $SSH_PORT' --delete $USER@$HOST:/srv/gacq/rsync/ rsync

gacq.com-put

# Run this at the notebook
# Upload new files to server, also remove files deleted here
#
HOST=gacq.com
USER=anysyncuser
SSH_PORT=22

rsync -avuze 'ssh -p $SSH_PORT' --delete rsync $USER@$HOST:/srv/gacq/