Diferencia entre revisiones de «Rsync»

De gacq wiki
Saltar a: navegación, buscar
(for incremental)
(script (ssh))
 
(No se muestran 36 ediciones intermedias de 2 usuarios)
Línea 1: Línea 1:
=rsync has a lot of parameters, here you have a selection=
+
=short selection of rsync parameters=
 
==-a, --archive==
 
==-a, --archive==
 
  -a, --archive              archive mode
 
  -a, --archive              archive mode
 
+
'''"-a" is the same as -rlptgoD (no -H)'''
'''same as -rlptgoD (no -H)'''
 
 
   -r, --recursive            recurse into directories
 
   -r, --recursive            recurse into directories
 
   -l, --links                copy symlinks as symlinks
 
   -l, --links                copy symlinks as symlinks
Línea 17: Línea 16:
 
  -z, --compress              compress file data during the transfer
 
  -z, --compress              compress file data during the transfer
 
  -e, --rsh=COMMAND          specify the remote shell to use
 
  -e, --rsh=COMMAND          specify the remote shell to use
  -h, --human-readable        output numbers in a human-readable format
+
  -h, --human-readable        output numbers in a human-readable format '''Only for Etch version'''
 
     --progress              show progress during transfer
 
     --progress              show progress during transfer
 
  -f, --filter=RULE          add a file-filtering RULE
 
  -f, --filter=RULE          add a file-filtering RULE
Línea 26: Línea 25:
  
 
==limits==
 
==limits==
  --max-size=SIZE
+
  --max-size=SIZE        don't transfer any file larger than SIZE
 
  --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
 
  --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
  
=My notebook syncronization=
+
= Entire host excluding some directories (ssh without using root) RECOMENDED =
==gacq.com-get==
+
==At host to backup (as root)==
 +
mkdir /var/backups/.ssh
 +
chmod 750 /var/backups/.ssh
 +
chown backup:root /var/backups/.ssh
 +
echo "backup ALL = NOPASSWD: /usr/bin/rsync" >> /etc/sudoers
 +
 
 +
==At backup server (as root)==
 +
ssh-keygen -f id_dsa
 +
scp id_dsa.pub server_to_backup:/var/backups/.ssh/authorized_keys
 +
 
 +
==script (ssh)==
 
<pre><nowiki>
 
<pre><nowiki>
echo Receiving.....
+
HOST=debian
rsync -avuze 'ssh -p 22' --delete anysyncuser@gacq.com:/srv/gacq/rsync/ rsync
 
</nowiki></pre>
 
  
==gacq.com-put==
+
dir=full_backup
<pre><nowiki>
+
date=`date +%F_%T | tr \: _`
echo Sending.....
+
incremental=incremental/incremental_$date
rsync -avuze 'ssh -p 22' --delete rsync anysyncuser@gacq.com:/srv/gacq/
 
</nowiki></pre>
 
  
 +
mkdir -p $dir
 +
mkdir -p $incremental
  
 +
rsync -auzv \
 +
  -e "ssh -l backup -i id_rsa" \
 +
  --rsync-path="sudo /usr/bin/rsync"  \
 +
  --delete \
 +
  --filter='- /sys/' \
 +
  --filter='- /tmp/' \
 +
  --filter='- /dev/' \
 +
  --filter='- /mnt/' \
 +
  --filter='- /media/' \
 +
  --filter='- /proc/' \
 +
  --filter='- /var/log/mysql/' \
 +
  --filter='- /var/lib/mysql/' \
 +
  --filter='- /var/cache/apt/archives/' \
 +
  --filter='- /var/tmp/' \
 +
  --filter='- /nobackup/' \
 +
  --filter='- /opt/lampp/var/mysql/' \
 +
  --backup --backup-dir=../$incremental \
 +
  $HOST:/ $dir
 +
</nowiki></pre>
  
=Backup using ssh=
+
= Backing up host configuration and selected files (ssh with using root user) =
==Commands to execute==
+
==Commands to run==
 
===At host to backup (as root)===
 
===At host to backup (as root)===
 +
'''If root ssh access is enable''' (/etc/ssh/sshd_config with "PermitRootLogin yes") you can create the user with UID=0, and no problems with permisions:
 +
useradd -g 0 -o --uid 0 rbackup
 +
 +
'''If not,''' for other servers with no ssh root access enabled, add user to system groups '''or use rsyncd'''
 
  useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup
 
  useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup
 +
 
  passwd rbackup
 
  passwd rbackup
 
  su - rbackup
 
  su - rbackup
Línea 80: Línea 111:
 
mkdir -p $dumps
 
mkdir -p $dumps
  
rsync -auzvh \
+
rsync -auzv \
 
   -e "ssh -l $USER -i $HOST-id_dsa" \
 
   -e "ssh -l $USER -i $HOST-id_dsa" \
 
   --delete \
 
   --delete \
Línea 100: Línea 131:
 
</nowiki></pre>
 
</nowiki></pre>
  
=Backup using rsyncd=
+
=Backup remote server (rsyncd - no root ssh)=
 +
'''(Without root ssh access available - /etc/ssh/sshd_config with "PermitRootLogin no")'''
 
==/etc/rsyncd.conf==
 
==/etc/rsyncd.conf==
 
<pre><nowiki>
 
<pre><nowiki>
Línea 139: Línea 171:
 
mkdir -p $dumps
 
mkdir -p $dumps
  
rsync -azvh \
+
rsync -azv \
 
   --delete \
 
   --delete \
 
   --max-size=50M \
 
   --max-size=50M \
Línea 163: Línea 195:
 
</nowiki></pre>
 
</nowiki></pre>
  
=Diferences between ryncd and ssh =
+
=NO root ssh access (and) NO rsyncd installed on remote host=
 +
Use non root user and put it in the necesaries systems groups, for example daemon,dip,lpadmin,shadow,sasl
 +
 
 +
=Diferences between ryncd and ssh=
 +
In the above scripts:
 
==add/remove this==
 
==add/remove this==
 
  -e "ssh -l $USER -i $HOST-id_dsa"
 
  -e "ssh -l $USER -i $HOST-id_dsa"
Línea 174: Línea 210:
  
 
=duplicating systems with sync=
 
=duplicating systems with sync=
 +
'''Not tested'''
 
<pre><nowiki>
 
<pre><nowiki>
 
rsync -v -r -p -o -g -D -t -S -l -H \
 
rsync -v -r -p -o -g -D -t -S -l -H \
Línea 188: Línea 225:
 
--exclude /etc/lilo.conf \
 
--exclude /etc/lilo.conf \
 
--delete  
 
--delete  
 +
</nowiki></pre>
 +
 +
=Working files syncronization / desktop <--> notebook=
 +
*IMPORTANT: If target not exist will be deleted, use with caution.
 +
*Remember to syncronize always after a change.
 +
*Use these scripts from your home directory
 +
 +
==gacq.com-get==
 +
<pre><nowiki>
 +
# Run this at the notebook
 +
# Download new files from server, also remove files not present on server
 +
#
 +
HOST=gacq.com
 +
USER=anysyncuser
 +
SSH_PORT=22
 +
 +
rsync -avuze 'ssh -p $SSH_PORT' --delete $USER@$HOST:/srv/gacq/rsync/ rsync
 +
</nowiki></pre>
 +
 +
==gacq.com-put==
 +
<pre><nowiki>
 +
# Run this at the notebook
 +
# Upload new files to server, also remove files deleted here
 +
#
 +
HOST=gacq.com
 +
USER=anysyncuser
 +
SSH_PORT=22
 +
 +
rsync -avuze 'ssh -p $SSH_PORT' --delete rsync $USER@$HOST:/srv/gacq/
 
</nowiki></pre>
 
</nowiki></pre>

Revisión actual del 17:47 22 jul 2010

short selection of rsync parameters

-a, --archive

-a, --archive               archive mode

"-a" is the same as -rlptgoD (no -H)

 -r, --recursive             recurse into directories
 -l, --links                 copy symlinks as symlinks
 -p, --perms                 preserve permissions
 -t, --times                 preserve times
 -g, --group                 preserve group
 -o, --owner                 preserve owner (super-user only)
 -D                          same as --devices --specials

commons

-v, --verbose               increase verbosity
-u, --update                skip files that are newer on the receiver
-z, --compress              compress file data during the transfer
-e, --rsh=COMMAND           specify the remote shell to use
-h, --human-readable        output numbers in a human-readable format Only for Etch version
    --progress              show progress during transfer
-f, --filter=RULE           add a file-filtering RULE

for incrementals

-b, --backup                destination files are renamed
  --backup-dir=DIR          for incremental backups

limits

--max-size=SIZE         don't transfer any file larger than SIZE
--bwlimit=KBPS          limit I/O bandwidth; KBytes per second

Entire host excluding some directories (ssh without using root) RECOMENDED

At host to backup (as root)

mkdir /var/backups/.ssh
chmod 750 /var/backups/.ssh
chown backup:root /var/backups/.ssh
echo "backup ALL = NOPASSWD: /usr/bin/rsync" >> /etc/sudoers

At backup server (as root)

ssh-keygen -f id_dsa
scp id_dsa.pub server_to_backup:/var/backups/.ssh/authorized_keys

script (ssh)

HOST=debian

dir=full_backup
date=`date +%F_%T | tr \: _`
incremental=incremental/incremental_$date

mkdir -p $dir
mkdir -p $incremental

rsync -auzv \
  -e "ssh -l backup -i id_rsa" \
  --rsync-path="sudo /usr/bin/rsync"  \
  --delete \
  --filter='- /sys/' \
  --filter='- /tmp/' \
  --filter='- /dev/' \
  --filter='- /mnt/' \
  --filter='- /media/' \
  --filter='- /proc/' \
  --filter='- /var/log/mysql/' \
  --filter='- /var/lib/mysql/' \
  --filter='- /var/cache/apt/archives/' \
  --filter='- /var/tmp/' \
  --filter='- /nobackup/' \
  --filter='- /opt/lampp/var/mysql/' \
  --backup --backup-dir=../$incremental \
  $HOST:/ $dir

Backing up host configuration and selected files (ssh with using root user)

Commands to run

At host to backup (as root)

If root ssh access is enable (/etc/ssh/sshd_config with "PermitRootLogin yes") you can create the user with UID=0, and no problems with permisions:

useradd -g 0 -o --uid 0 rbackup

If not, for other servers with no ssh root access enabled, add user to system groups or use rsyncd

useradd -g 0 -G daemon,dip,lpadmin,shadow,sasl rbackup
passwd rbackup
su - rbackup
mkdir -p /home/rbackup/.ssh
chmod 700 /home/rbackup/.ssh
chown -R rbackup.root /home/rbackup

At backup server (as root)

ssh-keygen -t dsa -f diskarray.gacq.com-id_dsa
scp diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com:.ssh/authorized_keys

Test the key:

ssh -i diskarray.gacq.com-id_dsa.pub rbackup@diskarray.gacq.com

script (ssh)

#!/bin/sh
#
# Configuration:
HOST=mynotebook.gacq.com
USER=gacq
MYSQLPASS=MYSQL_ROOT_PASS

#
dir=full_backup
date=`date +%F_%T | tr \: _`
incremental=incremental/incremental_$date
dumps=dumps/dump_$date

mkdir -p $dir
mkdir -p $incremental
mkdir -p $dumps

rsync -auzv \
  -e "ssh -l $USER -i $HOST-id_dsa" \
  --delete \
  --filter='+ /etc/' \
  --filter='+ /etc/**' \
  --filter='+ /root/' \
  --filter='+ /root/**' \
  --filter='+ /usr/' \
  --filter='+ /usr/local/' \
  --filter='+ /usr/local/scripts/' \
  --filter='+ /usr/local/scripts/**' \
  --filter='- /**' \
  --backup --backup-dir=../$incremental \
  $HOST:/ $dir

ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt

ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz

Backup remote server (rsyncd - no root ssh)

(Without root ssh access available - /etc/ssh/sshd_config with "PermitRootLogin no")

/etc/rsyncd.conf

[all]
        comment = from root
        path = /
        read only = yes
        list = yes
        uid = root
        gid = root
        strict modes = yes
        hosts allow = gacq.com
        ignore errors = no
        ignore nonreadable = yes
        transfer logging = no
        timeout = 600
        refuse options = checksum dry-run
        dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

script (rsyncd)

#!/bin/sh
#
# Configuration:
HOST=mycustomer.gacq.com
USER=gacq
MYSQLPASS=MYSQL_ROOT_PASS

#
dir=full_backup
date=`date +%F_%T | tr \: _`
incremental=incremental/incremental_$date
dumps=dumps/dump_$date

mkdir -p $dir
mkdir -p $incremental
mkdir -p $dumps

rsync -azv \
  --delete \
  --max-size=50M \
  --bwlimit=200 \
  --progress \
  --filter='+ /etc/' \
  --filter='+ /etc/**' \
  --filter='+ /root/' \
  --filter='+ /root/**' \
  --filter='+ /home/' \
  --filter='+ /home/**' \
  --filter='+ /usr/' \
  --filter='+ /usr/local/' \
  --filter='+ /usr/local/scripts/' \
  --filter='+ /usr/local/scripts/**' \
  --filter='- /**' \
  --backup --backup-dir=../$incremental \
  $HOST::all $dir

ssh -l $USER -i $HOST-id_dsa $HOST "/usr/bin/dpkg --get-selections" > $dumps/dpkg--get-selections.txt

ssh -l $USER -i $HOST-id_dsa $HOST "mysqldump --all-databases --password=$MYSQLPASS | gzip" > $dumps/all-databases.sql.gz

NO root ssh access (and) NO rsyncd installed on remote host

Use non root user and put it in the necesaries systems groups, for example daemon,dip,lpadmin,shadow,sasl

Diferences between ryncd and ssh

In the above scripts:

add/remove this

-e "ssh -l $USER -i $HOST-id_dsa"

and change the source

for rsyncd use:

$HOST::all $dir

for ssh use:

$HOST:/ $dir

duplicating systems with sync

Not tested

rsync -v -r -p -o -g -D -t -S -l -H \
--exclude /mnt/   \
--exclude /proc/ \
--exclude /tmp/ \
--exclude /home/ / /mnt/fireball/

Boot from a CD-ROM and mount the target-disk. Fix fstab and lilo.conf. Create /mnt and /proc and set the right permissions (for this directories). Then run lilo and reboot. Next time you run rsync, add these option to the command-line:

--exclude /etc/fstab \
--exclude /etc/lilo.conf \
--delete 

Working files syncronization / desktop <--> notebook

  • IMPORTANT: If target not exist will be deleted, use with caution.
  • Remember to syncronize always after a change.
  • Use these scripts from your home directory

gacq.com-get

# Run this at the notebook
# Download new files from server, also remove files not present on server
#
HOST=gacq.com
USER=anysyncuser
SSH_PORT=22

rsync -avuze 'ssh -p $SSH_PORT' --delete $USER@$HOST:/srv/gacq/rsync/ rsync

gacq.com-put

# Run this at the notebook
# Upload new files to server, also remove files deleted here
#
HOST=gacq.com
USER=anysyncuser
SSH_PORT=22

rsync -avuze 'ssh -p $SSH_PORT' --delete rsync $USER@$HOST:/srv/gacq/