Recursive SCP, An Efficient Way
At the back of my mind I was wondering whether we can make use of UNIX pipe to achieve all this. Not only I do not have to create a temporary tarball, also it should be pretty efficient to take advantage of the stream of data flowing over TCP/IP to keep the window size to the maximum.
Below experiment was carried from my Cygwin tmp directory (525K bytes in total, with 59 files and 5 sub-directories) to be copied over to my office CentOS5 box via broadband connection.
time scp -r tmp chihung@$MY_CENTOS5:. > /dev/nullreal 0m16.828s user 0m0.138s sys 0m0.139s $ tar cfzp - ./tmp | time ssh chihung@$MY_CENTOS5 tar xfzp -0.07user 0.06system 0:04.12elapsed 3%CPU (0avgtext+0avgdata 413440maxresident)k 0inputs+0outputs (1635major+0minor)pagefaults 0swaps
You can see that we are talking about 16.828 seconds vs 4.12seconds. Also, the 'tar' way can compress the stream and preserve the file permissions. The above ssh connection has been setup to be password-less to avoid additional time required to login.
Not all UNIX systems comes with GNU tar that can do gzip (-z) and Solaris is one of them. We can do the same trick to combine gzip and gunzip at both end to achieve the same effect as GNU tar. Also, to ensure the data transferred over is intact, you can do a md5sum on the tar stream.
tar cfp - ./tmp | gzip | time ssh chihung@$MY_CENTOS5 "gunzip | tar xfp -"0.09user 0.04system 0:04.09elapsed 3%CPU (0avgtext+0avgdata 413184maxresident)k 0inputs+0outputs (1634major+0minor)pagefaults 0swaps $ tar cf - ./tmp | md5sumd46a7b5985d0ea408186222c0257405f -