Welcome to the Info TEST server!

Skip to content. | Skip to navigation

Sections
Info Services > Computing Guide > Astronomy Support > How to copy data from/to ZFS filesystem

How to copy data from/to ZFS filesystem

This document explains how to move data to/from ZFS

The diagram above shows how the architecture how stash is connected with cluster nodes.

From Lustre to ZFS

To copy/move data from Lustre to ZFS we strongly recommend to use rsync. The best way to move data is through the infiniband network, in order to take advantage of the bandwith.


In the following example, we are going to copy 3 files (100G, 10G and 1G) from the directory /.lustre/aoc/sciops/lsaavedr/zfs/ to my zfs area /stash/users/lsaavedr/FromLustre. In this example, I am going to use nmpost039 to take advantage of the Infiniband network.

[lsaavedr@nmpost039 ~]$ mkdir /stash/users/lsaavedr/FromLustre
[lsaavedr@nmpost039 ~]$ ls -hl /.lustre/aoc/sciops/lsaavedr/zfs/
total 109G
-rw-r----- 1 lsaavedr nmstaff   98G Sep 23 13:55 test01-lustre-s100G.img
-rw-r----- 1 lsaavedr nmstaff  9.8G Sep 23 13:53 test01-lustre-s10G.img
-rw-r----- 1 lsaavedr nmstaff 1000M Sep 23 13:53 test01-lustre-s1G.img

[lsaavedr@nmpost039 ~]$ rsync -avh  /.lustre/aoc/sciops/lsaavedr/zfs/test01-lustre-s1* /stash/users/lsaavedr/FromLustre
sending incremental file list
test01-lustre-s100G.img
test01-lustre-s10G.img
test01-lustre-s1G.img

sent 116.42G bytes  received 73 bytes  275.55M bytes/sec
total size is 116.39G  speedup is 1.00

From ZFS to Lustre

In the following example, we are going to copy 3 files (100G, 10G and 1G) from the directory /stash/users/lsaavedr/lustre/ to my lustre area /.lustre/aoc/sciops/lsaavedr/FromZFS. In this example, I am going to use nmpost039 to take advantage of the Infiniband network.

[lsaavedr@nmpost039 ~]$ mkdir /.lustre/aoc/sciops/lsaavedr/FromZFS
[lsaavedr@nmpost039 ~]$ ls -l  /stash/users/lsaavedr/lustre/test01-lustre-s1*
-rw-r----- 1 lsaavedr nmstaff 104857600000 Sep 23 13:55 /stash/users/lsaavedr/lustre/test01-lustre-s100G.img
-rw-r----- 1 lsaavedr nmstaff  10485760000 Sep 23 13:53 /stash/users/lsaavedr/lustre/test01-lustre-s10G.img
-rw-r----- 1 lsaavedr nmstaff   1048576000 Sep 23 13:53 /stash/users/lsaavedr/lustre/test01-lustre-s1G.img

[lsaavedr@nmpost039 ~]$ rsync -avh /stash/users/lsaavedr/lustre/test01* /.lustre/aoc/sciops/lsaavedr/FromZFS
sending incremental file list
test01-lustre-s100G.img
test01-lustre-s10G.img
test01-lustre-s1G.img

sent 116.42G bytes  received 73 bytes  295.11M bytes/sec
total size is 116.39G  speedup is 1.00
Info Services Contacts
 
Search All NRAO