Welcome to the Info TEST server!

Skip to content. | Skip to navigation

Sections
Info Services > Computing Guide > Cluster Processing > Data Storage and Retrieval

Data Storage and Retrieval

Data Storage

The NRAO has two Lustre file systems: one at the NAASC (/lustre/naasc) and one at the NMASC (/lustre/aoc). Lustre is a parallel distributed filesystem used in many large-scale computing facilities. It allows NRAO desktops, public machines and clusters at a particular site to share a large file space thus removing the need for repeatedly copying data between systems for processing. They are primarily designed for performance, which is achieved by aggregating individual disk throughput across a large number of disks. As a side effect, the resulting storage volume is typically large compared to desktop storage.

Observer accounts like nm-4386 reside on the NMASC Lustre filesystem in /lustre/aoc/observers/<account name> while observer accounts like cv-4386 reside on the NAASC Lustre filesystem in /lustre/naasc/observers/<account name>.   Observers should store all data products and scratch files in this area.   NRAO staff will need an area like /lustre/aoc/users/ or /lustre/naasc/sciops/ set up for them by the local IT Helpdesk.  Lustre is a shared resource among staff and observers, so we ask that everyone keep their usage as far below the 5TB limit as possible.

NRAO staff users, please do not use your space on the filer (e.g. /users/krowe) to store large data as you probably have a quota measured in GBs nor use it for processing as it can be around ten times slower than Lustre.

For NMASC (nmpost) please see the following special instructions for retrieving data directly from the archive into a directory you own.

For more information on Lustre see the Lustre FAQ in the Appendix.

 

Data Retrieval

The NRAO supports the following methods for securely transporting data to remote facilities and has plans to support XSEDE's Globus Connect platform.  For the following examples, <account name> would be your nm-* or cv-* account name.

SFTP

SFTP is an encrypted ftp protocol.  Once connected, sftp behaves much like any ftp client.

NMASC (nmpost)

sftp <account name>@sftp.aoc.nrao.edu

NAASC (cvpost)

sftp <account name>@sftp.cv.nrao.edu

SCP

SCP is an encrypted copy that can transfer between remote hosts.  The format is scp <user>@<remotemachine>:/<remote path> <local path>.

The example below would copy all files ("*") in <account name>'s data sub-directory to the current directory (".") on your local machine.

NMASC (nmpost)

scp <account name>@ssh.aoc.nrao.edu:/lustre/aoc/observers/<account name>/data/* .

NAASC (cvpost)

scp <account name>@ssh.cv.nrao.edu:/lustre/naasc/observers/<account name>/data/* .

 

LFTP

LFTP is a more sophisticated version of the classic ftp protocol which, among other things, uses multiple channels to speed performance.

NMASC (nmpost)

lftp -u <account name> sftp://sftp.aoc.nrao.edu

NAASC (cvpost)

lftp -u <account name> sftp://sftp.cv.nrao.edu

RSYNC

RSYNC is a versatile, file-copying tool that only copies necessary files; that is the ones that are missing in your local copy.  This is useful if, for example, you have deleted some files from your local copy and want to copy just those missing files.

The example below would copy all the files in <account name>'s data sub-directory to a local directory on your machine.  Without the trailing "/" rsync would copy the directory and its contents , with a trailing "/" it copies only the contents of the directory.  Adding "--delete" to the arguments list will keep the two areas exactly in sync by removing files on your local machine if they have been removed from the remote copy.

NMASC (nmpost)

rsync -vaz <account name>@ssh.aoc.nrao.edu:/lustre/aoc/observers/<account name>/data/ .

NAASC (cvpost)

rsync -vaz <account name>@ssh.cv.nrao.edu:/lustre/naasc/observers/<account name>/data/ .

Browser Access

You can navigate through your filesystem on a browser to view and download files, but this is not recommended for large transfers. You will need to login using your account name and http://my.nrao.edu/ password.

NMASC (nmpost)

https://archive.nrao.edu/observers/<account name>

NAASC (cvpost)

https://bulk.cv.nrao.edu/observers/<account name>

 

GlobusOnline

The NRAO plans to add a Globus Connect portal.

Info Services Contacts
 
Search All NRAO