START08/Pre-HIPPO Data Management

Useful links

• Quick-look data server
• Field catalog
• Project archive
• Texas A&M forecast data
• START08 project web site (NCAR/ACD)
• START08 project web site (NCAR/EOL)
• HIPPO project web site (NCAR/EOL)

Data Management Plan

The data management and modeling component of the START08-HIPPO project has four primary goals:

• Provide real-time support for meteorological forecasting and flight planning.
• Provide quick-look data files and analysis products as soon as possible after each flight.
• Provide access for the START08 and pre-HIPPO teams to preliminary data sets during the period prior to the final data submission deadline.
• Provide a long term archive for START08-HIPPO related data sets.

Data Access and Upload

Accessing the data server

A login is required to access the Quick-Look Server (start08.tamu.edu). Project members who need access to start08 to read or write data sets should contact Ken Bowman and request a login. Please provide your preferred login id and Unix shell preference (if you have one). A password will be provided via e-mail.

The data server is a Mac running Mac OS X Server 10.5. (OS X is a flavor of Unix.) Users can access the server by using ssh/scp or by secure ftp (sftp):

• Sample session using ssh/scp from the command line:
In this case the user is logging in from the host named stretch. The file to be moved is in my home directory in the subdirectory start08_files on the host bowman.tamu.edu. This is an example of using scp to "get" a file from a different host by logging into start08.tamu.edu.

stretch> ssh bowman@start08.tamu.edu
Password:
start08> cd /Volumes/data/instruments/preliminary
start08> ls
UCATS_GC/	UCATS_H2O/	UCATS_O3/
start08> mkdir dummy
start08> cd dummy
start08> scp bowman.tamu.edu:~/start08_files/start08_rf01_dummy.prelim_v01.txt .
Password:
START08_rf01_dummy.prelim_v01.txt                 100% 2262KB  2.2MB/s  00:01  
start08> ll
total 4525
-rw-r--r--  1 bowman admin 2316428 Aug 29 09:10 start08_rf01_dummy.prelim_v01.txt
start08> cd ..
start08> ls
UCATS_GC/	UCATS_H2O/	UCATS_O3/	dummy/
start08> exit
logout
Connection to start08.tamu.edu closed.
stretch>

• Using sftp:
1. It is probably easier to use an ftp client program that supports sftp, but you can also use sftp from the command line.
2. Configure your client to connect to the host start08.tamu.edu using sftp.
3. Provide your login ID and password.
4. You can set your initial diredtory to /data.
• From a web browser:
1. Note: this is download only, no upload.
2. Connect to http://start08.tamu.edu/data.
3. Authenicate with your login ID and password.
4. Navigate to the file your want.
5. Use your web browser's file download mechanism.

Directory structures

The data archive is organized into directories as follows

/data
   /instruments
      /quick_look
         /AO2
            start08_rf01_ao2meas_quick_v01.txt
            start08_rf02_ao2meas_quick_v01.txt
            ...
         /AWAS
         ...
      /preliminary
         /AO2
            start08_rf01_ao2meas_prelim_v01.txt
            start08_rf02_ao2meas_prelim_v01.txt
            ...
         /AWAS
         ...
      /final
         /AO2
            start08_rf01_ao2meas_final_v01.txt
            start08_rf02_ao2meas_final_v01.txt
            ...
         /AWAS
         ...

In the new structure, data files will be organized at the top level into three status categories:

• quick_look - hot off the airplane
• preliminary - due July 31, 2008; some quality control; suitable for preliminary analysis (see footnote below)
• final - due December 31, 2008; final quality-controlled data files for science use

Note the addition to the file name convention of 'quick', 'prelim', or 'final'.

Because each file contains the flight number, under the new scheme the flight directories are redundant and are eliminated. If you upload an updated data file within a given status category (e.g., a new version of a preliminary file), increment the version number. The merging process will use the highest available version number. Please don't overwrite or delete previous versions. We need that information to keep track of changes in data content.

From this point on, mergers will be done on specified dates, rather than on an ad hoc basis as new or updated data files arrive. A contents file will be created in the relevant merged directory with a list of the data versions used in each merged file.

Instrument PI's should create a directory for their instrument(s) in the /instrument directory. Please organize your directories into subdirectories by flight name/number.

File name specification

PI's should use the following file naming convention. Here is a sample instrument file name:

start08_rf01_UCATSGC_prelim_v01.nc

Each file name has five parts separated by underscores, and a suffix, separated by a period. Please do not use periods or underscores in the product or instrument name. Following a suggestion by Steve Wofsy, please use the platform 'GV' as the file suffix for Ames-style ASCII files.

File Name Components
Part	Example
Project name (all files begin with this)	start08
Flight type and number (use two digits)	rf01
Product or instrument name	gv, gfs, vcsel, etc.
Stage	quick, prelim, or final
Version number (use two digits)	v01
Suffix	.txt, .nc, or .GV

Examples of File Names
File Type	Example File Name	Source
GV files	start08_rf01_gv_v01.nc	RAF
Instrument files (example)	start08_rf01_vcsel_quick_v01.txt	PI's
Selected RAF & GFS vars	start08_rf01_gfs_v01.nc	Bowman
Merged RAF/GFS/instrument)	start08_rf01_merge_v01.nc	Tilmes

Long-Term Project Archive

Long-term archiving of project data sets will be provided by NCAR EOL. The preliminary version of the START08 archive is here.