The Pool Observation Database System (ODS) and
the Data Reduction Pipeline at the IRAM 30m Telescope

Pool Observations

Most cutting edge scientific programs carried out at the IRAM observatories require excellent weather conditions. While the service observing mode at the Plateau de Bure interferometer allows a flexible scheduling of demanding programs, fixed scheduled projects at the 30m telescope often suffer from limitations due to inadequate weather conditions. To optimize the observing efficiency given the often quickly changing weather conditions, IRAM offers the pool observing mode at the 30m telescope. In this mode all bolometer and some very demanding spectroscopy programs share the allocated observing time together with some less weather demanding spectroscopy programs. Projects are observed according to the current weather conditions, their ranking from the program committee and the source visibility. This way the chances of all demanding projects have drastically improved: the average success rate of A ranked projects requiring good or excellent weather conditions in the last winter pool was 80%, compared to 20% of good weather conditions during the allocated pool observing time (Fig. 2).

  

Figure 2: Project and weather statistics during the pool

The pool observations are conducted mostly by the astronomers who's programs are included in the pool. They are aided by the IRAM astronomers and telescope operators. There are typically three guest astronomers, one IRAM astronomer, and the telescope operators at the telescope. Given the flexible use of different instruments and the large variety of scientific programs, pooled observations are part of IRAM's training efforts for students.

Pool observations were first tested at the IRAM 30m telescope during the winter 2000/2001, and have now expanded to 12 weeks in the winter semester,and about 4 weeks in the summer. The winter 2003/2004 pool included almost 50 different programs with more than 500 different targets. To manage such a pool requires an efficient organizational structure, so that at any time the status of a program, target priorities and weather/technical requirements can be assessed. This motivated the observational database system described here.

The PHP/MySQL software (http://mrt-lx3.iram.es/pooldatabase/index.php) system was first conceived by Alexandre Beelen (IAS Paris) and Frank Bertoldi (MPIfR Bonn) in December 2001, and subsequently developed further by Axel Weiss, who as an IRAM astronomer serves as the pool coordinator at the 30m telescope. All data reduction tools have been developed and are maintained by Robert Zylka (IRAM).

Technical Summary ODS

The IRAM pool observation database system (ODS) is based on PHP scripts executed from an apache web-server, which accesses a MySQL database. The database contains detailed information on all projects including technical and metrological requirements. Observational information is read from the fits data headers of each scan. This information is easily associated with the respective projects, and thereby allows bookkeeping, planning, data quality control, and in connection with an external software an automated pipeline data reduction. The system also permits data archiving and the easy access/download of data for the users. The users interactively enter and modify their target lists, observation instructions, and they are able to check the state of the project and the data quality.

The software used in the development of the ODS is part of all Linux distributions. Its main constituents are PHP and MySQL. PHP (recursive acronym for "PHP: Hypertext Preprocessor") is a widely-used Open Source general-purpose C-based scripting language that is especially suited for Web development and can be embedded into HTML (see e.g. Welling & Thomson 2003). The PHP package contains functions which allow to retrieve and store information in a MySQL database. MySQL is the most popular Open Source SQL (Structured Query Language) database management system. It provides an easy way to store and sort large amounts of information and to give limited access to different users on the MySQL server (e.g. DuBois 2000). MySQL in conjunction with PHP is becoming the most commonly used platform for web-based data access and processing applications. The database system at the IRAM 30m currently runs on PHP version 3 and MySQL 3.23.44 on SuSE Linux distribution, but has also been installed on RedHat.

The main body of ODS is a collection of PHP scripts (the PHP engine) which perform different kinds of operations. Information can be received and/or stored in the MySQL database (e.g. the amount of observing time spent on a specific project). Other PHP scripts generate files associated with the observations such as the observing log-files, the 30m source catalogs or Xephem files for astronomical visualization. PHP also allows to call external programs. This way the PHP engine can parse a list of scans to an external data reduction software, e.g. the IRAM continuum data processing software MOPSIC, and store data information such as the achieved rms in the MySQL database. All PHP scripts are executed by an Apache web-server. Thereby all information can be accessed through the Internet (see Fig. 3).

  

Figure 3: Online database system flow diagram

Access is restricted by PHP and by MySQL security applications as well as the Apache web-server configuration. This allows to setup different user accounts in ODS with limited access to specific information. For more information see http://mrt-lx3.iram.es/flexible/ods/

MOPSIC Monitor and Data Reduction Pipeline

The monitor

We have installed a new tool which allows the observer to see online the progress of observations. The data is displayed on mrt-lx10:0.1 at the observing desk. A MOPSIC script reduces automatically incomming data every 15 to 60 sec. The reduced observations are shown in the top left corner of the screen. Other plots show:

• the total power and the line-of-sight opacity calculated from it (bottom left plot)
• the rms of the data (bottom centre)
• the time series of a few bolometers before (in red) and after (in blue) the sky noise filtering (bottom right plot)
• the top right plot shows - depending on the observing mode - either the double beam map (for pointing, focus and map), the time series of the on-source bolometer (for OnOffs) or of the total-power bolometers (for skydips)

These additional plots allow to judge the quality of the data and the weather conditions during the observations, and therefore to optimize the observing strategy. The monitor warns if the observational parameters are not valid, e.g. if the map extent in scanning direction is too small, the pointing bolometer is not the central one for mapping, or if an OnOff data set has an odd number of subscans, a.s.o.. In such cases it changes the default reduction mode (if possible) or does not reduce the data.

Two examples of the graphical output of the monitor are shown in Fig. 4 and Fig. 5.

  

Figure 4: MOPSIC monitor (Pointing)

  

Figure 5: MOPSIC monitor (Mapping)

The ODS pipeline

For a quick look at the observations ODS provides the PIs and the observers with a pipeline reduction for all kinds of bolometer observations. The reduction is based on MOPSIC scripts which are developed and maintained by Robert Zylka. The pipeline reduction offers data reduction with and without skynoise subtraction. For maps also a shift & add reduction is available. The reduction scripts are executed by pressing the link associated with each source name as soon as the observations for this source have been carried out. The pipeline reduction window allows to select the scans which should be included in the reduction. For mapping projects the rms in each map as well as the combined rms are computed and entered in the MySQL database. The users can view the combined and individual maps. For OnOff projects the cumulative rms of all observations is determined and saved in the database. Detailed plots of the signal and the rms for individual scans and the sum of all scans are provided. This allows the PIs to continuously optimize the observing strategy during the pool. The online reduction also allows the observers to check the data quality very quickly and to estimate the achieved S/N. Contrary to the monitor which only emits a warning, the pipeline refuses to reduce data with invalid observational parameters, e.g. odd number of subscans for OnOffs, map extent in scanning direction too small (for no shift & add mode).

All pipeline scripts together with necessary calibration files can be obtained via the ODS. For further informations about the pipeline see ``About the IRAM Database'' at the ODS web page (http://www.iram.es/IRAMES/observing/flexible/flexible.html) and the README file available with the data reduction scripts.

References

DuBois, P., 2000, MySQL, New Riders Publishing
Welling, L. & Thomson, L., 2003, PHP and MySQL Web Development, Sams

Axel WEISS and Robert ZYLKA