CSC Command-line Interface

As an alternative to CSCview, tables of catalog data may be downloaded directly through a URL with Unix command-line tools such as cURL and Wget, using the ADQL query synytax or the IVOA-compliant VO Cone Search service.

This page presentes the cURL and wget approaches.

(5 Aug 2016) The crossmatch interface has been updated and the previous row-limit restriction has been significatnly increased to 200,000 rows.

(20 Oct 2014) The curl examples have been updated to use the --form syntax which forces URL encoding (eg using "%20" for spaces) which is now required by the CSC services.

(13 Aug 2013) For users with CIAO installed, the additional basic catalog query tools search_csc and obsid_search_csc are now available and demonstrated in this CIAO thread.

ADQL Queries

All source properties contained in the catalog may be accessed directly from the following URL in an ADQL command-line query:

http://cda.cfa.harvard.edu/csccli/getProperties

This URL accepts the following parameters:

Output: A Tab Separated Values (TSV) or VOTable TABLEDATA format table containing the sources and associated source properties requested in the catalog query.

Crossmatch queries

When a user supplies a table for crossmatch via the UPLOAD parameter, an association table between the user_table and the CSC master_source table is called cross_match is available in ADQL.

The user table may contain arbitrary columns not all of which are used. The RA, DEC, and ID parameters map column names from the user table to the RA, DEC, and ID columns used by the crossmatch. The user table is not available for ADQL queries, the ID column is copied and can be queried from the cross_match table.

The cross_match table contains columns named:

      usrid       - a key copied from the user_table column identified by the ID parameter
      msid        - a key to join on the master_source.msid column
      separation  - a floating point column representing the separation of the
                    two sources in arcseconds
      probability - a floating point column representing the probability that
                    the two sources are the same  
                    
    

Data product queries

The CSC data products - e.g. spectra, light curves, and images - may be accessed at the command line, but only after interactively querying the catalog within CSCview. After submitting a query in the GUI and selecting the desired data products, users are brought to the Products tab where they are given the option of downloading the selected data products via a download script or tar file. The download script contains a list of Wget commands, one for each file, which can be executed on the Unix command line for a batch download. See the CSC thread Retrieving Data Products to learn how to access catalog data products.

ADQL syntax

An ADQL SELECT statement returns a result set of records from one or more tables of astronomical data, located by the FROM clause. The available tables are 'master_source m' (Master Sources Table), 'obi_source o' (Source Observations Table), 'master_obi_assoc a' (Master Sources/Source Observations Associations), 'dataset d' (Data Products), and 'crossmatch c' (Crossmatch) for the CSC. Some of the optional clauses of a SELECT statement include:

• TOP specifies the number of rows to retrieve.
• WHERE specifies which rows to retrieve, according to the search criteria.
• ORDER BY specifies an order in which to return the rows.

ADQL SELECT, TOP, FROM, WHERE, ORDER BY statements may also be entered into the ADQL window in the Query tab of the data access GUI CSCview. This view is accessed by selecting the menu item View->Query->Show Language while the Query tab is open.

Examples

1. Perform a basic source property query.

unix% curl -o out.file --form query='SELECT TOP 50 m.name, m.significance, m.flux_aper_b, m.alpha FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7)' --form outputFormat=votable 'http://cda.cfa.harvard.edu/csccli/getProperties'

unix% wget -O out.file 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT TOP 50 m.name, m.significance, m.flux_aper_b, m.alpha FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7) &outputFormat=votable' 

This query accesses from the catalog the master source name, significance, broad-band aperture energy flux, and power-law model best-fit photon index for the first 50 master sources found with a significance greater than 10.0, a hard-to-soft hardness ratio greater than 0.7, and pileup fraction less than 10%. The result set is returned in VOTable TABLEDATA format.

2. Perform a basic cone search query.

unix% curl -o out.file --form query='SELECT m.name, m.ra, m.dec, m.flux_aper_b FROM master_source m WHERE dbo.cone_distance(m.ra,m.dec,83.77333,-5.68464)<=10' 'http://cda.cfa.harvard.edu/csccli/getProperties'

unix% wget -O out.file 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT m.name, m.ra, m.dec, m.flux_aper_b FROM master_source m WHERE dbo.cone_distance(m.ra,m.dec,83.77333,-5.68464)<=10' 

This query accesses from the catalog the master source name, R.A and Dec., and broad-band aperture energy flux for all sources located within 10 arcminutes of the celestial position R.A.=83.77333, Dec.=-5.68464 (in decimal degrees).

3. Upload a query from a file to the URL.

unix% curl -o out.file --form query=@query.txt 'http://cda.cfa.harvard.edu/csccli/getProperties'

The file should contain a query written in ADQL 2.0, e.g. a query save file output by CSCview, like the one shown here.

4. Specify how a catalog value of NULL should appear in the output table (instead of a blank space).

unix% curl -o out.file --form nullAppearance=NULL --form query="SELECT DISTINCT o.obsid, o.obi, o.region_id, o.theta, o.mjr_axis_raw_s FROM obi_source o WHERE (o.instrument = 'ACIS' AND o.theta < 0.5 AND o.edge_code = 0 AND o.detect_significance_b > 10.0) ORDER BY theta DESC" 'http://cda.cfa.harvard.edu/csccli/getProperties'

unix% wget -O out.file "http://cda.cfa.harvard.edu/csccli/getProperties?nullAppearance=NULL&query=SELECT DISTINCT o.obsid, o.obi, o.theta, o.mjr_axis_raw_s FROM obi_source o WHERE (o.instrument = 'ACIS' AND o.theta < 0.5 AND o.edge_code = 0 AND o.detect_significance_b > 10.0)"

This query accesses from the catalog the ObsID, OBI, off-axis angle, and (full) source ellipse major axis length for all distinctly identified ACIS-observed sources found in the Source Observations Table with an off-axis angle less than 0.5, a detect significance greater than 10.0, and which do not dither off chip boundaries during the observation. Here, what normally would appear as an empty character in the output table will appear as "NULL".

5. Specify the output R.A. and Dec. coordinate format. Default is sexagesimal.

unix% curl -o out.file --form 'query=SELECT TOP 1000 m.name, m.ra, m.dec FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7)' --form coordFormat=decimal   http://cda.cfa.harvard.edu/csccli/getProperties

unix% wget -O out.file 'http://cda.cfa.harvard.edu/csccli/getProperties?query=SELECT TOP 1000 m.name, m.ra, m.dec FROM master_source m WHERE (m.significance > 10.0 AND m.pileup_flag = 0 AND m.hard_hs > 0.7) &coordFormat=decimal'

This query accesses from the catalog the master source name, R.A. and Dec. for the first 1000 sources with a master source significance greater than 10.0, a hard-to-soft hardness ratio greater than 0.7, and a pileup fraction less than ~10%. The R.A. and Dec. values are output in decimal degrees.

6. Specify that floating point numbers should appear in native format.

unix% curl -o out.file --form 'query=SELECT top 10 o.obsid,o.obi,o.targname,o.ra_targ,o.dec_targ,o.ra_pnt,o.dec_pnt,o.roll_pnt,o.ra_nom,o.dec_nom,o.roll_nom,o.gti_start,o.gti_stop,o.gti_elapse,o.gti_end,o.gti_obs,o.gti_mjd_obs,o.mjd_ref,o.instrument,o.grating,o.datamode,o.readmode,o.cycle,o.exptime,o.timing_mode,o.ascdsver,o.caldbver,o.crdate,o.ao FROM obi_source o' --form floatFormat=native  http://cda.cfa.harvard.edu/csccli/getProperties

This query accesses from the catalog many observation-specific properties for the first 10 sources found in the Source Observations Table, with source property values output in native floating point format (the "native" catalog format includes numbers with many significant digits, exactly as they are output by the catalog processing pipeline).

7. Use the current database view as opposed to a specific release view.

unix% curl -o out.file --form "query=SELECT top 5 m.name,m.ra,m.dec,m.err_ellipse_r0,o.obsid,o.theta,o.flux_significance_b,o.conf_code,o.extent_code,o.sat_src_flag,o.mjr_axis_raw_b,o.mnr_axis_raw_b,o.pos_angle_raw_b,o.cnts_aper_b,o.src_rate_aper_b,o.src_rate_aper_hilim_b,o.src_rate_aper_lolim_b,o.flux_aper_b,o.flux_aper_hilim_b,o.flux_aper_lolim_b,o.hard_hm,o.hard_ms,o.var_index_b,o.livetime,o.detector,o.posid FROM master_obi_assoc a , master_source m , obi_source o WHERE ((a.match_type = 'u')) AND o.posid=a.posid AND m.msid=a.msid ORDER BY name ASC" --form version=current http://cda.cfa.harvard.edu/csccli/getProperties

This query accesses source properties from both the Master Sources and Source Observations tables, for the first five unambiguously identified sources in the current catalog database (as opposed to a static 'release' version of the database, frozen in time). The results are sorted on the master source name column; an example of a CSC master source name is 'CXO J080238.1-395249.'

8. Find CSC counterparts to sources of interest by performing a cross-match against the CSC using an input table of source positions.

unix% curl \
  --request POST \
  --form 'query=select m.name,m.ra,m.dec,c.usrid,c.separation,c.probability from cross_match c, master_source m where c.msid = m.msid' \
  --form RA=ra \
  --form DEC=dec \
  --form ID=my_catalog_id \
  --form UPLOAD=user_table,param:my_catalog \
  --form my_catalog="@/data/somewhere/table.xml" \
  http://cda.cfa.harvard.edu/csccli/getProperties

This query accesses from the catalog the separation in arcseconds between the matched catalog master sources and the corresponding sources in a user-input table, along with the names and RA & Dec. positions associated with the catalog sources in the CSC Master Sources Table. The probability value of each source match is also returned, where a value of 1.0 indicates that the returned match is very likely a true match (down to many significant digits in source position), and a value of 0.0 represents a very unlikely true match. The my_catalog_id column values in the input table is returned in the c.usrid output column for the matching sources.

VO Cone Search

A VO Cone Search service is available for conducting a search on source position in the CSC. It follows the IVOA cone search recommendations, with a fixed-call syntax which does not get combined with CSC cone searches in ADQL. The command-line query specifies a sky position and an angular distance to be used as the search radius, defining a cone on the sky. The query returns a list of astronomical sources from the catalog whose positions lie within the cone, in VOTable format.

The base URL for the VO cone search is http://cda.cfa.harvard.edu/cscvo/coneSearch

Examples

unix% curl 'http://cda.cfa.harvard.edu/cscvo/coneSearch?RA=83.77333&DEC=-5.68464&SR=.233&VERB=2'

unix% wget -O out.vot 'http://cda.cfa.harvard.edu/cscvo/coneSearch?RA=83.77333&DEC=-5.68464&SR=.233&VERB=2 '

See example output in 'out.vot' produced by the Wget example above.

In the VO Cone Search, one searches on RA, DEC and SR (search radius) in decimal degrees, and receives one of the three available result sets, based on the value of the parameter VERB (verbosity):

• verbosity = 1 reports the source name, RA and DEC values from the Master Sources Table for each matching source resulting from the cone search
• verbosity = 2 reports the master source summary results and Space Time Coordinate (STC) metadata
• verbosity = 3 reports all the columns in the Master Sources Table

The master source summary returned by verbosity level 2 is the result set of one of the standard queries available in CSCview. It includes the following CSC master source properties:

• name
• ra
• dec
• err_ellipse_r0
• conf_flag
• sat_src_flag
• significance
• flux_aper_b, flux_aper_lolim_b, flux_aper_hilim_b
• flux_aper_w, flux_aper_lolim_w, flux_aper_hilim_w
• extent_flag
• hard_hm, hard_hm_lolim, hard_hm_hilim
• hard_ms, hard_ms_lolim, hard_ms_hilim
• var_intra_index_b
• var_inter_index_b
• var_intra_index_w
• var_inter_index_w