If this is your first time
If you are my postdoc, and if you need me to revise English or double check the technical setup, forward me your first draft at least 2 weeks ahead of the proposal deadline.
If you are my student, forward me your first draft at least 4 weeks ahead of the proposal deadline. I will spend time iterating with you many times for education purposes.
1. Proposing
1.1 Timeline and interface
There are two proposal deadlines each year. They are normally in mid-March and mid-September but please pay attention to the call for proposals.
Please register an account at the SMA Observer Center well ahead of the proposal deadline (e.g., one month ahead or earlier). You need to log in and familiarize yourself with how to prepare for the cover-page online.
Usually, you try to form a rough idea about what you are going to propose (target source(s), resolution, spectral setup, and sensitivity). Let’s say, this is at least 1~1.5 months ahead of the proposal deadline. If you are my student and postdoc, please come discuss with me about this rough idea.
1.2 Preparation-1: Feasibility, complementary data, and duplication
After we agree that this idea is worth pursuing and is technically feasible , you should use the SMA Data Archive to check whether or not you are duplicating historical or ongoing projects. In the case that you are duplicating, you can (1) give up and try to come up with other proposal ideas, or (2) download the archival data can see if the calibrated data can achieve the quality that is needed for your science purpose. If the quality of the archival data is fair, then just continue to use that data to proceed with your research. If there is no archival data or if the quality of the archival data is poor, you can go ahead to complete the preparation of your proposal. You need to explicitly and convincingly explain what was the problem with the archival data if it exists. So prepare your proposal well ahead of the deadline in case you really need to look into some archival data or need other colleagues to check the archival for you. Close to the deadline, they (and I) have to prepare their own proposals thus may refuse to help you.
Do not waster your (and my) time to prepare a proposal that has a problem with unnecessary duplication. Such proposals will be rejected for 100% sure.
When making the duplication checks mentioned above, you may find that there are other observations on your target source(s) which can help your science case to some extent although they do not fully resolve the problem. For example, you may find that some short-spacing observations for your high-angular resolution experiment already exist. In this case, explicitly mentioning how you will make use of such complementary data in the proposal is a plus. Just mentioning you will use those data but without saying how usually does not help. If you have also checked the data archive of the observatories (e.g., ALMA, JVLA, Herschel, Spitzer, etc) and found some useful complementary data, you can also mention how you will make use of them in a joint analysis.
1.3 Preparation-2: Scientific justification and technical details
To complete the proposal submission, you need a title, an Abstract, a PDF scientific justifications with 2 pages of text and 2 pages of Figures/Tables. If you are not sure about how to compose the Abstract or the 2+2 pages scientific justification, check here.
Usually, we prepare the 2+2 pages justifications using LaTEX. If you are my student/postdoc, please use the Overleaf interface and share the link with me, if you need my revision. Use fontsize 12 as far as you can. Never make fontsize smaller than 11. Do not use a very small margin. Try not to make your proposal over texty. No reviewer wants to spend more than 5 minutes reading your proposal. Do not make it necessary for them to do so.
You will need to include your observing time requests through the online interface at the SMA Observer Center. The observation for each target source at each array configuration with each receiver tuning need an independent observing time request that states how many hours you need for that target source at that setup. Based on the record(s) of your observing time request, the reviewers will check whether or not you are duplicating historical or ongoing projects unnecessarily.
You can check your spectral setup (i.e., receiver tuning) at this page. Usually, SMA does not re-tune the receiver for a PI in a night (e.g., it will not consider a project that requests to make short observations on one single target with >2 different receiver tunings; using two different tunings in one project in a night is already very rare since that costs a lot of overhead time in the (re-)tuning and in the passband calibration). If you are not sure about what tuning is feasible or not feasible, come discuss with me.
You can check the angular resolution and sensitivity at your observing frequency at this page.
Use this page to check which gain and passband calibrators are ideal for your project. Usually, I try to pick a >0.7 Jy quasar as gain calibrator. It is the best if the angular separation between this quasar and your target source is less than 10 degrees. When the separation is in the range of 10~20 degrees, the phase errors can be noticible. A quasar that is >20 degrees separated from your target source should not be considered as you gain calibrator unless you are sure that your target source can be self-calibrated. Passband calibrator can be 3C84 or 3C279; bllac may be OK if you are not observing CO J=2-1 or CO J=3-2.
You can use this page to check which Solar System object can be your absolute flux calibrator. But don’t worry too much about this. The SMA scheduler always double-checks which is the best one on the date of your observations.
1.4 Preparation-3: Scopes (being sufficiently ambitious)
A regular SMA proposal can request up to 100 hours of observing time. If you need 100~1000 hours, there is a Large Project category which is relatively hard. If you want to submit a Large Project proposal, prepare early. If you are student or postdoc in Taiwan, I would consider that it is fair to request 2~4 nights of observing time in each proposal. If you are a Master’s student and if your thesis is going to base on this SMA proposal, I would encourage you to think big and consider a project with 5~10 nights of overall observing time. A night corresponds to 8~10 hours.
Pick a problem that can be solved by such an investment resource. You can think bigger once you have enough reputation to convince the reviewers that you can manage a larger-scale project.
2. Organizing observing scripts
Here is a step-by-step guide:
Check the previous section if you are not sure how to pick calibrators (the relevant parts are in between the two horizontal gray lines); check this page if you are confused about any thing (or everything).
- Log in to the SMA Observer Center.
- Click the My Projects tab on the left.
- From the list of your project, click into the relevant one. You will see the allocated observing tracks in the Summary of tracks section in the bottom. If there is not yet an observing script, there will be red text saying (needs script).
- Click Track overview & script next to the (needs script) text.
- Scroll down to the Observing Script section. If you have never prepared an SMA script, then click create new script using script generator. If you are experienced, or if you are modifying a previously created script, you can click manage script directly will allows you to edit the ASCII script file.
- The online script generator has self-contained explanation. You can follow through what it asks you to fill.
- After a preliminary script is prepared using the script generator, and after you save the script, the online interface will allow you to enter the prospective UTC time of this observation and then simulate how the script works. After the start of a semester, a rough array configuration schedule will be provided here. From there you can check in which month your project may be executed, and then use the simulator to see if all the calibrators can be observed above the elevation limit, and if the cycle time(s) of the target source loop(s) is indeed what you expect. You should definitely run this simulator and make sure there is no error message (e.g., any syntax error in your script that cause failure, or if your target source or calibrator goes above/below the elevation limit [85 degrees for target source and gaincal; 81 degrees for absolute flux calibrator; 87 degrees for passband calibrator]).
- Sometimes you may need to control the LST start/end time of your target source loops (e.g., when you have multiple groups of target sources that need different gain calibrator, or when you want to control the duration of each target source loop). You can then click manage script directly can use the command like the following example:
print "----- main science target observe loop (vytau, xztau) -----\n";
$LST_start=20; $LST_end=2;
&ObsLoop(cal1,targ3,targ4);
print "----- main science target observe loop (v1143ori, v1183ori, nyori)-----\n";
$LST_start=1.5; $LST_end=12;
&ObsLoop(cal0,targ1,targ2,targ3);
The print commands does not do anything to the telescopes; it just prints some text to let the operator see what the array is going to do. Do simulate the script after making such edits in case of any surprise.
If you are a experienced user who are preparing scripts for the SMA polarization observations, scroll down to a template in the bottom of this page.
3. Operation
A SMA operator will execute your observing script and will watch over the observations. So usually, there is nothing you need to do. But if you would like to participate in the operation or would like to monitor the operation, let me know.
4. Data calibration and imaging
4.1 Software
You will need some software. An overview of them is provided here.
The official software package to calibrated the SMA data is MIR IDL. You can find examples of MIR IDL calibration procedures in that MIR IDL webpage (or click here for the 2016 version and here for the 2018 version). To use MIR IDL, you will need to install IDL. It is not free (it is ultra expensive). If you do not have an access to an IDL license, come ask me.
Note that MIR IDL will load all raw data in a file to the memory of your computer. The loaded data will in fact occupy twice the raw data filesize in the memory due to using double precision variables (e.g., if the raw filesize is 100 GB, then it will occupy 200 GB in your memory). If the raw filesize is too large for your computer, you can use the SMA Data Rechunker code to bin the spectral channels to reduce the filesize, before loading into MIR IDL.
You can also use the CASA software package to calibrate the SMA data, which is less subject to memory issue (by design, CASA can handle large raw data with a computer with relatively small memory, although it can be very slow if the memory is way too small). By the time of early 2022, the CASA data reduction path remained somewhat experimental.
I usually use the Miriad software package to image the SMA data. You can also use it to calibrate the SMA data. You can also use CASA to image your SMA data although it is a lot slower than Miriad.
4.2 Overview
No matter which software we use steps we are going through the following steps:
- Load raw SMA data to MIR IDL format (and store it if you like).
- Inspect data and flag (bad) the problematic data. You can then omit using the flagged data by unselecting them.
- Convert the raw visibility amplitudes to physically meaningful units (e.g., Jy) and undone the atmospheric attenuation exp(-tauatm). This step is called Tsys application.
- Perform passband calibration.
- Derive the absolute flux level of the gain calibrator by referencing to an absolute flux calibrator (e.g., usually a planet, planet moon, or the young star MWC349a). We call this step the absolute flux calibration.
- Perform complex gain calibration.
- Export data to Miriad or FITS format.
You may have some confusion about Tsys application since:
- In some textbook, Tsys is defined as (Trx + neffTsky + (1- neff)Tambient)/neff, where neff if the forward efficiency of the telescope, Trx, Tsky, and Tambient are the receiver temperature, sky temperature, and the ambient temperature, respectively. SMA and ALMA defines Tsys as (Trx + neffTsky + (1- neff)Tambient) / ( exp(-tauatm) neff ).
- Some observatories already converted the raw data to physically meaningful units, while the raw SMA visibilities represent the fraction of correlated signal.
The Tsys application step multiplies the raw SMA visibility with the Tsys values to undone the atmospheric attenuation and to rescale the visibility to Jansky (or Kelvin) units.
4.3 MIR Data calibration (no polarization)
4.3.1 Reducing filesize
The raw data files of the SMA can be as large as 50~100 GB. If you load a raw data file with the MIR IDL software package, it will requires a memory that is at least twice your raw data file size, due to converting the data to double precision. If the memory of your workstation is not so big, you have to reduce the filesize by binning the spectral channels. This can be achived by running the c code, SMARechunker. Following are the commands to download and compile it.
git clone https://github.com/Smithsonian/SMARechunker
cd SMARechunker
make
This will produce an executable, SMARechunker. You can then set alias in your start-up shell, e.g., .bashrc. The following command bin every 4 spectral channels, which will produce a file that is 4 times smaller than the raw SMA data file. The file format is the same as the raw SMA data file.
SMARechunker -i input_filename -o output_filename -r 4
4.3.2 Environment
I assume you are in an x86_64 Linux (e.g., CentOS, Ubuntu, Rocky, Redhat, etc) environment. If you have installed MIR IDL, or if somebody has installed it (e.g., if you are using a workstation at ASIAA), you need to source the setup file to enable using it. For example, I have my own installation, then type (revise to your own path):
(base) [hyliu@hyliu ~]$ source /home/hyliu/softwares/astro_reduction/MIR/sma-mir/setup.bsh
If you do not want to type this every time you launch a new terminal, you can include the following alias in your .bashrc file:
alias getmir="source /home/hyliu/softwares/astro_reduction/MIR/sma-mir/setup.bsh"
Then you can source MIR IDL and launch by typing:
(base) [hyliu@hyliu ~]$ getmir
(base) [hyliu@hyliu ~]$ idl
IDL 8.7.3 (linux x86_64 m64).
(c) 2020, Harris Geospatial Solutions, Inc.
Licensed for use by: Academia Sinica
License: 60400
% Compiled module: PRO_MIR_INIT.
% Compiled module: PRO_SETUP_ENV.
% Compiled module: PRO_CONSTANTS.
idl_startup completed
Type the following command to allow using color instead of black/white only.
IDL> device, decomposed=0, retain=2
4.3.3 Full procedure
I am taking the calibration of Chia-Ying’s project SMA 2021B-A011 (one receiver and one sideband) as an example. I have binned every 16 spectral channels using the SMARechunker code at a workstation of SMA in Hawaii:
# under linux command line
/application/bin/SMARechunker -i /data/science/mir_data/220524_05:57:34 -o ./sma_2021Ba011_track1 -r 16
Until the data become archival, please contact us if you need the raw data for an exercise. The following is a full MIR IDL procedure. Each line is a command for an IDL command line (i.e., you copy and paste the command to the command line). Lines starting with a # symbol are comment lines. If you are my student, you are required to preserve your calibration commands in such a format in case any of us (me, your co-Is, or the reviewers of your paper, etc) need to check certain details. Do Bookkeep everything strange or suspecious in the Notes sections.
##### Software ###########################################################
# (Data reduction in RTDC 5)
linux> source /home/hlu/software/MIR/mir/setup
IDL> device, decomposed=0, retain=2
# ASIAA (Please ssh -Y to almat5, almat6, alma7, alma8)
# linux> source /asiaa/home/hyliu/software/MIR/sma-mir/setup.bsh
# IDL> device, decomposed=0, retain=2
##########################################################################
##### Way to check helpfile ##############################################
# prints help comments from beginning of .pro files.
IDL> pro_hlp
pro_lkf
# looks for a routine with a keyword in its first comment line.
IDL> pro_lkf
# provides brief online description of several routines and wrappers related to the SMA data-reduction.
mirhelp
# For example:
#
# IDL> pro_hlp,variable=<'variable_name'>
# to get help on a variable.
#
# IDL> pro_hlp,structure=<'structure_name'>
# to get help on a structure.
#
# IDL> pro_hlp,/program
# to get help on a program and print out the basic
# introduction and usage of the function.
#
# IDL> mirhelp,'<task name>'
# to get help on a wrapper program, e.g. gain_cal etc.
#
##########################################################################
##### Read Data in MIR IDL ###############################################
# science data (PWV: 0.1)
# load raw data
# readdata, dir='SMA2022B_A007_all'
# load MIR IDL format SMA data
mir_restore, 'ntnulecture_demo.mir'
# Set the number of spectral channel of the continuum spectral window to be 1
select,/p,/re,band='c1'
sp[psf].nch = 1
# re-generate continuum data by performing spectral averaging over the first 4 spectral windows
select, /p, /reset, band=['c1','s1','s2','s3','s4']
uti_avgband, swmch1=240, swmch2=3856
plot_continuum
# examing the calibrator data
select,/p,/reset,source=['1924-292','Neptune','3c111']
plot_continuum
# save MIR format data
select, /p, /reset
mir_save, 'ntnulecture_all.mir', /new
# inspect spectra
select,/p,/reset, source='1924-292'
plot_spectra,x_var='fsky',y_vars='amp,pha', frame_vars='blcd', color_vars='band',frames_per_page=4
plot_spectra,x_var='channel',y_vars='amp,pha', frame_vars='blcd', color_vars='band',frames_per_page=4
select,/p,/reset
### Notes ################################################################
# passband: 1924-292
# source : zztauirs
# Flux : Neptune
# gain : 3c111
# (PWV: oscillating between 0.05 and 0.15)
#
# Issues:
# rx240, antenna5 have no signal
##########################################################################
##### Baseline Correction ################################################
# IDL> select,/p,/reset
# IDL> sma_cal_bas
## Notes #################################################################
#
# This step is not needed.
#
# # Example steps to correct antenna files
# Enter the current ANTENNAS file: ./120520_07:02:06/antennas
# Enter the new ANTENNAS file: ./ant_tab/antennas
#
##########################################################################
##### Inspect elevation ################################
result = plo_var('dhrs','el',frames_per_page=1)
##### Notes ############################################
#
# Elevation of Neptune in between 50 and 55 deg.
# There is a scan on Neptune at ~20 deg El that has poor SNR
#
########################################################
##### Initial Data Insepction ############################################
select, /p, /reset, source=['1924-292','3c111']
result=dat_filter(s_f,' "rec" eq "230" and "wt" gt "0" ')
plot_continuum
plot_continuum, preavg, frames_p=4, y='amp'
plot_continuum, frames_p=4, y='pha'
select, /p, /reset, source=['1924-292','2232+117','2348-165','3c111','0501-019']
result=dat_filter(s_f,' "rec" eq "240" and "wt" gt "0" ')
plot_continuum
select, /p, /reset
## Notes #################################################################
#
# 1. ipoint:
#
# 2. Reference antenna: 1
#
##########################################################################
##### Flag the ipoint Data and Strange data ##############################
# flag bad telescope
select,/p,/reset
result=dat_filter(s_f, '"blcd" like "5" ')
result=dat_filter(s_f,' "rec" eq "240" and "wt" gt "0" ')
flag,/flag
# flag lot amplitude integration on Neptune
select, /p, /reset
result=dat_filter(s_f, '"int" gt "170" and "int" lt "177" ')
flag,/flag
select, /p, /reset
plot_continuum
mir_save, 'ntnulecture_all.flag.mir'
## Notes #################################################################
# Data quality is good
##########################################################################
#### Tsys Correction ####################################################
# inspect Tsys
select,/p,/reset
result=dat_filter(s_f,' "rec" eq "230" and "wt" gt "0" ')
plot_var, frames_p=6
select,/p,/reset
result=dat_filter(s_f,' "rec" eq "240" and "wt" gt "0" ')
plot_var, frames_p=6
select,/p,/reset
apply_tsys
plot_continuum, x='int'
mir_save, 'ntnulecture_all.tsys.mir'
## Notes #################################################################
#
# 230 GHz: Tsys is 250-400 K in the target loop
# 240 GHz: Tsys is 200-350 K in the target loop
#
##########################################################################
#### Passband Calibration ################################################
# remove spikes from passband calibrator
select,/p,/reset
uti_checkspike,source='1924-292',/baseline,ntrim=100, threshold=5, /fix, sample=1
# remove spikes from passband calibrator
select,/p,/re,band='s3'
result=dat_filter(s_f,' "rec" eq "240" and "wt" gt "0" ')
result=dat_filter(s_f,' "blcd" like "1" and "blcd" like "8" ')
uti_chanfix,chan=417, sample=2
uti_chanfix,chan=418, sample=2
select, /p, /reset
result=dat_filter(s_f, ' "wt" gt "0" and "nch" eq "4096" ', /reset)
pass_cal, cal_type='pha', smoothing=64, ntrim=64, frames_p=16,refant=1
# all no
# 1924-292 yes
select, /p, /reset, band=['c1','s1','s2','s3','s4','s5','s6']
uti_avgband, swmch1=240, swmch2=3856
select, /p, /reset
result=dat_filter(s_f, ' "wt" gt "0" and "nch" eq "4096" ', /reset)
pass_cal, cal_type='amp', smoothing=64, ntrim=64, frames_p=16,refant=1
select, /p, /reset
mir_save, 'ntnulecture_all.tsys.bp.mir'
select,/p,/reset, band='c1', source=['1924-292','3c111','Neptune']
mir_save, 'flux.mir',/new
## Notes #################################################################
# 1. Data should be ok.
# http://sma1.sma.hawaii.edu/planetvis.html
# Neptune: 15.224 Jy @ 240 GHz
# http://sma1.sma.hawaii.edu/callist/callist.html
# 3c111
1mm 13 Aug 2022 15:51 SMA 225.93 1.406 +/- 0.073 mgurwell
1mm 03 Sep 2022 15:22 SMA 225.54 1.457 +/- 0.073 mgurwell
# 1924-292
1mm 13 May 2022 15:14 SMA 225.53 6.173 +/- 0.312 mgurwell
1mm 05 Sep 2022 05:11 SMA 229.55 5.719 +/- 0.286 mgurwell
##########################################################################
#### Measure the Absolute Flux ###########################################
select,/p,/reset, ant='-5'
gain_cal,cal_type='pha',x_var='hours',tel_bsl='telescope',refant=1,/connect,/non_point
# all yes
select,/p,/reset
result=dat_filter(s_f, '"int" gt "1670" and "int" lt "1715" ')
flag,/flag
select, /p, /reset, ant='-5'
gain_cal,cal_type='amp',x_var='hours',tel_bsl='telescope',refant=1,/non_point,poly=0,/preavg
# all no
# Neptune yes 1
select,/p,/re,rx=230,sideband='l'
result=dat_filter(s_f, ' "el" gt "50" and "el" lt "55"' )
flux_measure
Vector average:
# Source Flags Nscans Flux(Jy) SNR meantime REAL IMAG
3c111 113 1.3061 251 9.06 1.3060 -0.0115
select,/p,/re,rx=230,sideband='u'
result=dat_filter(s_f, ' "el" gt "50" and "el" lt "55"' )
flux_measure
Vector average:
# Source Flags Nscans Flux(Jy) SNR meantime REAL IMAG
3c111 20 1.1951 120 8.97 1.1951 0.0024
select,/p,/re,rx=240,sideband='l'
result=dat_filter(s_f, ' "el" gt "50" and "el" lt "55"' )
flux_measure
Vector average:
# Source Flags Nscans Flux(Jy) SNR meantime REAL IMAG
3c111 20 1.1741 140 8.97 1.1741 0.0014
select,/p,/re,rx=240,sideband='u'
result=dat_filter(s_f, ' "el" gt "50" and "el" lt "55"' )
flux_measure
Vector average:
# Source Flags Nscans Flux(Jy) SNR meantime REAL IMAG
3c111 20 1.1458 127 8.97 1.1458 0.0028
select,/p,/re
mir_save,/new,'flux.mir'
## Notes #################################################################
##########################################################################
#### Gain Calibration and Miriad Files Output ############################
# IDL> device, decomposed=0, retain=2
# IDL> .compile idl2miriad
mir_restore, 'ntnulecture_all.tsys.bp.mir'
# gain phase calibration
select,/p,/reset, rx=230
gain_cal,cal_type='pha',x_var='hours',tel_bsl='telescope',refant=1,/connect,/non_point,/preavg
3c111 yes 1
select,/p,/reset, rx=240, ant='-5'
gain_cal,cal_type='pha',x_var='hours',tel_bsl='telescope',refant=1,/connect,/non_point,/preavg
# gain amplitude calibration
select,/p,/reset, rx=230, sideband='l'
gain_cal,cal_type='amp',x_var='hours',tel_bsl='telescope',refant=1,/non_point,smooth=1.,/preavg
all no
3c111 yes 1.3061
select,/p,/reset, rx=230, sideband='u'
gain_cal,cal_type='amp',x_var='hours',tel_bsl='telescope',refant=1,/non_point,smooth=1.,/preavg
all no
3c111 yes 1.1951
select,/p,/reset, rx=240, sideband='l', ant='-5'
gain_cal,cal_type='amp',x_var='hours',tel_bsl='telescope',refant=1,/non_point,smooth=1.,/preavg
all no
3c111 yes 1.1741
select,/p,/reset, rx=240, sideband='u', ant='-5'
gain_cal,cal_type='amp',x_var='hours',tel_bsl='telescope',refant=1,/non_point,smooth=1.,/preavg
all no
3c111 yes 1.1458
select, /p, /reset, source=['1924-292','3c111','Neptune']
plot_continuum
# save calibrated data
select,/p,/reset
mir_save, 'ntnulecture_all.cal.mir', /new
# Export Miriad format data
rx=230
targ_list = ['zztauirs']
spw_list = ['s1', 's2','s3','s4','s5','s6']
sideband_list = ['l','u']
select,/p,/reset, rx=rx
foreach spw, spw_list do foreach sideband, sideband_list do foreach targ, targ_list do idl2miriad, dir=targ+'.rx'+string(rx)+'.'+sideband+'sb'+'.'+spw+'.cal.miriad', sideband=sideband, source=targ, band=[spw]
rx=240
targ_list = ['zztauirs']
spw_list = ['s1', 's2','s3','s4','s5','s6']
sideband_list = ['l','u']
select,/p,/reset, rx=rx
foreach spw, spw_list do foreach sideband, sideband_list do foreach targ, targ_list do idl2miriad, dir=targ+'.rx'+string(rx)+'.'+sideband+'sb'+'.'+spw+'.cal.miriad', sideband=sideband, source=targ, band=[spw]
## Notes #################################################################
#
#
##########################################################################
Sometimes I also check the ALMA Calibrator Grid Survey and see if we can get consistent absolute flux measurement:
# In CASA 5 command line
import sys
sys.path.append("./analysis_scripts")
import analysisUtils as au
# 1517-243
CASA <27>: au.getALMAFlux('J1517-243', '357GHz', lowband=3, highband=7, date='20220524')
Closest Band 3 measurement: 1.850 +- 0.030 (age=+5 days) 103.5 GHz (will extrapolate from this datum using spectral index)
Closest Band 3 measurement: 1.890 +- 0.030 (age=+5 days) 91.5 GHz (will extrapolate from this datum using spectral index)
Closest Band 7 measurement: 1.117 +- 0.100 (age=+1 days) 362.7 GHz
Closest Band 7 measurement: 1.100 +- 0.114 (age=+1 days) 362.7 GHz
getALMAFluxCSV(Cycle6): Fitting for spectral index with 1 measurement pair of age -4.0 days from 20220524, with age separation of 2 days
20220529 -- 20220531: freqs=[91.47, 350.5], fluxes=[1.86, 1.236]
Median Monte-Carlo result for 357.000000 = 1.231747 +- 0.149251 (scaled MAD = 0.147478)
Result using spectral index of -0.304237 for 357.000 GHz from 1.870 Jy at 97.485 GHz = 1.259895 +- 0.149251 Jy
# 3c279
CASA <28>: au.getALMAFlux('J1256-057', '357GHz', lowband=3, highband=7, date='20220524')
Closest Band 3 measurement: 16.970 +- 0.150 (age=+2 days) 103.5 GHz (will extrapolate from this datum using spectral index)
Closest Band 3 measurement: 16.667 +- 0.537 (age=+2 days) 92.9 GHz (will extrapolate from this datum using spectral index)
Closest Band 3 measurement: 17.820 +- 0.220 (age=+2 days) 91.5 GHz (will extrapolate from this datum using spectral index)
Closest Band 7 measurement: 8.759 +- 0.453 (age=+1 days) 341.5 GHz
getALMAFluxCSV(Cycle6): Fitting for spectral index with 1 measurement pair of age 1.5 days from 20220524, with age separation of 1 days
20220522 -- 20220523: freqs=[103.5, 92.9, 91.47, 341.5], fluxes=[16.97, 16.667, 17.82, 8.759]
Median Monte-Carlo result for 357.000000 = 8.771665 +- 1.036714 (scaled MAD = 1.012588)
Result using spectral index of -0.523435 for 357.000 GHz from 17.152 Jy at 95.957 GHz = 8.622924 +- 1.036714 Jy
# bllac
CASA <7>: au.getALMAFlux('J2202+422', '357GHz', lowband=3, highband=7, date='20220524')
Closest Band 3 measurement: 8.910 +- 0.450 (age=+3858 days) 109.8 GHz (will extrapolate from this datum using spectral index)
Closest Band 3 measurement: 8.960 +- 0.450 (age=+3858 days) 97.7 GHz (will extrapolate from this datum using spectral index)
Closest Band 7 measurement: 1.109 +- 0.055 (age=+1384 days) 343.5 GHz
getALMAFluxCSV(Cycle6): Fitting for spectral index with 1 measurement pair of age 3223.5 days from 20220524, with age separation of 1269 days
20111031 -- 20150422: freqs=[109.8, 97.7, 334.92], fluxes=[8.91, 8.96, 3.2]
Median Monte-Carlo result for 357.000000 = 3.040985 +- 0.402249 (scaled MAD = 0.393821)
Result using spectral index of -0.867940 for 357.000 GHz from 8.935 Jy at 103.750 GHz = 3.056952 +- 0.402249 Jy
WARNING: the mean time separation between the target date and the flux monitoring observations is 3858 days
4.4 Miriad imaging
If you have installed Miriad, you can source it with a Linux command (can also make an alias for this):
source /home/hyliu/softwares/Miriad/carma/miriad/miriad_start.sh
Note that you may need two packages in order to run Miriad tasks, libquadmath and libgfortran.so.3. The former can be installed by typing in the Linux command line (if using CentOS linux)
yum install libquadmath
You can try to find a pre-compiled libgfortran.so.3 and copy it into your /lib64 directory. I found one in the /lib directory in the CASA software package. If you still cannot find one, please contact me. I can send you a copy.
For an online documentation for the individual Miriad tasks, please check the Miriad task list.
4.4.1 Flagging edge channels and continuum subtraction
We are using the Miriad software package here, either under linux command line or using a BASH script.
Sometimes the edge channels of a spectral window can have poor response. You can inspect the spectral data using the Miriad command like:
# uvspec vis=IP_Tau_track5.rx400.lsb.cal.miriad axes=freq,amp nxy=1,1 device=/xw options=nobase,avall interval=999999 select='window(1,2,3)'
You can flag them using a BASH script like:
#!/bin/bash
config="com"
flagval="f"
targets='zztauirs'
sidebands='usb lsb'
ifbands='rx230 rx240'
spws='s1 s2 s3 s4 s5 s6'
for field in $targets
do
for sideband in $sidebands
do
for ifband in $ifbands
do
for spw in $spws
do
uvflag vis=$field'.'$ifband'.'$sideband'.'$spw'.cal.miriad' \
edge=256,256,0 flagval=$flagval
done
done
done
done
After inspecting spectral lines in the data, you can perform continuum baseline fitting and subtraction using a BASH script like:
#!/bin/bash
config="com"
flagval="f"
targets='zztauirs'
sidebands='usb lsb'
ifbands='rx230 rx240'
spws='s1 s2 s3 s4 s5 s6'
ch0quicklook='yes'
linequicklook='no'
for field in $targets
do
for sideband in $sidebands
do
for ifband in $ifbands
do
for spw in $spws
do
vishead=$field'.'$ifband'.'$sideband'.'$spw
echo $vishead
# obtain continuum channel by directly averaging over spectral domain
# syntax of the line parameter: nchan, start, width, step
# rm -rf $vishead'.ch0.miriad'
# uvaver vis=$vishead'.cal.miriad' \
# out=$vishead'.ch0.miriad' \
# line='channel,1,1,4096,1' options='nocal,nopass,nopol'
# obtain continuum channel by performing polynomial fitting
rm -rf $vishead'.ch0.miriad'
uvlin vis=$vishead'.cal.miriad' \
out=$vishead'.ch0.miriad' \
chans='64,4032' \
mode=chan0 options=nocal,nopass,nopol,nowin order=1
# obtain continuum channel by performing polynomial fitting
rm -rf $vishead'.line.miriad'
uvlin vis=$vishead'.cal.miriad' \
out=$vishead'.line.miriad' \
chans='64,4032' \
mode=line options=nocal,nopass,nopol,nowin order=1
# generating quicklook continuum dirty images
if [ $ch0quicklook = "yes" ]
then
rm -rf $vishead'.ch0.map'
rm -rf $vishead'.ch0.beam'
invert vis=$vishead'.ch0.miriad' \
map=$vishead'.ch0.map' \
beam=$vishead'.ch0.beam' \
options=systemp,double,mfs robust=2.0 \
imsize=128 cell=0.3
# producing quicklook continuum FITS image
rm -rf map=$vishead'.ch0.fits'
fits in=$vishead'.ch0.map' op=xyout \
out=$vishead'.ch0.fits'
fi
# generating quicklook line dirty images
if [ $linequicklook = "yes" ]
then
rm -rf $vishead'.line.map'
rm -rf $vishead'.line.beam'
invert vis=$vishead'.line.miriad' \
map=$vishead'.line.map' \
beam=$vishead'.line.beam' \
line='channel,100,257,1,1' \
options=systemp,double robust=2.0 \
imsize=128 cell=0.3
# producing quicklook line FITS image
rm -rf map=$vishead'.line.fits'
fits in=$vishead'.line.map' op=xyout \
out=$vishead'.line.fits'
fi
done
done
done
done
4.4.2 Single pointing, continuum imaging
You can perform continuum imaging using a BASH script like the following one:
#!/bin/bash
# Miriad task list
# https://www.atnf.csiro.au/computing/software/miriad/taskindex.html
field="BP_Tau"
# creat dirty image (i.e., inverse Fourier transform)
rm -rf $field'.dirty'
rm -rf $field'.beam'
invert vis=$field'_track5.rx400.usb.cal.miriad' options=systemp,mfs,double robust=2.0 map=$field'.dirty' beam=$field'.beam' cell=0.25 imsize=256
rm -rf $field'.dirty.fits'
rm -rf $field'.beam.fits'
fits in=$field'.dirty' op=xyout out=$field'.dirty.fits'
fits in=$field'.beam' op=xyout out=$field'.beam.fits'
# deconvolve image
rm -rf $field'.model'
rm -rf $field'.model.fits'
clean map=$field'.dirty' beam=$field'.beam' out=$field'.model' cutoff=0.015 niters=100
fits in=$field'.model' op=xyout out=$field'.model.fits'
# restore image (i.e., convolve the clean components with a clean Gaussian beam)
rm -rf $field'.clean'
rm -rf $field'.clean.fits'
restor map=$field'.dirty' beam=$field'.beam' model=$field'.model' mode=clean out=$field'.clean'
fits in=$field'.clean' op=xyout out=$field'.clean.fits'
# generate residual image
rm -rf $field'.residual'
rm -rf $field'.residual.fits'
restor map=$field'.dirty' beam=$field'.beam' model=$field'.model' mode=residual out=$field'.residual'
fits in=$field'.residual' op=xyout out=$field'.residual.fits'
# Perform primary beam correction
rm -rf $field'.clean.pbcor'
rm -rf $field'.clean.pbcor.fits'
linmos in=$field'.clean' out=$field'.clean.pbcor'
fits in=$field'.clean.pbcor' op=xyout out=$field'.clean.pbcor.fits'
4.4.3 Single pointing, spectral lines imaging
You can either image the entire continuum-subtracted line visibility files, or using the following script to extract the line you want to image. This procedure includes (1) inputing the rest-frequency information of the spectral line into the header information, (2) performing off-line doppler correction, and (3) re-grid the velocity channels:
#!/bin/bash
config="com"
flagval="f"
targets='zztauirs'
sidebands='usb'
ifbands='rx230'
spws='s2'
linename='co'
restfreq='230.538'
lineparms='velocity,50,-30.0.0,1.2,1.2'
for field in $targets
do
for sideband in $sidebands
do
for ifband in $ifbands
do
for spw in $spws
do
vishead=$field'.'$ifband'.'$sideband'.'$spw
invis=$vishead'.line.miriad'
outvis=$field'.'$linename'.miriad'
rm -rf $outvis
# include the information of spectraline
rm -rf $vishead'.miriad.temp'
uvputhd vis=$invis type=d \
hdvar=restfreq varval=$restfreq \
out=$vishead'.miriad.temp'
# offline doppler correction
rm -rf $vishead'.miriad.temp2'
uvredo vis=$vishead'.miriad.temp' \
options='velocity,nopass,nocal,nopol' \
out=$vishead'.miriad.temp2' velocity=lsr
# regridding velocity channel
uvaver vis=$vishead'.miriad.temp2' line=$lineparms= \
out=$outvis
rm -rf *.temp
rm -rf *.temp2
done
done
done
done
The commands to image the spectral line data (i.e., invert, clean, restor, and linmos) are similar to those for continuum imaging, just we should omit mfs in the options keyword of invert.
4.4.4 Mosaic imaging
4.4.5 Combining single dish and interferometric observations
For a Miriad based procedure, please check my documentation under the ALMICA repository.
4.5 Miriad self-calibration
If you have imaged your SMA data using the Miriad software package, and if the residual phase errors appear large, you may try a few iterations of gain phase self-calibration using the Miriad software package. Embedded is an example using an image model to self-calibrate multiple epochs of observations on the same target source. Note that the Miriad task selcal can only deal with Stokes I, Q, U, and V. If your data were stored as XX, YY, RR, LL correlations, then you need to use the uvaver task to cover the data to Stokes I.
#!/bin/csh
targets='AB_Aur'
tracks='track6'
rxs='rx400'
sidebands='usb lsb'
refant='3'
# creating a directory to host self-calibrated data
rm -rf selcal_Miriad
mkdir selcal_Miriad
# looping over observations on varios target sources
for target in $targets
do
for track in $tracks
do
datadir='../../calibrated_Miriad/'$track'/'
imdir='../../imaging/ch0/'$track'/'$target'/'
for rx in $rxs
do
# set reference antenna and solution interval
if [ $track = 'track6' ]
then
interval='0.1'
refant='3'
fi
for sideband in $sidebands
do
# copy over visibility data
vis=$target'_'$track'.'$rx'.'$sideband'.cal.miriad'
cp -r $datadir$vis ./
# copy over image model
model=$target'.rx345.usb.model'
cp -r $imdir$model ./
# convert to Stokes I data
rm -rf $vis'.i'
uvaver vis=$vis options=nopass,nocal,nopol out=$vis'.i' stokes=ii
# produce ascii output for the self-calibration solution
gaintable=$target'_'$track'.'$rx'.'$sideband'.1p.gain'
rm -rf $gaintable
int='0.1'
selfcal vis=$vis'.i' model=$model \
out=$gaintable \
options='pha,mfs' \
interval=$interval refant=$refant
# perform gain self-claibration solution
selfcal vis=$vis'.i' model=$model \
options='pha,mfs' \
interval=$interval refant=$refant
# inspecting the solution and yield ascii output for solution table
rm -rf $gaintable'.txt'
gpplt vis=$gaintable yaxis=phase nxy=1,1 log=$gaintable'.txt' # device=/xw
# apply calibration solution
rm -rf $vis'.sel'
uvaver vis=$vis'.i' options=nopass,nopol out=$vis'.sel'
# creating non-self-calibrated dirty image
rm -rf $vis'.map'
rm -rf $vis'.beam'
invert vis=$vis map=$vis'.map' beam=$vis'.beam' \
imsize=512,512 cell=0.1,0.1 robust=2.0 options=systemp,double,mfs
# creating self-calibrated dirty image
rm -rf $vis'.sel.map'
rm -rf $vis'.sel.beam'
invert vis=$vis'.sel' map=$vis'.sel.map' beam=$vis'.sel.beam' \
imsize=512,512 cell=0.1,0.1 robust=2.0 options=systemp,double,mfs
# collecting self-calibrated visibilities into a folder
mv *.sel ./selcal_Miriad
# collecting solution tables into the folder
mv *.gain ./selcal_Miriad
mv *.gain.txt ./selcal_Miriad
done
done
done
done
In this case, the gain-phase self-calibration solutions for the upper and lower sidebands were derived independently using the selfcal task. The target-source model in the self-calibration procedure was a clean-model constructed from the observations taken in good weather conditions, ZCMa.model.temp. In principle, you can use observations taken at slightly different frequencies, or even the observations taken with ALMA, to initialize your self-calibration procedure. If there is no such a priori high-quality target-source model available for your self-calibration, you can iterate between imaging and self-calibration a few times, which may be done with a script. You can use the Miriad task cgcurs to set clean boxes interactively if it is needed.
The solution interval was set to be 5 minutes; you have to adjust your solution intervals case-by-case to ensure that the solutions were derived at high signal-to-noise ratios.
The solution tables were stored in your visibility files. You can use the Miriad task gplist to check what tables have been stored in a visibility file. You can use the Miriad task gpplt to inspect the solution tables, to ensure that the solutions are not jumping around randomly (in most cases, this is a sign of poor solution quality). You can use the Miriad task gpedit to edit the self-calibration solution tables. If you know what you are doing, this can sometimes improve the self-calibration (e.g., lowing the noise level and enhancing the peak intensity). But you should only do this edit when you really know what you are doing. Finally, the script used the uvaver task to apply the solution table.
4.6 CASA Data calibration and imaging (under construction)
4.7 MIR Data calibration (polarization)
Here I provide the example MIR IDL + Miriad scripts of calibrating the dual RX full polarization data taken after 2017 August and single RX polarization data taken before 2016. The polarization data taken in between may be subject to some hardware or software issues. If you really need to use them, it is recommended to contact Dr. Ramprasad Rao.
4.7.1 Dual RX Data taken after 2017 August (under construction)
4.7.2 Single RX Data taken before 2016 (under construction)
5. Publishing
You are required to include the following statement in the Acknowledgement:
* The Submillimeter Array is a joint project between the Smithsonian Astrophysical Observatory and the Academia Sinica Institute of Astronomy and Astrophysics, and is funded by the Smithsonian Institution and the Academia Sinica (Ho et al. 2004). *
You are required to cite Ho et al. (2004).
Your are required to cite Sault et al. (1995), Qi et al. (2003), or McMullin et al. (2007) if you use Miriad, MIR IDL, or CASA to process your data.
When you are carrying out a precision experiment instead of a detection experiment you may want/need to refer to the SMA Calibrator List. If you are quoting the records explicitly in your paper, you need to make an E-mail contact with Dr. Mark Gurwell and obtain his agreement. You need to inquire how he would like to be credited (e.g., as a co-author or being mentioned in the acknowledgement).
6. When you get stuck
If you are located in Taiwan, you can either ask me or my present Master’s student, Chia-Ying Chung.
Note that she is not obligated to work for you or answer your questions regardless of your career stage. If you are making a very big request to her, for example, if you need her to teach you or walk you through data calibrations, or even calibrate data for you, then you should regard this as a formal collaboration and should include her as a co-I when you are making a journal publication. Please appreciate that it takes a very significant effort for her to pick up this skill. She still has the right to say no to you (so be nice to her). I also have the right to say no to you for her, to ensure that she can be sufficiently focused on her thesis project.
7. Polarization observing script
Here is a template. You are likely experienced enough to understand the syntax. You should do the necessary modification. Please be sure to find all XXX using the “find” function of your browser and replace them with appropriate values.
You need to use the manage script directly method to edit in the polarization script. Please do simulate the script online.
#!/usr/bin/perl -w
{ BEGIN {$^W =0}
#
################## Script Header Info #####################
#
# Experiment Code: 2022A-AXXX
# Experiment XXX
# PI: XXX
# Contact Person: XXX
# Email : XXX
# Cell : XXX
# Array : XXX
#
############## SPECIAL INSTRUCTIONS ################
#
# Please observe the polarization calibrator 3CXXX for XXX minutes
# before (or after) the target source loop.
#
# It would be best if we can observe the polarization calibrator
# for at least 40 minutes during its transit (or before and after
# the transit if the calibrator transits at too high elevation).
#
################## Priming ################################
# observe -s MyPetSource -r XX:XX:XX.XX -d -XX:XX:XX.X -e 2000 -v XX
# dopplerTrack -r XXX.XX -u -s1 -f 0.0000 -h 10 -R h -r XXX.XX -u -s1 -f 0.0000 -h 10
# POL_CAL: 3c279 or bllac (before target source loop)
# or 3c84 (after target source loop; preferable)
#
################## Pointing ###############################
#
# Pointing: At start of track and after transit
# Syntax Example: point -i 60 -r 3 -L -l -t -Q
#
################## Source, Calibrator and Limits ##########
#
# set scan time
$inttime="15";
#
# choose which antennas with waveplates crossed during the cross RX calibration
# !!! Antennas chosen MUST BE IN THE ARRAY !!!
$pants='4,6';
# POL_CAL
$cal0="3c84"; $ncal0="12";
$calpol0="3C84_LR -r 03:19:48.1601 -d 41:30:42.103 -e 2000 -v 0"; $ncalpol0="12";
$cal1="3c279"; $ncal1="12";
$calpol1="3C279_LR";$ncalpol1="12";
$cal2="bllac"; $ncal2="12";
$calpol2="bllac_LR";$ncalpol2="12";
# Gain cal
$cal3="XXXX-XXX"; $ncal3="16";
$calpol3="XXXX-XXX_LR -r XX:XX:XX.XXXX -d -XX:XX:XX.XXX -e 2000 -v 0"; $ncalpol3="15";
$targ1="MyPetSource -r XX:XX:XX.XX -d -XX:XX:XX.X -e 2000 -v XXX"; $ntarg1="24";
# FluxCal
$flux0="Callisto"; $nflux0="20";
XXX Change all MINEL to 33 degree for the subcompact track
$MINEL_TARG = 20; $MAXEL_TARG = 85;
$MINEL_GAIN = 20; $MAXEL_GAIN = 85;
$MINEL_FLUX = 20; $MAXEL_FLUX = 81;
$MINEL_BPASS= 20; $MAXEL_BPASS= 87;
$MINEL_CHECK= 20;
#
################## Script Initialization ##################
#
do 'sma.pl';
do 'sma_add.pl';
checkANT();
command("radio");
command("integrate -t $inttime");
$myPID=$$;
command("project -r -p 'SMA' -d '2022A-AXXX'");
print "----- initialization done, starting script -----\n";
#
################## Science Script #########################
#
print "####################################################\n";
print "########### Polcal loop 3C279 ##\n";
print "####################################################\n";
$LST_start=XX.X; $LST_end=XX.X;
&DoPolPass(cal1,ncal1,calpol1,ncalpol1);
#
print "#############################################\n";
print "## Observing target source ##################\n";
print "## Gaincal: 1743-038 ########################\n";
print "## Cross hand phase: 1743-038_LR ########\n";
print "## End the loop with 1743-038 ###############\n";
print "#############################################\n";
$LST_start=XX.X; $LST_end=XX.X;
&DualPolLoop(calpol3,ncalpol3,cal3,ncal3,targ1,ntarg1);
#
print "####################################################\n";
print "######### flux Calibration loop on fluxcal ##########\n";
print "####################################################\n";
$LST_start=XX.X; $LST_end=XX.X;
$max_loops=3;
&DoPolFlux(flux0,nflux0);
#
print "####################################################\n";
print "########### Polcal loop 3C84 ##\n";
print "####################################################\n";
$maxloops="200";
$LST_start=XX.X; $LST_end=XX.X;
&DoPolPass(cal0,ncal0,calpol0,ncalpol0);
#
print "----- Congratulations! This is the end of the script. -----\n";}
#
################## File End ###############################