process.xmm

process.xmm.assemble

daxa.process.xmm.assemble.epchain(obs_archive, process_unscheduled=True, num_cores=1, disable_progress=False, timeout=None)

This function runs the epchain SAS process on XMM missions in the passed archive, which assembles the PN-specific ODFs into combined photon event lists - rather than the per CCD files that existed before. A run of epchain for out of time (OOT) events is also performed as part of this function call. The epchain manual can be found here (https://xmm-tools.cosmos.esa.int/external/sas/current/doc/epchain.pdf) and gives detailed explanations of the process.

Per the advice of the SAS epchain manual, the OOT event list epchain call is performed first, and its intermediate files are saved and then used for the normal call to epchain.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with PN observations for which epchain should be run. This function will fail if no XMM missions are present in the archive.

• process_unscheduled (bool) – Whether this function should also process sub-exposures marked ‘U’, for unscheduled. Default is True, in which case they will be processed.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire epchain process, but a timeout for individual ObsID-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.assemble.emchain(obs_archive, process_unscheduled=True, num_cores=1, disable_progress=False, timeout=None)

This function runs the emchain SAS process on XMM missions in the passed archive, which assembles the MOS-specific ODFs into combined photon event lists - rather than the per CCD files that existed before. The emchain manual can be found here (https://xmm-tools.cosmos.esa.int/external/sas/current/doc/emchain.pdf) and gives detailed explanations of the process.

The DAXA wrapper does not allow emchain to automatically loop through all the sub-exposures for a given ObsID-MOSX combo, but rather creates a separate process call for each of them. This allows for greater parallelisation (if on a system with a significant core count), but also allows the same level of granularity in the logging of processing of different sub-exposures as in DAXA’s epchain implementation.

The particular CCDs to be processed are not specified in emchain, unlike in epchain, because it can sometimes have unintended consequences. For instance processing a MOS observation in FastUncompressed mode, with timing on CCD 1 and imaging everywhere else, can cause emchain to fail (even though no actual failure occurs) because the submode is set to Unknown, rather than FastUncompressed.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with MOS observations for which emchain should be run. This function will fail if no XMM missions are present in the archive.

• process_unscheduled (bool) – Whether this function should also process sub-exposures marked ‘U’, for unscheduled. Default is True, in which case they will be processed.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire emchain process, but a timeout for individual ObsID-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.assemble.rgs_events(obs_archive, process_unscheduled=True, num_cores=1, disable_progress=False, timeout=None)

This function runs the first step of the SAS RGS processing pipeline, rgsproc. This should prepare the RGS event lists by calibrating and combining the separate CCD event lists into RGS events. This happens separately for RGS1 and RGS2, and for each sub-exposure of the two instruments.

None of the calculations performed in this step should be affected by the choice of source, the first step where the choice of primary source should be taken into consideration is the next step, rgs_angles; though as DAXA processes data to be generally useful we will not define a primary source, that is for the user in the future as the aspect drift calculations can be re-run.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with RGS observations for which RGS processing should be run. This function will fail if no XMM missions are present in the archive.

• process_unscheduled (bool) – Whether this function should also process sub-exposures marked ‘U’, for unscheduled. Default is True, in which case they will be processed.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire rgs_events process, but a timeout for individual ObsID-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.assemble.rgs_angles(obs_archive, num_cores=1, disable_progress=False, timeout=None)

This function runs the second step of the SAS RGS processing pipeline, rgsproc. This should calculate aspect drift corrections for some ‘uninformative’ source, and should likely be refined later when these data are used to analyse a specific source. This happens separately for RGS1 and RGS2, and for each sub-exposure of the two instruments.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with RGS observations for which RGS processing should be run. This function will fail if no XMM missions are present in the archive.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire rgs_events process, but a timeout for individual ObsID-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.assemble.cleaned_rgs_event_lists(obs_archive, num_cores=1, disable_progress=False, timeout=None)

This function runs the third step of the SAS RGS processing pipeline, rgsproc. Here we filter the events to only those which should be useful for scientific analysis. The attitude and house-keeping GTIs are also applied. This happens separately for RGS1 and RGS2, and for each sub-exposure of the two instruments.

Unfortunately it seems that combining sub-exposure event lists for a given ObsID-Instrument combo is not supported/recommended, combinations of data are generally done after spectrum generation, and even then they don’t exactly recommend it - of course spectrum generation doesn’t get done in DAXA. As such this function will produce individual event lists for RGS sub-exposures.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with RGS observations for which RGS processing should be run. This function will fail if no XMM missions are present in the archive.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire rgs_events process, but a timeout for individual ObsID-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.assemble.cleaned_evt_lists(obs_archive, lo_en=None, hi_en=None, pn_filt_expr=('#XMMEA_EP', '(PATTERN <= 4)', '(FLAG .eq. 0)'), mos_filt_expr=('#XMMEA_EM', '(PATTERN <= 12)', '(FLAG .eq. 0)'), filt_mos_anom_state=False, acc_mos_anom_states=('G', 'I', 'U'), num_cores=1, disable_progress=False, timeout=None)

This function is used to apply the soft-proton filtering (along with any other filtering you may desire, including the setting of energy limits) to XMM-Newton event lists, resulting in the creation of sets of cleaned event lists which are ready to be analysed (or merged together, if there are multiple exposures for a particular observation-instrument combination).

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances for which cleaned event lists should be created. This function will fail if no XMM missions are present in the archive.

• lo_en (Quantity) – The lower bound of an energy filter to be applied to the cleaned, filtered, event lists. If ‘lo_en’ is set to an Astropy Quantity, then ‘hi_en’ must be as well. Default is None, in which case no energy filter is applied.

• hi_en (Quantity) – The upper bound of an energy filter to be applied to the cleaned, filtered, event lists. If ‘hi_en’ is set to an Astropy Quantity, then ‘lo_en’ must be as well. Default is None, in which case no energy filter is applied.

• pn_filt_expr (str/List[str]/Tuple[str]) – The filter expressions to be applied to EPIC-PN event lists. Either a single string expression can be passed, or a list/tuple of separate expressions, which will be combined using ‘&&’ logic before being used as the expression for evselect. Other expression components can be added during the process of the function, such as GTI filtering and energy filtering.

• mos_filt_expr (str/List[str]/Tuple[str]) – The filter expressions to be applied to EPIC-MOS event lists. Either a single string expression can be passed, or a list/tuple of separate expressions, which will be combined using ‘&&’ logic before being used as the expression for evselect. Other expression components can be added during the process of the function, such as GTI filtering, energy filtering, and anomalous state CCD filtering.

• filt_mos_anom_state (bool) – Whether this function should use the results of an ‘emanom’ run to identify and remove MOS CCDs that are in anomolous states. If ‘False’ is passed then no such filtering will be applied.

• acc_mos_anom_states (List[str]/str) – A list/tuple of acceptable MOS CCD status codes found by emanom (status- G is good at all energies, I is intermediate for E<1 keV, B is bad for E<1 keV, O is off, chip not in use, U is undetermined (low band counts <= 0)).

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire cleaned_evt_lists process, but a timeout for individual ObsID-Inst-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.assemble.merge_subexposures(obs_archive, num_cores=1, disable_progress=False, timeout=None)

A function to identify cases where an instrument for a particular XMM observation has multiple sub-exposures, for which the event lists can be merged. This produces a final event list, which is a combination of the sub-exposures.

For those observation-instrument combinations with only a single exposure, this function will rename the cleaned event list so that the naming convention is comparable to the merged event list naming convention (i.e. sub-exposure identifier will be removed).

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances for which cleaned event lists should be created. This function will fail if no XMM missions are present in the archive.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire merge_subexposures process, but a timeout for individual ObsID-Inst processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

process.xmm.check

daxa.process.xmm.check.emanom(obs_archive, num_cores=1, disable_progress=False, timeout=None)

This function runs the SAS emanom function, which attempts to identify when MOS CCDs are have operated in an ‘anomalous’ state, where the background at E < 1 keV is strongly enhanced. Data above 2 keV are unaffected, so CCDs in anomalous states used for science where the soft X-rays are unnecessary do not need to be excluded.

The emanom task calculates the (2.5-5.0 keV)/(0.4-0.8 keV) hardness ratio from the corner data to determine whether a chip is in an anomalous state. However, it should be noted that the “anonymous” anomalous state of MOS1 CCD#4 is not always detectable from the unexposed corner data.

This functionality is only usable if you have SAS v19.0.0 or higher - a version check will be performed and a warning raised (though no error will be raised) if you use this function with an earlier SAS version.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with MOS observations for which emchain should be run. This function will fail if no XMM missions are present in the archive.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire emanom process, but a timeout for individual ObsID-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.check.parse_emanom_out(log_file_path, acceptable_states=('G', 'I', 'U'))

A simple function to parse the output file created by the emanom SAS task, which seeks to identify which MOS CCDs are in ‘anomalous states’. The most relevant pieces of information contained in the output file are the CCD IDs, and the state code; the ‘state’ values have the following meanings:

Status- G is good at all energies

• I is intermediate for E<1 keV

• B is bad for E<1 keV

• O is off, chip not in use

• U is undetermined (low band counts <= 0)

Users can specify which states they would like to keep using the ‘acceptable_states’ parameter.

Parameters:

• log_file_path (str) – The path to the output file created by emanom.

• acceptable_states (List[str]/str) – The CCD states which should be accepted. If a CCD is accepted then its ID will be returned by this function.

Returns:

A list of CCD IDs which are in ‘acceptable’ states.

Return type:

List[int]

process.xmm.clean

daxa.process.xmm.clean.espfilt(obs_archive, method='histogram', with_smoothing=True, with_binning=True, ratio=1.2, filter_lo_en=<Quantity 2500. eV>, filter_hi_en=<Quantity 8500. eV>, range_scale=None, allowed_sigma=3.0, gauss_fit_lims=(0.1, 6.5), num_cores=1, disable_progress=False, timeout=None)

The DAXA wrapper for the XMM SAS task espfilt, which attempts to identify good time intervals with minimal soft-proton flaring for individual sub-exposures (if multiple have been taken) of XMM ObsID-Instrument combinations. Both EPIC-PN and EPIC-MOS observations will be processed by this function.

This function does not generate final event lists, but instead is used to create good-time-interval files which are then applied to the creation of final event lists, along with other user-specified filters, in the ‘cleaned_evt_lists’ function.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances with PN/MOS observations for which espfilt should be run. This function will fail if no XMM missions are present in the archive.

• method (str) – The method that espfilt should use to find soft proton flaring. Either ‘ratio’ or ‘histogram’ can be selected. The default is ‘histogram’.

• with_smoothing (bool/Quantity) – Should smoothing be applied to the light curve data. If set to True (the default) a smoothing factor of 51 seconds is used, if set to False smoothing will be turned off, if an astropy Quantity is passed (with units convertible to seconds) then that value will be used for the smoothing factor.

• with_binning (bool/Quantity) – Should binning be applied to the light curve data. If set to True (the default) a bin size of 60 seconds is used, if set to False binning will be turned off, if an astropy Quantity is passed (with units convertible to seconds) then that value will be used for the bin size.

• ratio (float) – Flaring ratio of annulus counts.

• filter_lo_en (Quantity) – The lower energy bound for the event lists used for soft proton flaring identification.

• filter_hi_en (Quantity) – The upper energy bound for the event lists used for soft proton flaring identification.

• range_scale (dict) – Histogram fit range scale factor. The default is a dictionary with an entry for ‘pn’ (15.0) and an entry for ‘mos’ (6.0).

• allowed_sigma (float) – Limit in sigma for unflared rates.

• gauss_fit_lims (Tuple[float, float]) – The parameter limits for gaussian fits.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire espfilt process, but a timeout for individual ObsID-Inst-subexposure processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

process.xmm.generate

daxa.process.xmm.generate.generate_images_expmaps(obs_archive, lo_en=<Quantity [0.5, 2. ] keV>, hi_en=<Quantity [ 2., 10.] keV>, num_cores=1)

A function to generate images and exposure maps for a processed XMM mission dataset contained within an archive. Users can select the energy band(s) that they wish to generate images and exposure maps within.

Parameters:

• obs_archive (Archive)

• lo_en – The lower energy bound(s) for the product being generated. This can either be passed as a scalar Astropy Quantity or, if sets of the same product in different energy bands are to be generated, as a non-scalar Astropy Quantity. If multiple lower bounds are passed, they must each have an entry in the hi_en argument. The default is ‘Quantity([0.5, 2.0], ‘keV’)’, which will generate two sets of products, one with lower bound 0.5 keV, the other with lower bound 2 keV.

• hi_en – The upper energy bound(s) for the product being generated. This can either be passed as a scalar Astropy Quantity or, if sets of the same product in different energy bands are to be generated, as a non-scalar Astropy Quantity. If multiple upper bounds are passed, they must each have an entry in the lo_en argument. The default is ‘Quantity([2.0, 10.0], ‘keV’)’, which will generate two sets of products, one with upper bound 2.0 keV, the other with upper bound 10 keV.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

process.xmm.setup

daxa.process.xmm.setup.cif_build(obs_archive, num_cores=1, disable_progress=False, analysis_date='now', timeout=None)

A DAXA Python interface for the SAS cifbuild command, used to generate calibration files for XMM observations prior to processing. The observation date is supplied by the XMM mission instance(s), and is the date when the observation was started (as acquired from the XSA).

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances for which observation calibration files should be generated. This function will fail if no XMM missions are present in the archive.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• analysis_date (str/datetime) – The analysis date for which to generate calibration file. The default is ‘now’, but this parameter can be used to create calibration files as they would have been on a past date.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire cif_build process, but a timeout for individual ObsID processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.setup.odf_ingest(obs_archive, num_cores=1, disable_progress=False, timeout=None)

This function runs the SAS odfingest task, which creates a summary of the raw data available in the ODF directory, and is used by many SAS processing tasks.

Parameters:

• obs_archive (Archive) – An Archive instance containing XMM mission instances for which observation summary files should be generated. This function will fail if no XMM missions are present in the archive.

• num_cores (int) – The number of cores to use, default is set to 90% of available.

• disable_progress (bool) – Setting this to true will turn off the SAS generation progress bar.

• timeout (Quantity) – The amount of time each individual process is allowed to run for, the default is None. Please note that this is not a timeout for the entire odf_ingest process, but a timeout for individual ObsID processes.

Returns:

Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and f) whether the progress bar should be hidden or not.

Return type:

Tuple[dict, dict, dict, str, int, bool, Quantity]

daxa.process.xmm.setup.parse_odf_sum(sum_path, obs_id=None)

A function that takes a path to an XMM SAS summary file generated by the odf ingest command. The file will be filtered and parsed so that data relevant to DAXA processing valid scientific observations can be extracted. This includes things like whether a particular instrument was active, the number of sub-exposures, whether those sub-exposures were in a science mode that produces data useful for the study of astrophysical objects (i.e. not in a calibration or diagnosis mode). Data relevant to SAS procedures that calibrate and construct exposure lists is not included in the output of this function.

Parameters:

• sum_path (str) – The path to the odf_ingest-generated summary file that is to be parsed into a dictionary of relevant information.

• obs_id (str) – Optionally, the observation ID that goes with this summary file can be passed, purely to make a possible error message more useful.

Returns:

Multi-level dictionary of information, with top-level keys being instrument names. Next level contains information on whether the instrument was active and the number of exposures. This level has an ‘exposures’ key which is for a dictionary where the keys are all the exp_ids available for this instrument. Those keys are for dictionaries of exposure specific information, including mode, whether the exposure was scheduled, and the modes of the individual CCDs.

Return type:

dict