Processing raw XMM-Newton Pointed data
This tutorial will teach you how to use DAXA to process raw XMM-Newton data into a science ready state using one line of Python code (or several lines, if you wish to have more control over the settings for each step). This relies on there being an initialised (either manually before launching Python, or in your bash profile/rc) backend installation of the XMM Science Analysis System (SAS), including accessible calibration files - DAXA will check for such an installation, and will not allow processing to start without it.
All DAXA processing steps will parallelise as much as possible - processes running on different ObsIDs/instruments/sub-exposures will be run simultaneously (if cores are available)
Import Statements
[29]:
from daxa.mission import XMMPointed
from daxa.archive import Archive
from daxa.process.simple import full_process_xmm
from daxa.process.xmm.setup import cif_build, odf_ingest
from daxa.process.xmm.assemble import emchain, epchain, cleaned_evt_lists, merge_subexposures, \
rgs_events, rgs_angles, cleaned_rgs_event_lists
from daxa.process.xmm.check import emanom
from daxa.process.xmm.clean import espfilt
from daxa.process.xmm.generate import generate_images_expmaps
An Archive to be processed
Every processing function implemented in DAXA takes an Archive instance as its first argument; if you don’t already know what that is then you should go back and read the following tutorials:
Creating a DAXA archive - This explains how to create an archive, load an existing archive, and the various properties and features of DAXA archives.
Using DAXA missions - Here we explain what DAXA mission classes are and how to use them to select only the data you need.
Here we create an archive of XMM observations of the galaxy cluster Abell 907:
[5]:
xm = XMMPointed()
xm.filter_on_name('A907')
arch = Archive('Abell907', xm)
/Users/dt237/code/DAXA/daxa/mission/xmm.py:83: UserWarning: 140 of the 17689 observations located for this mission have been removed due to NaN RA or Dec values
self._fetch_obs_info()
Downloading XMM-Newton Pointed data: 100%|██████████████████████████████████████| 3/3 [00:36<00:00, 12.05s/it]
One-line solution
Though we provide individual functions that wrap the various steps required to reduce and prepare XMM data, and they can be used separately for greater control over the configuration parameters, we also include a one-line solution which executes the processing steps with default configuration.
We believe that the default parameters are adequate for most use cases, and this allows for users unfamiliar with the intricacies of XMM data to easily start working with it. Executing the following will automatically generate cleaned event lists for MOS1, MOS2, PN, RGS1, and RGS2 (if they were selected during mission creation), as well as images and exposure maps for MOS1, MOS2, and PN:
[6]:
full_process_xmm(arch)
XMM-Newton Pointed - Generating calibration files: 100%|████████████████████████| 3/3 [00:19<00:00, 6.51s/it]
XMM-Newton Pointed - Generating ODF summary files: 100%|████████████████████████| 3/3 [00:03<00:00, 1.25s/it]
XMM-Newton Pointed - Assembling PN and PN-OOT event lists: 100%|████████████████| 3/3 [03:37<00:00, 72.36s/it]
XMM-Newton Pointed - Assembling MOS event lists: 100%|██████████████████████████| 6/6 [01:22<00:00, 13.70s/it]
XMM-Newton Pointed - Finding PN/MOS soft-proton flares: 100%|███████████████████| 9/9 [00:17<00:00, 1.93s/it]
XMM-Newton Pointed - Generating cleaned PN/MOS event lists: 100%|███████████████| 8/8 [00:03<00:00, 2.62it/s]
XMM-Newton Pointed - Generating final PN/MOS event lists: 100%|████████████████| 8/8 [00:00<00:00, 120.67it/s]
Generating products of type(s) ccf: 100%|███████████████████████████████████████| 3/3 [00:19<00:00, 6.37s/it]
Generating products of type(s) image: 100%|█████████████████████████████████████| 8/8 [00:01<00:00, 7.19it/s]
Generating products of type(s) expmap: 100%|████████████████████████████████████| 8/8 [00:22<00:00, 2.84s/it]
Generating products of type(s) image: 100%|█████████████████████████████████████| 8/8 [00:01<00:00, 7.01it/s]
Generating products of type(s) expmap: 100%|████████████████████████████████████| 8/8 [00:22<00:00, 2.84s/it]
General arguments for all processing functions
There are some arguments that all processing functions will take - they don’t control the behaviour of the processing step itself, but instead how the commands are executed:
num_cores - The number of cores that the function is allowed to use. This is set globally by the daxa.NUM_CORES constant (if set before any other DAXA function is imported), and by default is 90% of the cores available on the system.
timeout - This controls how long an individual instance of the process is allowed to run before it is cancelled; the whole function may run for longer depending how many pieces of data need processing and how many cores are allocated. The default is generally null, but it can be set using an astropy quantity with time units.
Breaking down the XMM processing steps
Setup steps
These functions set up the necessary environment/files for the further processing/reduction of XMM data.
Building calibration files (cif_build)
The cif_build function function is a wrapper for the XMM SAS tool of the same name - it assembles a CIF, which points other SAS tools to the current calibration files required for whatever XMM instrument they are working on.
Only the analysis_date parameter of this function affects the files produced by this function (other things can be controlled, such as the number of cores or timeout) - this date determines which calibration files are selected for the generated CIF, and the default is the current date - another date can be passed as a Python datetime object.
All other processing steps depend on this one
[33]:
help(cif_build)
Help on function cif_build in module daxa.process.xmm.setup:
cif_build(obs_archive: daxa.archive.base.Archive, num_cores: int = 9, disable_progress: bool = False, analysis_date: Union[str, datetime.datetime] = 'now', timeout: astropy.units.quantity.Quantity = None) -> Tuple[dict, dict, dict, str, int, bool, astropy.units.quantity.Quantity]
A DAXA Python interface for the SAS cifbuild command, used to generate calibration files for XMM observations
prior to processing. The observation date is supplied by the XMM mission instance(s), and is the date when the
observation was started (as acquired from the XSA).
:param Archive obs_archive: An Archive instance containing XMM mission instances for which observation calibration
files should be generated. This function will fail if no XMM missions are present in the archive.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param str/datetime analysis_date: The analysis date for which to generate calibration file. The default is
'now', but this parameter can be used to create calibration files as they would have been on a past date.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire cif_build process, but a timeout for individual
ObsID processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Summarising the available data files (odf_ingest)
The odf_ingest function function simply examines the observation data file (ODF) directory, and determines the instruments (including observing modes and sub-exposures) that have data present - it then creates a SAS summary file.
Our implementation of this function wraps the original SAS tool, and then adds a second step - this parses the SAS summary file and extracts the information that is relevant to whether we can use a particular instrument (and sub-exposure of an instrument) for astrophysics. This is what populates the XMM entry in the observation_summaries property of the archive class.
All subsequent steps depend on this one - and observation_summaries will have no XMM entry until this is run
[34]:
help(odf_ingest)
Help on function odf_ingest in module daxa.process.xmm.setup:
odf_ingest(obs_archive: daxa.archive.base.Archive, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the SAS odfingest task, which creates a summary of the raw data available in the ODF
directory, and is used by many SAS processing tasks.
:param Archive obs_archive: An Archive instance containing XMM mission instances for which observation summary
files should be generated. This function will fail if no XMM missions are present in the archive.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire odf_ingest process, but a timeout for individual
ObsID processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
EPIC Cameras (CCD)
These functions are specifically for the preparation of data from the three EPIC cameras on XMM; PN, MOS1, and MOS2.
Assembling whole-camera MOS initial event lists (emchain)
The emchain function is what assembles the raw, separate-CCD-level, event lists and files into a single raw events list for a whole XMM MOS camera exposure. If there are multiple sub-exposures which DAXA has identified as being usable, then a separate raw event list is created for each of them - they are combined in a later processing step.
We have implemented this method so that MOS1 and MOS2 (and sub-exposures, if present) data are processed in parallel for a given observation (and all observations are processed in parallel as well).
Only the process_unscheduled argument passed to emchain can change the files produced, as it controls whether unscheduled sub-exposures should be processed or not - the default is True.
[35]:
help(emchain)
Help on function emchain in module daxa.process.xmm.assemble:
emchain(obs_archive: daxa.archive.base.Archive, process_unscheduled: bool = True, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the emchain SAS process on XMM missions in the passed archive, which assembles the
MOS-specific ODFs into combined photon event lists - rather than the per CCD files that existed before. The
emchain manual can be found here (https://xmm-tools.cosmos.esa.int/external/sas/current/doc/emchain.pdf) and
gives detailed explanations of the process.
The DAXA wrapper does not allow emchain to automatically loop through all the sub-exposures for a given
ObsID-MOSX combo, but rather creates a separate process call for each of them. This allows for greater
parallelisation (if on a system with a significant core count), but also allows the same level of granularity
in the logging of processing of different sub-exposures as in DAXA's epchain implementation.
The particular CCDs to be processed are not specified in emchain, unlike in epchain, because it can sometimes
have unintended consequences. For instance processing a MOS observation in FastUncompressed mode, with timing
on CCD 1 and imaging everywhere else, can cause emchain to fail (even though no actual failure occurs) because
the submode is set to Unknown, rather than FastUncompressed.
:param Archive obs_archive: An Archive instance containing XMM mission instances with MOS observations for
which emchain should be run. This function will fail if no XMM missions are present in the archive.
:param bool process_unscheduled: Whether this function should also process sub-exposures marked 'U', for
unscheduled. Default is True, in which case they will be processed.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire emchain process, but a timeout for individual
ObsID-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Assembling whole-camera PN initial event lists (epchain)
The epchain function serves much the same purpose as emchain, but for PN camera data. There is only one PN camera, but the presence of multiple sub-exposures is still possible. This function generates raw, whole-camera, event lists just like emchain, but also creates whole-camera out-of-time (OOT) event lists, which attempts to identify event lists that were detected during a CCD readout (this can leave a characteristic streak in the direction of readout for bright sources in PN observations).
[36]:
help(epchain)
Help on function epchain in module daxa.process.xmm.assemble:
epchain(obs_archive: daxa.archive.base.Archive, process_unscheduled: bool = True, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the epchain SAS process on XMM missions in the passed archive, which assembles the
PN-specific ODFs into combined photon event lists - rather than the per CCD files that existed before. A run of
epchain for out of time (OOT) events is also performed as part of this function call. The epchain manual can be
found here (https://xmm-tools.cosmos.esa.int/external/sas/current/doc/epchain.pdf) and gives detailed
explanations of the process.
Per the advice of the SAS epchain manual, the OOT event list epchain call is performed first, and its intermediate
files are saved and then used for the normal call to epchain.
:param Archive obs_archive: An Archive instance containing XMM mission instances with PN observations for
which epchain should be run. This function will fail if no XMM missions are present in the archive.
:param bool process_unscheduled: Whether this function should also process sub-exposures marked 'U', for
unscheduled. Default is True, in which case they will be processed.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire epchain process, but a timeout for individual
ObsID-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Searching for anomalous MOS CCD states (emanom)
The emanom function wraps the SAS function of the same name, and is an optional step that finds MOS CCDs which have operated in an ‘anomalous’ state, where the background at E < 1 keV is strongly enhanced. However, it should be noted that the “anonymous” anomalous state of MOS1 CCD#4 is not always detectable from the unexposed corner data.
We note that this is not enabled by default in the ``full_process_xmm`` function, and your results may vary.
[37]:
help(emanom)
Help on function emanom in module daxa.process.xmm.check:
emanom(obs_archive: daxa.archive.base.Archive, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the SAS emanom function, which attempts to identify when MOS CCDs are have operated in an
'anomalous' state, where the background at E < 1 keV is strongly enhanced. Data above 2 keV are unaffected, so
CCDs in anomalous states used for science where the soft X-rays are unnecessary do not need to be excluded.
The emanom task calculates the (2.5-5.0 keV)/(0.4-0.8 keV) hardness ratio from the corner data to determine
whether a chip is in an anomalous state. However, it should be noted that the "anonymous" anomalous state of
MOS1 CCD#4 is not always detectable from the unexposed corner data.
This functionality is only usable if you have SAS v19.0.0 or higher - a version check will be performed and
a warning raised (though no error will be raised) if you use this function with an earlier SAS version.
:param Archive obs_archive: An Archive instance containing XMM mission instances with MOS observations for
which emchain should be run. This function will fail if no XMM missions are present in the archive.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire emanom process, but a timeout for individual
ObsID-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Searching for flared periods of the observations (espfilt)
This particularly important function (espfilt) wraps the SAS tool of the same name, and attempts to automatically detect periods of observations which have been overwhelmed by soft-proton flaring. The ultimate result of this process is a set of ‘good time intervals’, which are periods of a particular observation that have been deemed safe to use; this check if performed on all instruments and sub-exposures of an observation independently.
Several configuration options are provided, mirroring those that can be passed into the espfilt SAS tool:
method - There are two methods of identifying flaring available, ‘histogram’ (the default) and ‘ratio’. The histogram method fits a gaussian to the whole-FoV count-rate distribution (in order to find a mean and a standard deviation), and then defines good-time intervals by finding the time periods where the whole-FoV lightcurve count-rate is within
allowed_sigmaof the mean. The ‘ratio’ method compares the count-rate from the FoV with the count-rate in the unexposed corners of XMM observations (i.e. there is an active CCD but the telescope is not focusing photons or soft-protons onto it); an acceptable ratio is defined byratio, and points where the lightcurve ratio is lower than or equal to this are considered good-time intervals.with_smoothing - Should smoothing be applied to the light curve data. The default is True, in which case a smoothing factor of 51 seconds is used - an astropy quantity with time units may also be passed to set a custom value.
with_binning - Should binning be applied to the light curve data. The default is True, in which case a binning size of 60 seconds is used - an astropy quantity with time units may also be passed to set a custom value.
ratio - The acceptable ratio for the ‘ratio’ method.
filter_lo_en - The lower energy bound used for soft-proton flaring identification; the default is 2.5 keV.
filter_hi_en - The upper energy bound used for soft-proton flaring identification; the default is 8.5 keV.
range_scale - Histogram fit range scale factor. The default is a dictionary with an entry for ‘pn’ (15.0) and an entry for ‘mos’ (6.0).
allowed_sigma - Number of stdev from the mean count-rate allowed before a period is considered to not be a good time interval. Default is 3.
gauss_fit_lims - The parameter limts for gaussian fits for the histogram method, the default is (0.1, 6.5).
[38]:
help(espfilt)
Help on function espfilt in module daxa.process.xmm.clean:
espfilt(obs_archive: daxa.archive.base.Archive, method: str = 'histogram', with_smoothing: Union[bool, astropy.units.quantity.Quantity] = True, with_binning: Union[bool, astropy.units.quantity.Quantity] = True, ratio: float = 1.2, filter_lo_en: astropy.units.quantity.Quantity = <Quantity 2500. eV>, filter_hi_en: astropy.units.quantity.Quantity = <Quantity 8500. eV>, range_scale: dict = None, allowed_sigma: float = 3.0, gauss_fit_lims: Tuple[float, float] = (0.1, 6.5), num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
The DAXA wrapper for the XMM SAS task espfilt, which attempts to identify good time intervals with minimal
soft-proton flaring for individual sub-exposures (if multiple have been taken) of XMM ObsID-Instrument
combinations. Both EPIC-PN and EPIC-MOS observations will be processed by this function.
This function does not generate final event lists, but instead is used to create good-time-interval files
which are then applied to the creation of final event lists, along with other user-specified filters, in the
'cleaned_evt_lists' function.
:param Archive obs_archive: An Archive instance containing XMM mission instances with PN/MOS observations for
which espfilt should be run. This function will fail if no XMM missions are present in the archive.
:param str method: The method that espfilt should use to find soft proton flaring. Either 'ratio' or 'histogram'
can be selected. The default is 'histogram'.
:param bool/Quantity with_smoothing: Should smoothing be applied to the light curve data. If set to True (the
default) a smoothing factor of 51 seconds is used, if set to False smoothing will be turned off, if an astropy
Quantity is passed (with units convertible to seconds) then that value will be used for the smoothing factor.
:param bool/Quantity with_binning: Should binning be applied to the light curve data. If set to True (the
default) a bin size of 60 seconds is used, if set to False binning will be turned off, if an astropy
Quantity is passed (with units convertible to seconds) then that value will be used for the bin size.
:param float ratio: Flaring ratio of annulus counts.
:param Quantity filter_lo_en: The lower energy bound for the event lists used for soft proton flaring
identification.
:param Quantity filter_hi_en: The upper energy bound for the event lists used for soft proton flaring
identification.
:param dict range_scale: Histogram fit range scale factor. The default is a dictionary with an entry for 'pn'
(15.0) and an entry for 'mos' (6.0).
:param float allowed_sigma: Limit in sigma for unflared rates.
:param Tuple[float, float] gauss_fit_lims: The parameter limits for gaussian fits.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire espfilt process, but a timeout for individual
ObsID-Inst-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Applying event filtering and good-time-intervals to the raw event lists (cleaned_evt_lists)
The cleaned_evt_lists function uses SAS’ evselect tool to both apply the GTIs generated by the espfilt function, and to apply further filtering of events as directed by the user.
We allow for filtering based on flag, pattern, energy, and MOS anomolous states (if the emanon task was run) - the user can also define custom filtering expressions. The MOS and PN filtering statements are defined separately, as it is standard practise to use different filtering for the different instruments. The exact behaviour is controlled by passing the following arguments:
lo_en - The lowest energy of event that should be left in the output cleaned event list. The default is None, in which case no energy filter is applied.
hi_en - The highest energy of event that should be left in the output cleaned event list. The default is None, in which case no energy filter is applied.
pn_filt_expr - This essentially lets the user set the evselect expression keyword, and determines what filters should be applied to the PN event lists (other than energy range and good time interval application) - the default is (“#XMMEA_EP”, “(PATTERN <= 4)”, “(FLAG .eq. 0)”), where the elements of the tuple are combined into the base filtering expression.
mos_filt_expr - This essentially lets the user set the evselect expression keyword, and determines what filters should be applied to the MOS event lists (other than energy range and good time interval application) - the default is (“#XMMEA_EM”, “(PATTERN <= 12)”, “(FLAG .eq. 0)”), where the elements of the tuple are combined into the base filtering expression.
filt_mos_anom_state - Whether the function should use the results of an ‘emanom’ run to identify and remove MOS CCDs that are in anomolous states. Default is ‘False’, meaning no such filtering would be applied.
acc_mos_anom_states - A list/tuple of acceptable MOS CCD status codes found by emanom (status- G is good at all energies, I is intermediate for E<1 keV, B is bad for E<1 keV, O is off, chip not in use, U is undetermined (low band counts <= 0)).
[39]:
help(cleaned_evt_lists)
Help on function cleaned_evt_lists in module daxa.process.xmm.assemble:
cleaned_evt_lists(obs_archive: daxa.archive.base.Archive, lo_en: astropy.units.quantity.Quantity = None, hi_en: astropy.units.quantity.Quantity = None, pn_filt_expr: Union[str, List[str]] = ('#XMMEA_EP', '(PATTERN <= 4)', '(FLAG .eq. 0)'), mos_filt_expr: Union[str, List[str]] = ('#XMMEA_EM', '(PATTERN <= 12)', '(FLAG .eq. 0)'), filt_mos_anom_state: bool = False, acc_mos_anom_states: Union[List[str], str] = ('G', 'I', 'U'), num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function is used to apply the soft-proton filtering (along with any other filtering you may desire, including
the setting of energy limits) to XMM-Newton event lists, resulting in the creation of sets of cleaned event lists
which are ready to be analysed (or merged together, if there are multiple exposures for a particular
observation-instrument combination).
:param Archive obs_archive: An Archive instance containing XMM mission instances for which cleaned event lists
should be created. This function will fail if no XMM missions are present in the archive.
:param Quantity lo_en: The lower bound of an energy filter to be applied to the cleaned, filtered, event lists. If
'lo_en' is set to an Astropy Quantity, then 'hi_en' must be as well. Default is None, in which case no
energy filter is applied.
:param Quantity hi_en: The upper bound of an energy filter to be applied to the cleaned, filtered, event lists. If
'hi_en' is set to an Astropy Quantity, then 'lo_en' must be as well. Default is None, in which case no
energy filter is applied.
:param str/List[str]/Tuple[str] pn_filt_expr: The filter expressions to be applied to EPIC-PN event lists. Either
a single string expression can be passed, or a list/tuple of separate expressions, which will be combined
using '&&' logic before being used as the expression for evselect. Other expression components can be
added during the process of the function, such as GTI filtering and energy filtering.
:param str/List[str]/Tuple[str] mos_filt_expr: The filter expressions to be applied to EPIC-MOS event lists. Either
a single string expression can be passed, or a list/tuple of separate expressions, which will be combined
using '&&' logic before being used as the expression for evselect. Other expression components can be
added during the process of the function, such as GTI filtering, energy filtering, and anomalous state CCD
filtering..
:param bool filt_mos_anom_state: Whether this function should use the results of an 'emanom' run
to identify and remove MOS CCDs that are in anomolous states. If 'False' is passed then no such filtering
will be applied.
:param List[str]/str acc_mos_anom_states: A list/tuple of acceptable MOS CCD status codes found by emanom
(status- G is good at all energies, I is intermediate for E<1 keV, B is bad for E<1 keV, O is off, chip
not in use, U is undetermined (low band counts <= 0)).
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire cleaned_evt_lists process, but a timeout for individual
ObsID-Inst-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Finalising the event lists by combining sub-exposures (merge_subexposures)
The merge_subexposures function will automatically check for cases where a particular instrument of a particular ObsID has multiple sub-exposures, and then it will automatically combine them. There are no user-provided arguments which will affect the output of this function, only the standard arguments to set number of cores and timeout.
[40]:
help(merge_subexposures)
Help on function merge_subexposures in module daxa.process.xmm.assemble:
merge_subexposures(obs_archive: daxa.archive.base.Archive, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
A function to identify cases where an instrument for a particular XMM observation has multiple
sub-exposures, for which the event lists can be merged. This produces a final event list, which is a
combination of the sub-exposures.
For those observation-instrument combinations with only a single
exposure, this function will rename the cleaned event list so that the naming convention is comparable
to the merged event list naming convention (i.e. sub-exposure identifier will be removed).
:param Archive obs_archive: An Archive instance containing XMM mission instances for which cleaned event lists
should be created. This function will fail if no XMM missions are present in the archive.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire merge_subexposures process, but a timeout for individual
ObsID-Inst processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Generating images and exposure maps (generate_images_expmaps)
This function (generate_images_expmaps) uses XGA (another module in our open source X-ray astrophysics ecosystem) to generate images and exposure maps for the archive that has been created, from the final, cleaned and merged, event lists produced by merge_subexposures. The user can control the upper and lower energy bounds for the images and exposure maps by passing astropy quantities in keV - the default
setup creates images and exposure maps in the 0.5-2.0 keV and 2.0-10.0 keV bands.
[41]:
help(generate_images_expmaps)
Help on function generate_images_expmaps in module daxa.process.xmm.generate:
generate_images_expmaps(obs_archive: daxa.archive.base.Archive, lo_en: astropy.units.quantity.Quantity = <Quantity [0.5, 2. ] keV>, hi_en: astropy.units.quantity.Quantity = <Quantity [ 2., 10.] keV>, num_cores: int = 9)
A function to generate images and exposure maps for a processed XMM mission dataset contained within an
archive. Users can select the energy band(s) that they wish to generate images and exposure maps within.
:param Archive obs_archive:
:param lo_en: The lower energy bound(s) for the product being generated. This can either be passed as a
scalar Astropy Quantity or, if sets of the same product in different energy bands are to be generated, as a
non-scalar Astropy Quantity. If multiple lower bounds are passed, they must each have an entry in the
hi_en argument. The default is 'Quantity([0.5, 2.0], 'keV')', which will generate two sets of products, one
with lower bound 0.5 keV, the other with lower bound 2 keV.
:param hi_en: The upper energy bound(s) for the product being generated. This can either be passed as a
scalar Astropy Quantity or, if sets of the same product in different energy bands are to be generated, as a
non-scalar Astropy Quantity. If multiple upper bounds are passed, they must each have an entry in the
lo_en argument. The default is 'Quantity([2.0, 10.0], 'keV')', which will generate two sets of products, one
with upper bound 2.0 keV, the other with upper bound 10 keV.
:param int num_cores:
Reflection Grating Spectrometer (RGS)
These functions prepare data for the two RGS instruments on XMM.
Preparing the initial raw RGS event lists (rgs_events)
The rgs_events function creates the initial raw event lists by calibrating and combining the separate CCD outputs in RGS event lists. As with the analogous EPIC camera functions, epchain and emchain, this happens separately for the two RGS instruments for each observation, and separately for each sub-exposure of the RGS instruments. There are no user-passed arguments that will control the output of this step.
[42]:
help(rgs_events)
Help on function rgs_events in module daxa.process.xmm.assemble:
rgs_events(obs_archive: daxa.archive.base.Archive, process_unscheduled: bool = True, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the first step of the SAS RGS processing pipeline, rgsproc. This should prepare the RGS event
lists by calibrating and combining the separate CCD event lists into RGS events. This happens separately for RGS1
and RGS2, and for each sub-exposure of the two instruments.
None of the calculations performed in this step should be affected by the choice of source, the first step where
the choice of primary source should be taken into consideration is the next step, rgs_angles; though as DAXA
processes data to be generally useful we will not define a primary source, that is for the user in the future as
the aspect drift calculations can be re-run.
:param Archive obs_archive: An Archive instance containing XMM mission instances with RGS observations for
which RGS processing should be run. This function will fail if no XMM missions are present in the archive.
:param bool process_unscheduled: Whether this function should also process sub-exposures marked 'U', for
unscheduled. Default is True, in which case they will be processed.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire rgs_events process, but a timeout for individual
ObsID-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Calculating the aspect drift for the event lists (rgs_angles)
The rgs_angles function calculates aspect drift for the RGS event lists - i.e. corrections for shifts in measured properties of events caused by the changing pointing position of the telescope (necessary because RGS instruments are grating spectrometers).
This step is source dependant because it depends on the central coordinate - however we run it with an ‘uninformative’ central coordinate, and for full accuracy this process should be repeated when you come to analyse the data.
[43]:
help(rgs_angles)
Help on function rgs_angles in module daxa.process.xmm.assemble:
rgs_angles(obs_archive: daxa.archive.base.Archive, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the second step of the SAS RGS processing pipeline, rgsproc. This should calculate aspect drift
corrections for some 'uninformative' source, and should likely be refined later when these data are used to analyse
a specific source. This happens separately for RGS1 and RGS2, and for each sub-exposure of the two instruments.
:param Archive obs_archive: An Archive instance containing XMM mission instances with RGS observations for
which RGS processing should be run. This function will fail if no XMM missions are present in the archive.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire rgs_events process, but a timeout for individual
ObsID-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]
Filtering RGS event lists (cleaned_rgs_event_lists)
This function (cleaned_rgs_event_lists) is similar in purpose to the cleaned_evt_lists function applied to EPIC data, in that it produces filtered and cleaned event lists. No soft proton filtering is applied however
[44]:
help(cleaned_rgs_event_lists)
Help on function cleaned_rgs_event_lists in module daxa.process.xmm.assemble:
cleaned_rgs_event_lists(obs_archive: daxa.archive.base.Archive, num_cores: int = 9, disable_progress: bool = False, timeout: astropy.units.quantity.Quantity = None)
This function runs the third step of the SAS RGS processing pipeline, rgsproc. Here we filter the events to only
those which should be useful for scientific analysis. The attitude and house-keeping GTIs are also applied. This
happens separately for RGS1 and RGS2, and for each sub-exposure of the two instruments.
Unfortunately it seems that combining sub-exposure event lists for a given ObsID-Instrument combo is not
supported/recommended, combinations of data are generally done after spectrum generation, and even then they
don't exactly recommend it - of course spectrum generation doesn't get done in DAXA. As such this function
will produce individual event lists for RGS sub-exposures.
:param Archive obs_archive: An Archive instance containing XMM mission instances with RGS observations for
which RGS processing should be run. This function will fail if no XMM missions are present in the archive.
:param int num_cores: The number of cores to use, default is set to 90% of available.
:param bool disable_progress: Setting this to true will turn off the SAS generation progress bar.
:param Quantity timeout: The amount of time each individual process is allowed to run for, the default is None.
Please note that this is not a timeout for the entire rgs_events process, but a timeout for individual
ObsID-subexposure processes.
:return: Information required by the SAS decorator that will run commands. Top level keys of any dictionaries are
internal DAXA mission names, next level keys are ObsIDs. The return is a tuple containing a) a dictionary of
bash commands, b) a dictionary of final output paths to check, c) a dictionary of extra info (in this case
obs and analysis dates), d) a generation message for the progress bar, e) the number of cores allowed, and
f) whether the progress bar should be hidden or not.
:rtype: Tuple[dict, dict, dict, str, int, bool, Quantity]