Non-Imaging Transient Reconstruction And TEmporal Search (NITRATES)

Overview

Non-Imaging Transient Reconstruction And TEmporal Search (NITRATES)

This repo contains the full NITRATES pipeline for maximum likelihood-driven discovery and localization of Gamma Ray Bursts in the Neil Gehrels Swift Observatory's Burst Alert Telescope (BAT) instrument.

A description of the method can be found in DeLaunay & Tohuvavohu (2021). We ask scientific users of this code to cite the paper.

The BAT instrumental response functions necessary for this pipeline can be found in this Zenodo community.

Current Analysis Scripts

run_stuff_grb2.sh Used to run the full targeted analysis. Runs mkdb.py, do_data_setup.py, do_full_rates.py, then do_manage2.py first arg is the trigger time, second arg is the Name of the trigger, and the optional third arg is the minimum duration to use

mkdb.py Creates an sqlite DB that contains the trigger time and important file names DB not used much in the analysis, used to be used to store results and is kind of a relic now

do_data_setup.py Gathers the event, attitude, and enabled detectors files Chooses which dets to mask, based on any hot or cold dets or any det glitches Makes a "filtered" event file that has the events removed outside the usable energy range or far away from the analysis time Also adds a GTI table to the event file for when it's not slewing and there's no multi-det glitches Also makes a partial coding image if there's a usable set of HEASOFT tools

do_full_rates.py Runs the full rates analysis to pick time bins as seeds for the analysis

do_manage2.py Manages the rest of the analysis Submits jobs to the cluster, organizes results, and emails out top results First submits a job for the bkg fit to off-time data Then submits several jobs for the split detector rates analysis Gathers the split rates results and makes the final set of position and time seeds Assigns which jobs will processes which seeds and writes them to rate_seeds.csv (for inside FoV jobs) and out_job_table.csv (for out of FoV jobs) Submits several jobs to the cluster for both inside FoV and outside FoV analysis Gathers results and emails out top results when all of the jobs are done

do_bkg_estimation_wPSs_mp2.py Script to perform the bkg fit to off-time data Ran as a single job, usually with 4 procs

do_rates_mle_InOutFoV2.py Script to perform the split rates analysis Ran as several single proc jobs

do_llh_inFoV4realtime2.py Script to perform the likelihood analysis for seeds that are inside the FoV Ran as several single proc jobs

do_llh_outFoV4realtime2.py Script to perform the likelihood analysis for seeds that are outside the FoV Ran as several single proc jobs

Important Modules

LLH.py

  • Has class and functions to compute the LLH
  • The LLH_webins class handles the data and LLH calculation for a given model and paramaters
    • It takes a model object, the event data, detmask, and start and stop time for inputs
    • Converts the event data within the start and stop time into a 2D histogram in det and energy bins
    • Can then compute the LLH for a given set of paramaters for the model
    • Can do a straight Poisson likelihood or Poisson convovled with a Gaussian error

minimizers.py

  • Funtctions and classes to handle numerically minimizing the NLLH
  • Most minimizer objects are subclasses of NLLH_Minimizer
    • Contains functions for doing parameter transformations and setting bounds
    • Also handles mapping the tuple of paramter values used for a standard scipy minimizer to the dict of paramater names and values used by the LLH and model objects

models.py

  • Has the models that convert input paramaters into the count rate expectations for each det and energy bin in the LLH
  • The models are sub classes of the Model class
  • Currently used diffuse model is Bkg_Model_wFlatA
  • Currently used point source model is Source_Model_InOutFoV, which supports both in and out of FoV positions
  • Currently used simple point source model for known sources is Point_Source_Model_Binned_Rates
  • CompoundModel takes a list of models to make a single model object that can give the total count expectations from all models used

flux_models.py

  • Has functions and classes to handle computing fluxes for different flux models
  • The different flux model object as subclasses of Flux_Model
    • Flux_Model contains methods to calculate the photon fluxes in a set of photon energy bins
    • Used by the response and point source model objects
  • The different available flux models are:
    • Plaw_Flux for a simple power-law
    • Cutoff_Plaw_Flux for a power-law with an exponential cut-off energy
    • Band_Flux for a Band spectrum

response.py

  • Contains the functions and objects for the point source model
  • Most current response object is ResponseInFoV2 and is used in the Source_Model_InOutFoV model

ray_trace_funcs.py

  • Contains the functions and objects to read and perform bilinear interpolation of the foward ray trace images that give the shadowed fraction of detectors at different in FoV sky positions
  • RayTraces class manages the reading and interpolation and is used by the point source response function and simple point source model
Comments
  • Example_LLH_setup_fixed_dirs.ipynb Incorrect import

    Example_LLH_setup_fixed_dirs.ipynb Incorrect import

    In the Example_LLH_setup_fixed_dirs.ipynb there is a block that states:

    from do_manage import im_dist
    

    this gives an error since there is no file in the package with this name. Instead there is a file called do_manage2. Should the line instead be:

    from do_manage2 import im_dist
    

    Thanks!

    opened by parsotat 20
  • Residual batml updates

    Residual batml updates

    Added in some updates I've done in the private repo.

    do_manage2.py -

    • Made it so that long ssh cmds are split up (there's a limit to length)
    • added a --q arg to pass to sub_jobs, which is needed to submit jobs to the virtual queue
    • create started directories to put files in that say if each seed has been started for in and out of FoV LLH jobs

    do_llh_inFoV4realtime2.py -

    • Made it so that each job, when it starts processing a seed, will create a file saying it has started
    • Skips and seed that has already been started
    • Each job, after it has finished its own seeds will check the other jobs' seeds to see if they have been started, and if not, will run them

    do_llh_oFoV4realtime2.py -

    • Made it so that each job, when it starts processing a seed, will create a file saying it has started
    • Skips and seed that has already been started
    • Each job, after it has finished its own seeds will check the other jobs' seeds to see if they have been started, and if not, will run them

    do_rates_mle_InOutFoV2.py -

    • Made it so that it doesn't crash if there's no partial coding image to use

    pyscript_template_rhel7.pbs -

    • Added arg pmem, which give memory per processor and lets the multi-core bkg job have more memory
    opened by jjd330 12
  • optimized model and LLH, slight response speed up

    optimized model and LLH, slight response speed up

    Added model, LLH, and response optimizations. Changed LLH in FoV script to use these updates. Everything is python 3 compatible and tested in python 3.10. Individual parts have been testes and a whole analysis has been run with no errors.
    See summary of changes below.

    Changes in LLH.py

    • Added a LLH_webins2 class with updates to LLH_webins
    • Works the same as LLH_webins, but expects the error to be error^2 instead of error
    • Has added support for models that return counts instead of rates (less multiplications)
    • Caches data dpis and other data products selected for certain time bins in a dictionary so they don't have to be remade
    • Added new pois_norm_conv_n*2 functions that take error^2 and are optimized to do less logs and exps

    Changes in models.py

    • Made it so Source_Model_InOutFoV caches the normalized photon fluxes for each set of spectral params so it doesn't recalculate it every time set_flux_params is called.
    • New model, Sig_Bkg_Model added.
    • It takes an already made bkg model (any model is fine, that keeps all of it's parameters fixed), and a Source_Model_InOutFoV model.
    • Model only has one parameter, "A" that's used to update the signal DPIs in the Source_Model_InOutFoV model while keeping all other internal parameters fixed, bypassing calling any of the bkg models and keeping the bkg DPIs cached.
    • Also has new functions that return count DPIs instead of rate DPIs, that LLH_webins2 uses and saves time by not doing time exposure every LLH eval.
    • Also has new functions to get error DPIs squared, saves time by not doing the square root. LLH_webins2 support these new functions.

    Changes in do_llh_inFoV4realtime2.py

    • Added the new model, Sig_Bkg_Model and LLH object, LLH_webins2.
    • Changed how parameters and set and updated to reflect the new model used.

    Changes in response.py

    • removed 2 unnecessary additions of large arrays in calc_tot_resp_dpis.
    enhancement 
    opened by jjd330 7
  • Updated files/jupyter notebooks to be compatible with python3.

    Updated files/jupyter notebooks to be compatible with python3.

    I have updated the jupyter and python files to be compatible with python3. I have also changed some imports to also be compatible with python 3 (while retaining compatibility with python2).

    opened by parsotat 5
  • Example_Reading_Results Notebook Error

    Example_Reading_Results Notebook Error

    When running the Example_Reading_Results.ipynb, I run into an error in the 13th cell where the function get_rate_res_fnames() is not finding any files with 'rates' in the name. It is searching in the F646018360/ directory and there are none of these files in the repo. Are they supposed to be there or included with the Zenodo data files?

    opened by parsotat 3
  • Making NITRATES Pip Installable

    Making NITRATES Pip Installable

    opened by parsotat 2
  • Separate out operations code from analysis

    Separate out operations code from analysis

    Listening for alerts, downloading data, should all be removed from the NITRATES repo and developed in a separate repo. This repo (Conductor or Orchestrator or...) should import NITRATES code (after https://github.com/Swift-BAT/NITRATES/pull/7 is merged) and run the analyses.

    opened by Tohuvavohu 0
  • Using Jamie's API to get data

    Using Jamie's API to get data

    Currently many cronjobs in the data_scraping folder are used to constantly download all the data. This is overkill and also often breaks. We should instead use Jamie's API to find and download the data, most likely inside of do_data_setup.py . This would also make it much easier to run NITRATES elsewhere.

    good first issue 
    opened by jjd330 0
  • Using new, efficient model and LLH object for out of FoV analysis

    Using new, efficient model and LLH object for out of FoV analysis

    The new model, Sig_Bkg_Model and new LLH object LLH_webins2 that were made and merged here #9 , were only applied to the in FoV analysis in script do_llh_inFoV4realtime2.py . To apply it to the out of FoV analysis similar changes will need to be made to the script do_llh_outFoV4realtime2.py. The changes being using LLH_webins2 instead of LLH_webins and using Sig_Bkg_Model instead of the usual compound model to combine the signal and bkg models. Along with the new way to set the bkg and signal parameters.

    opened by jjd330 2
  • Making NITRATES Pip Installable

    Making NITRATES Pip Installable

    opened by parsotat 1
  • Creating Automated Testing Pipeline

    Creating Automated Testing Pipeline

    There are a few things that we need to think about in order to start implementing automated testing. These are:

    1. What are the things that we need to test? We definitely want to test the llh calculation and the bkg estimation, what else?
    2. Can we create any codes that are self-contained to test the points in item 1?
    3. What GRB do we want to make the default test case that we test everything against?
    opened by parsotat 9
Releases(v0.0.0)
Owner
null
Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

This repository is the official PyTorch implementation of Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

hippopmonkey 4 Dec 11, 2022
Official implementation of NLOS-OT: Passive Non-Line-of-Sight Imaging Using Optimal Transport (IEEE TIP, accepted)

NLOS-OT Official implementation of NLOS-OT: Passive Non-Line-of-Sight Imaging Using Optimal Transport (IEEE TIP, accepted) Description In this reposit

Ruixu Geng(耿瑞旭) 16 Dec 16, 2022
PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020).

NHDRRNet-PyTorch This is the PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020). 0. Differences between Original Paper and

Yutong Zhang 1 Mar 1, 2022
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

null 117 Dec 28, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 274 Jan 5, 2023
Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

Deep Image Search - AI-Based Image Search Engine Deep Image Search is an AI-based image search engine that includes deep transfer learning features Ex

null 139 Jan 1, 2023
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

Aljaz Bozic 134 Dec 16, 2022
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting.

Non-AR Spatial-Temporal Transformer Introduction Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series For

Chen Kai 66 Nov 28, 2022
A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

Confluence: A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection 1. 介绍 用以替代 NMS,在所有 bbox 中挑选出最优的集合。 NMS 仅考虑了 bbox 的得分,然后根据 IOU 来

null 44 Sep 15, 2022
A non-linear, non-parametric Machine Learning method capable of modeling complex datasets

Fast Symbolic Regression Symbolic Regression is a non-linear, non-parametric Machine Learning method capable of modeling complex data sets. fastsr aim

VAMSHI CHOWDARY 3 Jun 22, 2022
Code for "Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search"

Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search This is an implementation for our paper Contextual Non-Loca

Tencent YouTu Research 50 Dec 3, 2022
Useful materials and tutorials for 110-1 NTU DBME5028 (Application of Deep Learning in Medical Imaging)

Useful materials and tutorials for 110-1 NTU DBME5028 (Application of Deep Learning in Medical Imaging)

null 7 Jun 22, 2022
MRQy is a quality assurance and checking tool for quantitative assessment of magnetic resonance imaging (MRI) data.

Front-end View Backend View Table of Contents Description Prerequisites Running Basic Information Measurements User Interface Feedback and usage Descr

Center for Computational Imaging and Personalized Diagnostics 58 Dec 2, 2022
Official Keras Implementation for UNet++ in IEEE Transactions on Medical Imaging and DLMIA 2018

UNet++: A Nested U-Net Architecture for Medical Image Segmentation UNet++ is a new general purpose image segmentation architecture for more accurate i

Zongwei Zhou 1.8k Jan 7, 2023
A package, and script, to perform imaging transcriptomics on a neuroimaging scan.

Imaging Transcriptomics Imaging transcriptomics is a methodology that allows to identify patterns of correlation between gene expression and some prop

Alessio Giacomel 10 Dec 27, 2022
The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

Domain Generalization for Medical Imaging Classification with Linear Dependency Regularization The code release of paper 'Domain Generalization for Me

Yufei Wang 56 Dec 28, 2022