Summary

This PR documents a working prototype which enables the CutoutFactory.cube_cut() method to read TESS cube files directly from AWS S3 storage, by-passing the need for users to download the large cubes locally. (TESS cubes amount to ~110 GB per CCD in Sector 35!)

Example use:

The motivation for this change is to enable fast cloud-based analyses of TESS FFI data, i.e., this feature would enable users to obtain thousands of TESS cut-outs concurrently, limited only by the throughput of S3.

This PR is rough and not ready for review -- it lacks tests, docstrings, etc. I also haven't investigated whether this change would be compatible with @ceb8's awesome recent work on moving cut-outs. However I am happy to prepare a proper PR if we think this is a good idea!

Technical details

This prototype defines two compatible classes, LocalCubeFile and S3CubeFile, which encapsulate two different data access patterns:

• LocalCubeFile uses astropy.io.fits.open() to access a cube as normal.
• S3CubeFile uses the boto3 and asyncio libraries to asynchronously extract slices from the cube using Byte-range GET requests. The number of requests scales with the vertical size of the cutout (requests = n_rows + 3), leveraging the smart columnar way in which the cubes are organized for this purpose.

The S3CubeFile will only work with astrocut-produced cubes at this time. In the future we can hopefully integrate cloud streaming features in a more generic way into astropy.io.fits, but that would be a much more expensive project. See existing discussions at https://github.com/astropy/astropy/issues/3446 and https://github.com/jbcurtin/astro-cloud/issues/1.

Inspiration

The idea to stream partial FITS files from the cloud is not mine and has been explored by many others before me, including:

• @astrofrog, cf. https://gist.github.com/astrofrog/404a5682311bae221fd5308c1498da94
• @pllim & @mustaric, cf. https://github.com/mustaric/lambda-tess-search/blob/master/sandbox/aws_remote_fits.py
• @jbcurtin, cf. https://github.com/jbcurtin/astro-cloud/
• and others, no doubt!

Verification

Click here to see the code snippet I used to verify that S3-based cutouts match local cutouts:

from astrocut import CutoutFactory
path = "/Users/gb/tmp/tess-s0035-1-1-cube.fits"
CutoutFactory().cube_cut(path, "147 -7.5", 10, "local-cutout.fits")
uri = "s3://stpubdata/tess/public/mast/tess-s0035-1-1-cube.fits"
CutoutFactory().cube_cut(uri, "147 -7.5", 10, "s3-cutout.fits")

import lightkurve as lk
lk.read("local-cutout.fits").plot(title="Local Cube");
lk.read("s3-cutout.fits").plot(title="S3 Cube");

Implements a class based off the existing CubeFactory class that can generate cubes out of TICA FFIs.

Note: This is one of the core four projects for MAST Q3 and must be implemented before June 30.

• [x] Implement TICACubeFactory
• [x] Verify file structural similarity between TICA and TESS Cubes
• [x] Verify image similarity between TICA and TESS Cubes
• [x] Add functionality to append incoming FFIs to existing cubes (starting in Sept we get new ones every week, EM1 we get them ever 2 weeks...?)
• [x] Fix truncated string issue: StringTruncateWarning: truncated right side string(s) longer than 9 character(s) during assignment self.info_table[kwd][i] = ffi_data[0].header.get(kwd, nulval)
• [x] First test that the duplicate columns issue is still an issue in delivery 1 of future sectors - so far I think I've only seen it in sector 27. ~Fix duplicate columns issue in build_info_table(). For some reason there are duplicate columns in the order 1 deliveries for keywords: TESS_X, TESS_Y, TESS_VX, TESS_VY etc~
• [x] Code cleanup: Make sure class variables are under init so they get reset for every run. There was an issue with cube_shape being too small, that would get fixed when I reinitialized the class, investigate that.
• [x] Unit test(s)

UPDATE: After investigating TruncatedString warning, I found the source of the issue being under _build_info_table where the default length of a string keyword value is set based on the length of the string keyword value in the designated template_file . In the event that the string kwd value in one FFI is longer than the string kwd value in the template file, a stringTruncated Warning will pop up. I fixed the default value and compared the generated info_table with the info_table before fixing, and it doesn't seem to clip off the kwd value the way I expected it would. So I'm going to leave this error alone and avoid fiddling with the string column length in case that creates a memory issue. The Warning will be noted here for future reference in case I need to revisit.

Screenshot below shows the DATASUM values for 2 FFIs in an info table where one value is 9 characters and the other is 10. This is the string column causing the Warnings.

TICA vs SPOC Cube Analysis

HDU Lists for SPOC and TICA cubes (respectively)

Image comparison between SPOC and TICA FFis within the cubes

Sector 27, Camera 1, CCD 2

Normalized (Array/MEDIAN(Array))

Using this star as a reference point to verify alignment between two FFIs within the respective cubes

Ratio image (TESS/TICA) between exact cutouts of that star for both SPOC and TICA

Verifying time-cadence between SPOC and TESS. Start time between consecutive FFIs lapsed for 10 minutes. Consistent with the EM1 protocol.

Fixing an error originating from having forward slashes and spaces in the target name (e.g. C/2014 UN271). This edit catches any / or spaces in the name of the target and replaces them accordingly, which allows the script to continue writing the output Target Pixel File.

This closes https://github.com/spacetelescope/astrocut/issues/50

With the previous updates, the check confirming that the file exists was removed. Unfortunately, there appears to be edge cases where the file may be empty and this causes an index error. This work adds a check prior to looping through the cutout file dictionary. Also correcting a minor spelling mistake.

FYI: Due to mission freezes, we will have to coordinate the timing of the merging and promotion to PyPi.

Implements a class based off the existing CutoutFactory class that can generate TPFs out of TICA cubes.

Note: This is one of the core four projects for MAST Q3 and must be implemented before June.

• [x] Implement TICACutoutFactory
• [x] Verify file structural similarity between TICA and TESS TPFs
• [x] Verify image similarity between TICA and TESS TPFs
• [x] Ensure TPF header keywords should be the same between TICA and TESS
• [x] Ensure TICA header keyword values are unique from TESS and accurate
• [x] Add new header keyword in TPF that lets users know what FFIs were used to make the TPF (TICA or TESS?)
• [x] Unit test(s)

The PR includes a major update to the look and feel of the Astrocut documentation, as well as added information in the output file formats for various functionalities.

A live version of the new docs can be viewed here: https://astrocut.readthedocs.io/en/doc_updates/

I appear to be experiencing issues with TESScut. For example, requesting a cut-out for Pi Men in sector 1 yields a Cutout Error: No FFIs found for input target. Cutout not performed..

Example:

On the s3-support branch for making direct s3 cutouts, seeing an error with this set of input data:

from astrocut import CutoutFactory

astrocut_cutter = CutoutFactory()

cube_file = 's3://stpubdata/tess/public/mast/tess-s0030-4-3-cube.fits'
coordinates = '102.7 -70.5'
cutout_size = (3, 3)
output_path = '/tmp'

astrocut_cutter.cube_cut(cube_file, coordinates, cutout_size, output_path=output_path)

Traceback:

Traceback (most recent call last):
  File "/Users/bfalk/repos/astrocut/astrocut/tests/test_small_s3_cutout.py", line 12, in <module>
    astrocut_cutter.cube_cut(cube_file, coordinates, cutout_size, output_path=output_path)
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 736, in cube_cut
    self._parse_table_info(cube.table.data, verbose)
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 930, in table
    return self.loop.run_until_complete(self.cube_async.table)
  File "/Users/bfalk/miniconda3/envs/astrocut/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 1018, in table
    self._table = await self.read_table()
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 1032, in read_table
    hdu2_str = await self._read_block(hdu2_offset, length=None)
  File "/Users/bfalk/repos/astrocut/astrocut/cube_cut.py", line 1064, in _read_block
    resp = await self.s3_client.get_object(
  File "/Users/bfalk/miniconda3/envs/astrocut/lib/python3.9/site-packages/aiobotocore/client.py", line 187, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRange) when calling the GetObject operation: The requested range is not satisfiable

Specifying a local file for the cube_file, I don't see this error, and the cutout succeeds.

I'm wondering if this particular sector/file needs some special casing when reading in the table information?

@barentsen wondering if you might be able to take a look at this at some point?

You are using a deprecated version of codecov-action, see https://github.com/codecov/codecov-action for more details. xref astropy/astropy#12245

This is an automated update made by the batchpr tool :robot: - feel free to close if it doesn't look good! You can report issues to @pllim.

To whom it may concern,

If you are using https://github.com/astropy/ci-helpers in your appveyor.yml , please know that the Astropy project has dropped active development/support for Appveyor CI. If it still works, good for you, because we did not remove the relevant files (yet). But if it ever stops working, we have no plans to fix anything for Appveyor CI. Please consider using native Windows support other CI, e.g., Travis CI (see https://docs.travis-ci.com/user/reference/windows/). We apologize for any inconvenience caused.

If this issue is opened in error or irrelevant to you, feel free to close. Thank you.

xref astropy/ci-helpers#464

Adding error handling so that if one file in a list for fits_cut or image_cut raises an error, the user still gets a cutout with the rest of the files. A warning is raised showing which files caused a problem and what the error was.

Add header keyword value quality checks to unit tests for make_cube.py.

There was an incident where the EXT1 header keyword NAXIS2 did not reflect the correct number of FFIs that were added to the cube. We are still unsure of what happened, but as a preventative measure, it would be good to have some checks in place to verify the quality of the primary and image header keyword values.

Suggestion from @falkben RE: Pull Request #59

"Can we use class inheritance here? It looks like most of the changes would just be in the class attributes. If we make those changes, I think we can make these changes much simpler and should be easier to maintain."

Also some commentary I initially made in TicaCubeFactory class:

NOTE: Later down the line 
        we might want to generalize CubeFactory so that we no longer need 
        a separate class for the TICA cubes. 
        
        Modifying the __init__ 
        in CubeFactory so that it can switch parameters based on whether 
        we want cubes for TICA or TESS will probably be the route I take.
        Or see how this is generalized when Zcut wraps CubeFactory?

Cutouts are able to be made on FITS Cut out of the box, but future mission like Roman will be housing data in ASDF files. Find a way to support this. ASDF Cut?