I added two files that allow easy installation of TFODAPI w/ TF 1.15 on an Azure Ubuntu 16.04 DSVM. Both of the files are modified copies of the existing TFODAPI w/ TF 1.12 dockerfile and sh scripts.

I found that TF 1.15 is a much smoother experience. I was hitting memory errors on computationally expensive models using TF v1.2, even using batch sizes of 1 and small images. I haven't had any issues since making the switch to TF 1.15. My choice of TF 1.15 came from a lot of research, with many recommendations from issues such asthis one.

This may not be super useful in the long run. I see that Azure has 18.04 DSVMs out for testing. Also, I read somewhere that TFODAPI will eventually move to TF 2.0. However, in the meantime, it may help someone!

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Please accept this contribution adding the standard Microsoft SECURITY.MD :lock: file to help the community understand the security policy and how to safely report security issues. GitHub uses the presence of this file to light-up security reminders and a link to the file. This pull request commits the latest official SECURITY.MD file from https://github.com/microsoft/repo-templates/blob/main/shared/SECURITY.md.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

I made the class from the NAIP example notebook (https://azure.microsoft.com/en-us/services/open-datasets/catalog/naip/) a little more portable as I've copied/modified it in several new projects now.

@Siyu: I updated sas_blob_utils to use the new Azure Blob Storage Python SDK v12.

There are several places where I changed functionality and/or the interface:

1. in check_blob_existence(), I removed the following check because it would unnecessarily raise an error on public Azure blobs such as: https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan097_img0003.jpg

if SasBlob.get_resource_type_from_uri(sas_uri) != 'blob':
    raise ValueError('The SAS token provided is not for a blob.')

2. I replaced create_blob_from_bytes(), create_blob_from_text(), and create_blob_from_stream() with a single method upload_blob() because the new v12 API no longer distinguishes between the different upload types.

3. I renamed the 1st parameter of generate_blob_sas_uri() from sas_uri to container_sas_uri.

4. I sort the list of blob names from list_blobs_in_container() for determinism.

I manually tested each of the methods without issue:

from sas_blob_utils import SasBlob

from azure.core.exceptions import HttpResponseError


PUBLIC_BLOB_URI = 'https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan097_img0003.jpg'
PUBLIC_CONTAINER_URI = 'https://lilablobssc.blob.core.windows.net/nacti-unzipped'
ANOTHER_PUBLIC_CONTAINER_URI = 'https://lilablobssc.blob.core.windows.net/wcs'

CONTAINER_SAS = 'st=2020-01-01T00%3A00%3A00Z&se=2034-01-01T00%3A00%3A00Z&sp=rl&sv=2019-07-07&sr=c&sig=rsgUcvoniBu/Vjkjzubh6gliU3XGvpE2A30Y0XPW4Vc%3D'
PUBLIC_CONTAINER_URI_WITH_SAS = PUBLIC_CONTAINER_URI + '?' + CONTAINER_SAS

PRIVATE_BLOB_URI = **
PRIVATE_CONTAINER_SAS = **
PRIVATE_BLOB_URI_WITH_SAS = PRIVATE_BLOB_URI + '?' + PRIVATE_CONTAINER_SAS

INVALID_BLOB_URI = "https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan000_img0003.jpg"

PRIVATE_ACCOUNT_NAME = **
PRIVATE_ACCOUNT_SAS = **
PRIVATE_UPLOAD_CONTAINER_URI = **
PRIVATE_UPLOAD_CONTAINER_SAS = **
PRIVATE_UPLOAD_CONTAINER_URI_WITH_SAS = PRIVATE_UPLOAD_CONTAINER_URI + '?' + PRIVATE_UPLOAD_CONTAINER_SAS


def test_get_account_from_uri():
    print('test_get_account_from_uri')
    assert SasBlob.get_account_from_uri(PUBLIC_BLOB_URI) == 'lilablobssc'


def test_get_container_from_uri():
    print('test_get_container_from_uri')
    assert SasBlob.get_container_from_uri(PUBLIC_BLOB_URI) == 'nacti-unzipped'


def test_get_blob_from_uri():
    print('test_get_blob_from_uri')
    expected_blobname = 'part0/sub000/2010_Unit150_Ivan097_img0003.jpg'
    assert SasBlob.get_blob_from_uri(PUBLIC_BLOB_URI) == expected_blobname

    assert SasBlob.get_blob_from_uri(PUBLIC_CONTAINER_URI) is None


def test_get_sas_key_from_uri():
    print('test_get_sas_key_from_uri')
    assert SasBlob.get_sas_key_from_uri(PUBLIC_CONTAINER_URI) is None
    assert SasBlob.get_sas_key_from_uri(PUBLIC_CONTAINER_URI_WITH_SAS) == CONTAINER_SAS


def test_check_blob_existence():
    print('test_check_blob_existence')
    print('- PUBLIC_BLOB_URI...')
    assert SasBlob.check_blob_existence(PUBLIC_BLOB_URI) == True

    print('- PRIVATE_BLOB_URI...')
    assert SasBlob.check_blob_existence(PRIVATE_BLOB_URI) == False
    print('- PUBLIC_CONTAINER_URI...')
    try:
        SasBlob.check_blob_existence(PUBLIC_CONTAINER_URI)
        assert False
    except ValueError:
        pass
    print('- PUBLIC_CONTAINER_URI with blob name...')
    assert SasBlob.check_blob_existence(
        PUBLIC_CONTAINER_URI,
        provided_blob_name='part0/sub000/2010_Unit150_Ivan097_img0003.jpg') == True
    print('- INVALID_BLOB_URI...')
    assert SasBlob.check_blob_existence(INVALID_BLOB_URI) == False


def test_list_blobs_in_container():
    print('test_list_blobs_in_container')
    blobs_list = SasBlob.list_blobs_in_container(ANOTHER_PUBLIC_CONTAINER_URI,
        max_number_to_list=100)
    expected = sorted(['wcs_20200403_bboxes.json.zip', 'wcs_camera_traps.json.zip', 'wcs_camera_traps_00.zip', 'wcs_camera_traps_01.zip', 'wcs_camera_traps_02.zip', 'wcs_camera_traps_03.zip', 'wcs_camera_traps_04.zip', 'wcs_camera_traps_05.zip', 'wcs_camera_traps_06.zip', 'wcs_specieslist.csv', 'wcs_splits.json'])
    assert blobs_list == expected


def test_generate_writable_container_sas():
    new_sas_uri = SasBlob.generate_writable_container_sas(
        account_name=PRIVATE_ACCOUNT_NAME,
        account_key=PRIVATE_ACCOUNT_SAS,
        container_name='christest',
        access_duration_hrs=1)
    print(new_sas_uri)


def test_upload_blob():
    print('test_upload_blob')
    try:
        print('- testing upload to public container, should fail...')
        SasBlob.upload_blob(PUBLIC_CONTAINER_URI_WITH_SAS, 'failblob', data='fail')
        assert False
    except HttpResponseError:
        # HttpResponseError('This request is not authorized to perform this operation using this permission.')
        pass

    try:
        print('- testing upload to private container, should succeed...')
        blob_url = SasBlob.upload_blob(PRIVATE_UPLOAD_CONTAINER_URI_WITH_SAS, 'successblob', data='success')
        assert SasBlob.get_blob_from_uri(blob_url) == 'successblob'
    except Exception:
        assert False


def test_get_blob_to_stream():
    output, props = SasBlob.get_blob_to_stream(PRIVATE_BLOB_URI_WITH_SAS)
    print(props)
    return


if __name__ == '__main__':
    test_get_account_from_uri()
    test_get_container_from_uri()
    test_get_blob_from_uri()
    test_get_sas_key_from_uri()
    test_check_blob_existence()
    test_list_blobs_in_container()
    test_generate_writable_container_sas()
    test_upload_blob()
    test_get_blob_to_stream()

https://github.com/microsoft/ai4eutils/blob/eedfca42ba1a2b4a0b7628c1ea9439d5720accf5/gDrive_download.py#L54

https://github.com/microsoft/ai4eutils/blob/eedfca42ba1a2b4a0b7628c1ea9439d5720accf5/gDrive_download.py#L73

The path split symbol problem is different for different operating systems. Hard-coded file separators should not be used. Instead, use a platform-independent API provided by the language library,such as os.path.join or os.sep

(Reviews optional - just want to make you aware of this)

Functions in this module are used to parse various parts in the SAS token and carry out actions on the blob store based on a SAS. This consolidates sas_blob.py from the AIforEarth-API-Development repo and a few new functions to for example list all blobs in a container.

Things needing review:

• Should we use enumerations in get_resource_type_from_uri and get_permissions_from_uri
• list_blobs_in_container

Please test the functions against your expected output when calling them.