gt4sd.s3 module

S3 storage utilities.

Summary

Classes:

GT4SDS3Client

Functions:

sync_folder_with_s3	Sync the cache with the S3 remote storage.
upload_file_to_s3	Sync the cache with the S3 remote storage.

Reference

class GT4SDS3Client(host, access_key, secret_key, secure=True)

Bases: object

__init__(host, access_key, secret_key, secure=True)

Construct an S3 client.

Parameters

• host (str) – s3 host address.

• access_key (str) – s3 access key.

• secret_key (str) – s3 secret key.

• secure (bool) – whether the connection is secure or not. Defaults to True.

list_bucket_names()

List all available s3 bucket names.

Returns

list with bucket names.

Return type

List[str]

list_object_names(bucket, prefix=None)

List all available objects (recursive) in the given bucket based on a given prefix.

Parameters

• bucket (str) – bucket name to search for objects.

• prefix (Optional[str, None]) – prefix for objects in the bucket. Defaults to None, a.k.a., no prefix filter.

Returns

list with object names.

Return type

List[str]

list_directories(bucket, prefix=None)

List all available “directories” in the given bucket based on a given prefix.

Parameters

• bucket (str) – bucket name to search for objects.

• prefix (Optional[str, None]) – prefix for objects in the bucket. Defaults to None, a.k.a., no prefix filter. Needs to be a “directory” itself.

Returns

list with directory names.

Return type

List[str]

upload_file(bucket, target_filepath, source_filepath)

Upload a local file to S3 bucket.

Parameters

• bucket (str) – bucket name to upload to.

• target_filepath (str) – path to the file in S3.

• source_filepath (str) – path to the file to upload.

Return type

None

sync_folder(bucket, path, prefix=None, force=False)

Sync an entire folder from S3 recursively and save it under the given path.

If prefix is given, every file under prefix/ in S3 will be saver under path/ in disk (i.e. prefix/ is replaced by path/).

Parameters

• bucket (str) – bucket name to search for objects.

• path (str) – path to save the objects in disk.

• prefix (Optional[str, None]) – prefix for objects in the bucket. Defaults to None, a.k.a., no prefix filter.

• force (bool) – force download even if a file with the same name is present. Defaults to False.

Return type

None

__dict__ = mappingproxy({'__module__': 'gt4sd.s3', '__init__': <function GT4SDS3Client.__init__>, 'list_bucket_names': <function GT4SDS3Client.list_bucket_names>, 'list_object_names': <function GT4SDS3Client.list_object_names>, 'list_directories': <function GT4SDS3Client.list_directories>, 'upload_file': <function GT4SDS3Client.upload_file>, 'sync_folder': <function GT4SDS3Client.sync_folder>, '__dict__': <attribute '__dict__' of 'GT4SDS3Client' objects>, '__weakref__': <attribute '__weakref__' of 'GT4SDS3Client' objects>, '__doc__': None, '__annotations__': {}})

__doc__ = None

__module__ = 'gt4sd.s3'

__weakref__

list of weak references to the object (if defined)

upload_file_to_s3(host, access_key, secret_key, bucket, target_filepath, source_filepath, secure=True)

Sync the cache with the S3 remote storage.

Parameters

• host (str) – s3 host address.

• access_key (str) – s3 access key.

• secret_key (str) – s3 secret key.

• bucket (str) – bucket name to search for objects.

• target_filepath (str) – path to save the objects in s3.

• source_filepath (str) – path to the file to sync.

• secure (bool) – whether the connection is secure or not. Defaults to True.

Raises

S3SyncError – in case of S3 syncing errors.

Return type

None

sync_folder_with_s3(host, access_key, secret_key, bucket, folder_path, prefix=None, secure=True)

Sync the cache with the S3 remote storage.

Parameters

• host (str) – s3 host address.

• access_key (str) – s3 access key.

• secret_key (str) – s3 secret key.

• bucket (str) – bucket name to search for objects.

• folder_path (str) – folder path.

• prefix (Optional[str, None]) – prefix for objects in the bucket. Defaults to None, a.k.a., no prefix filter.

• secure (bool) – whether the connection is secure or not. Defaults to True.

Raises

S3SyncError – in case of S3 syncing errors.

Return type

None