gt4sd.configuration module¶

Module configuration.

Summary¶

Classes:

`GT4SDArtifactManagementConfiguration`	Artifact management configuration.
`GT4SDConfiguration`	GT4SDConfiguration settings from environment variables.

Functions:

`get_algorithm_subdirectories_from_s3_coordinates`	Wrapper to initialize a client and list the directories in a bucket.
`get_algorithm_subdirectories_in_cache`	Get algorithm subdirectories from the cache.
`get_algorithm_subdirectories_with_s3`	Get algorithms in the s3 buckets.
`get_cached_algorithm_path`	rtype `str`
`reset_logging_root_logger`	Reset the root logger from logging library.
`sync_algorithm_with_s3`	Sync an algorithm in the local cache using environment variables.
`upload_to_s3`	Upload an algorithm in source_filepath in target_filepath on a bucket in the model hub. :type target_filepath: `str` :param target_filepath: path to save the objects in s3. :type source_filepath: `str` :param source_filepath: path to the file to sync. :type module: `str` :param module: the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Reference¶

class GT4SDConfiguration(_case_sensitive=None, _nested_model_default_partial_update=None, _env_prefix=None, _env_file=PosixPath('.'), _env_file_encoding=None, _env_ignore_empty=None, _env_nested_delimiter=None, _env_parse_none_str=None, _env_parse_enums=None, _cli_prog_name=None, _cli_parse_args=None, _cli_settings_source=None, _cli_parse_none_str=None, _cli_hide_none_type=None, _cli_avoid_json=None, _cli_enforce_required=None, _cli_use_class_docs_for_groups=None, _cli_exit_on_error=None, _cli_prefix=None, _cli_implicit_flags=None, _secrets_dir=None, **values)[source]¶

Bases: BaseSettings

GT4SDConfiguration settings from environment variables.

Default configurations for gt4sd including a read-only COS for algorithms’ artifacts. Default configuration for gt4sd hub including a read-write COS for algorithms’ artifacts uploaded by users.

gt4sd_local_cache_path: str¶

gt4sd_local_cache_path_algorithms: str¶

gt4sd_local_cache_path_properties: str¶

gt4sd_max_number_of_stuck_calls: int¶

gt4sd_max_number_of_samples: int¶

gt4sd_max_runtime: int¶

gt4sd_create_unverified_ssl_context: bool¶

gt4sd_disable_cudnn: bool¶

gt4sd_s3_host: str¶

gt4sd_s3_access_key: str¶

gt4sd_s3_secret_key: str¶

gt4sd_s3_secure: bool¶

gt4sd_s3_bucket_algorithms: str¶

gt4sd_s3_bucket_properties: str¶

gt4sd_s3_host_hub: str¶

gt4sd_s3_access_key_hub: str¶

gt4sd_s3_secret_key_hub: str¶

gt4sd_s3_secure_hub: bool¶

gt4sd_s3_bucket_hub_algorithms: str¶

gt4sd_s3_bucket_hub_properties: str¶

model_config: ClassVar[SettingsConfigDict] = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_hide_none_type': False, 'cli_implicit_flags': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_settings_source': None, 'cli_use_class_docs_for_groups': False, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': '', 'extra': 'forbid', 'frozen': True, 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_', 'settings_'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_file': None, 'yaml_file_encoding': None}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

static get_instance()[source]¶

Return type: GT4SDConfiguration

__dict__¶

__pydantic_fields_set__: set[str]¶: The names of fields explicitly set during instantiation.

__pydantic_extra__: dict[str, Any] | None¶: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_private__: dict[str, Any] | None¶: Values of private attributes set on the model instance.

__abstractmethods__ = frozenset({})¶

__annotations__ = {'__class_vars__': 'ClassVar[set[str]]', '__private_attributes__': 'ClassVar[Dict[str, ModelPrivateAttr]]', '__pydantic_complete__': 'ClassVar[bool]', '__pydantic_core_schema__': 'ClassVar[CoreSchema]', '__pydantic_custom_init__': 'ClassVar[bool]', '__pydantic_decorators__': 'ClassVar[_decorators.DecoratorInfos]', '__pydantic_extra__': 'dict[str, Any] | None', '__pydantic_fields_set__': 'set[str]', '__pydantic_generic_metadata__': 'ClassVar[_generics.PydanticGenericMetadata]', '__pydantic_parent_namespace__': 'ClassVar[Dict[str, Any] | None]', '__pydantic_post_init__': "ClassVar[None | Literal['model_post_init']]", '__pydantic_private__': 'dict[str, Any] | None', '__pydantic_root_model__': 'ClassVar[bool]', '__pydantic_serializer__': 'ClassVar[SchemaSerializer]', '__pydantic_validator__': 'ClassVar[SchemaValidator | PluggableSchemaValidator]', '__signature__': 'ClassVar[Signature]', 'gt4sd_create_unverified_ssl_context': <class 'bool'>, 'gt4sd_disable_cudnn': <class 'bool'>, 'gt4sd_local_cache_path': <class 'str'>, 'gt4sd_local_cache_path_algorithms': <class 'str'>, 'gt4sd_local_cache_path_properties': <class 'str'>, 'gt4sd_max_number_of_samples': <class 'int'>, 'gt4sd_max_number_of_stuck_calls': <class 'int'>, 'gt4sd_max_runtime': <class 'int'>, 'gt4sd_s3_access_key': <class 'str'>, 'gt4sd_s3_access_key_hub': <class 'str'>, 'gt4sd_s3_bucket_algorithms': <class 'str'>, 'gt4sd_s3_bucket_hub_algorithms': <class 'str'>, 'gt4sd_s3_bucket_hub_properties': <class 'str'>, 'gt4sd_s3_bucket_properties': <class 'str'>, 'gt4sd_s3_host': <class 'str'>, 'gt4sd_s3_host_hub': <class 'str'>, 'gt4sd_s3_secret_key': <class 'str'>, 'gt4sd_s3_secret_key_hub': <class 'str'>, 'gt4sd_s3_secure': <class 'bool'>, 'gt4sd_s3_secure_hub': <class 'bool'>, 'model_computed_fields': 'ClassVar[Dict[str, ComputedFieldInfo]]', 'model_config': 'ClassVar[SettingsConfigDict]', 'model_fields': 'ClassVar[Dict[str, FieldInfo]]'}¶

__class_vars__: ClassVar[set[str]] = {}¶: The names of the class variables defined on the model.

__doc__ = "GT4SDConfiguration settings from environment variables.\n\n Default configurations for gt4sd including a read-only COS for algorithms' artifacts.\n Default configuration for gt4sd hub including a read-write COS for algorithms' artifacts uploaded by users.\n "¶

__hash__() → int¶

Return hash(self).

Return type: int

__module__ = 'gt4sd.configuration'¶

__private_attributes__: ClassVar[Dict[str, ModelPrivateAttr]] = {}¶: Metadata about the private attributes of the model.

__pydantic_complete__: ClassVar[bool] = True¶: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__: ClassVar[CoreSchema] = {'cls': <class 'gt4sd.configuration.GT4SDConfiguration'>, 'config': {'extra_fields_behavior': 'forbid', 'title': 'GT4SDConfiguration', 'validate_default': True}, 'custom_init': True, 'metadata': {'pydantic_js_annotation_functions': [], 'pydantic_js_functions': [functools.partial(<function modify_model_json_schema>, cls=<class 'gt4sd.configuration.GT4SDConfiguration'>, title=None), <bound method BaseModel.__get_pydantic_json_schema__ of <class 'gt4sd.configuration.GT4SDConfiguration'>>]}, 'ref': 'gt4sd.configuration.GT4SDConfiguration:94662636486736', 'root_model': False, 'schema': {'computed_fields': [], 'fields': {'gt4sd_create_unverified_ssl_context': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': False, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_disable_cudnn': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': False, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_local_cache_path': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': '/home/runner/.gt4sd', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_local_cache_path_algorithms': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'algorithms', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_local_cache_path_properties': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'properties', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_max_number_of_samples': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 1000000, 'schema': {'type': 'int'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_max_number_of_stuck_calls': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 50, 'schema': {'type': 'int'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_max_runtime': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 86400, 'schema': {'type': 'int'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_access_key': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': '6e9891531d724da89997575a65f4592e', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_access_key_hub': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'd9536662ebcf462f937efb9f58012830', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_algorithms': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'gt4sd-cos-algorithms-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_hub_algorithms': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'gt4sd-cos-hub-algorithms-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_hub_properties': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'gt4sd-cos-hub-properties-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_properties': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 'gt4sd-cos-properties-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_host': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 's3.par01.cloud-object-storage.appdomain.cloud', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_host_hub': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': 's3.par01.cloud-object-storage.appdomain.cloud', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secret_key': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': '5997d63c4002cc04e13c03dc0c2db9dae751293dab106ac5', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secret_key_hub': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': '934d1f3afdaea55ac586f6c2f729ac2ba2694bb8e975ee0b', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secure': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': True, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secure_hub': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func>], 'pydantic_js_functions': []}, 'schema': {'default': True, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}}, 'model_name': 'GT4SDConfiguration', 'type': 'model-fields'}, 'type': 'model'}¶: The core schema of the model.

__pydantic_custom_init__: ClassVar[bool] = True¶: Whether the model has a custom __init__ method.

__pydantic_decorators__: ClassVar[_decorators.DecoratorInfos] = DecoratorInfos(validators={}, field_validators={}, root_validators={}, field_serializers={}, model_serializers={}, model_validators={}, computed_fields={})¶: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: ClassVar[_generics.PydanticGenericMetadata] = {'args': (), 'origin': None, 'parameters': ()}¶: Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: ClassVar[Dict[str, Any] | None] = None¶: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: ClassVar[None | Literal['model_post_init']] = None¶: The name of the post-init method for the model, if defined.

__pydantic_serializer__: ClassVar[SchemaSerializer] = SchemaSerializer(serializer=Model( ModelSerializer { class: Py( 0x000056185cd24050, ), serializer: Fields( GeneralFieldsSerializer { fields: { "gt4sd_s3_bucket_hub_properties": SerField { key_py: Py( 0x00007f1ea60f12f0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60cdf50, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_create_unverified_ssl_context": SerField { key_py: Py( 0x00007f1ea60cdcb0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000056185a463580, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_local_cache_path_algorithms": SerField { key_py: Py( 0x00007f1ea60cdd10, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea6ed57b0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_max_number_of_samples": SerField { key_py: Py( 0x00007f1ea60f0e90, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea625c8b0, ), ), serializer: Int( IntSerializer, ), }, ), ), required: true, }, "gt4sd_s3_host": SerField { key_py: Py( 0x00007f1ea60e7d30, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60cddd0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secure_hub": SerField { key_py: Py( 0x00007f1ea60f1250, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000056185a4635a0, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_max_number_of_stuck_calls": SerField { key_py: Py( 0x00007f1ea60f0e40, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea9468710, ), ), serializer: Int( IntSerializer, ), }, ), ), required: true, }, "gt4sd_disable_cudnn": SerField { key_py: Py( 0x00007f1ea60f0f30, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000056185a463580, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_s3_host_hub": SerField { key_py: Py( 0x00007f1ea60f1160, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60cddd0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_max_runtime": SerField { key_py: Py( 0x00007f1ea60f0ee0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea625c830, ), ), serializer: Int( IntSerializer, ), }, ), ), required: true, }, "gt4sd_s3_access_key": SerField { key_py: Py( 0x00007f1ea60f0f80, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60cde30, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secret_key": SerField { key_py: Py( 0x00007f1ea60f0fd0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60b23a0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secure": SerField { key_py: Py( 0x00007f1ea60e7d70, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000056185a4635a0, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_local_cache_path": SerField { key_py: Py( 0x00007f1ea60f0df0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea54f9ca0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_properties": SerField { key_py: Py( 0x00007f1ea60f1110, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60f10c0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_access_key_hub": SerField { key_py: Py( 0x00007f1ea60f11b0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60cde90, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_local_cache_path_properties": SerField { key_py: Py( 0x00007f1ea60cdd70, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea87ac5b0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_algorithms": SerField { key_py: Py( 0x00007f1ea60f1070, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60f1020, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secret_key_hub": SerField { key_py: Py( 0x00007f1ea60f1200, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60b2480, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_hub_algorithms": SerField { key_py: Py( 0x00007f1ea60f12a0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f1ea60cdef0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, }, computed_fields: Some( ComputedFields( [], ), ), mode: SimpleDict, extra_serializer: None, filter: SchemaFilter { include: None, exclude: None, }, required_fields: 20, }, ), has_extra: false, root_model: false, name: "GT4SDConfiguration", }, ), definitions=[])¶: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: ClassVar[SchemaValidator | PluggableSchemaValidator] = SchemaValidator(title="GT4SDConfiguration", validator=Model( ModelValidator { revalidate: Never, validator: ModelFields( ModelFieldsValidator { fields: [ Field { name: "gt4sd_local_cache_path", lookup_key: Simple { key: "gt4sd_local_cache_path", py_key: Py( 0x00007f1ea54fa7e0, ), path: LookupPath( [ S( "gt4sd_local_cache_path", Py( 0x00007f1ea54fa830, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0df0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea54f9ca0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_local_cache_path_algorithms", lookup_key: Simple { key: "gt4sd_local_cache_path_algorithms", py_key: Py( 0x00007f1ea5541dd0, ), path: LookupPath( [ S( "gt4sd_local_cache_path_algorithms", Py( 0x00007f1ea5541e30, ), ), ], ), }, name_py: Py( 0x00007f1ea60cdd10, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea6ed57b0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_local_cache_path_properties", lookup_key: Simple { key: "gt4sd_local_cache_path_properties", py_key: Py( 0x00007f1ea5541e90, ), path: LookupPath( [ S( "gt4sd_local_cache_path_properties", Py( 0x00007f1ea5541ef0, ), ), ], ), }, name_py: Py( 0x00007f1ea60cdd70, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea87ac5b0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_max_number_of_stuck_calls", lookup_key: Simple { key: "gt4sd_max_number_of_stuck_calls", py_key: Py( 0x00007f1ea54fa790, ), path: LookupPath( [ S( "gt4sd_max_number_of_stuck_calls", Py( 0x00007f1ea54fa880, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0e40, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea9468710, ), ), on_error: Raise, validator: Int( IntValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[int]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_max_number_of_samples", lookup_key: Simple { key: "gt4sd_max_number_of_samples", py_key: Py( 0x00007f1ea54fa8d0, ), path: LookupPath( [ S( "gt4sd_max_number_of_samples", Py( 0x00007f1ea54fa920, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0e90, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea625c8b0, ), ), on_error: Raise, validator: Int( IntValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[int]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_max_runtime", lookup_key: Simple { key: "gt4sd_max_runtime", py_key: Py( 0x00007f1ea54fa970, ), path: LookupPath( [ S( "gt4sd_max_runtime", Py( 0x00007f1ea54fa9c0, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0ee0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea625c830, ), ), on_error: Raise, validator: Int( IntValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[int]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_create_unverified_ssl_context", lookup_key: Simple { key: "gt4sd_create_unverified_ssl_context", py_key: Py( 0x00007f1ea5541f50, ), path: LookupPath( [ S( "gt4sd_create_unverified_ssl_context", Py( 0x00007f1ea5541fb0, ), ), ], ), }, name_py: Py( 0x00007f1ea60cdcb0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000056185a463580, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_disable_cudnn", lookup_key: Simple { key: "gt4sd_disable_cudnn", py_key: Py( 0x00007f1ea54faa10, ), path: LookupPath( [ S( "gt4sd_disable_cudnn", Py( 0x00007f1ea54faa60, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0f30, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000056185a463580, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_host", lookup_key: Simple { key: "gt4sd_s3_host", py_key: Py( 0x00007f1ea54db7f0, ), path: LookupPath( [ S( "gt4sd_s3_host", Py( 0x00007f1ea54db7b0, ), ), ], ), }, name_py: Py( 0x00007f1ea60e7d30, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60cddd0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_access_key", lookup_key: Simple { key: "gt4sd_s3_access_key", py_key: Py( 0x00007f1ea54faab0, ), path: LookupPath( [ S( "gt4sd_s3_access_key", Py( 0x00007f1ea54fab00, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0f80, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60cde30, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secret_key", lookup_key: Simple { key: "gt4sd_s3_secret_key", py_key: Py( 0x00007f1ea54fab50, ), path: LookupPath( [ S( "gt4sd_s3_secret_key", Py( 0x00007f1ea54faba0, ), ), ], ), }, name_py: Py( 0x00007f1ea60f0fd0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60b23a0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secure", lookup_key: Simple { key: "gt4sd_s3_secure", py_key: Py( 0x00007f1ea54db770, ), path: LookupPath( [ S( "gt4sd_s3_secure", Py( 0x00007f1ea54db730, ), ), ], ), }, name_py: Py( 0x00007f1ea60e7d70, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000056185a4635a0, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_algorithms", lookup_key: Simple { key: "gt4sd_s3_bucket_algorithms", py_key: Py( 0x00007f1ea54fabf0, ), path: LookupPath( [ S( "gt4sd_s3_bucket_algorithms", Py( 0x00007f1ea54fac40, ), ), ], ), }, name_py: Py( 0x00007f1ea60f1070, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60f1020, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_properties", lookup_key: Simple { key: "gt4sd_s3_bucket_properties", py_key: Py( 0x00007f1ea54fac90, ), path: LookupPath( [ S( "gt4sd_s3_bucket_properties", Py( 0x00007f1ea54face0, ), ), ], ), }, name_py: Py( 0x00007f1ea60f1110, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60f10c0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_host_hub", lookup_key: Simple { key: "gt4sd_s3_host_hub", py_key: Py( 0x00007f1ea54fad30, ), path: LookupPath( [ S( "gt4sd_s3_host_hub", Py( 0x00007f1ea54fad80, ), ), ], ), }, name_py: Py( 0x00007f1ea60f1160, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60cddd0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_access_key_hub", lookup_key: Simple { key: "gt4sd_s3_access_key_hub", py_key: Py( 0x00007f1ea54fadd0, ), path: LookupPath( [ S( "gt4sd_s3_access_key_hub", Py( 0x00007f1ea54fae20, ), ), ], ), }, name_py: Py( 0x00007f1ea60f11b0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60cde90, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secret_key_hub", lookup_key: Simple { key: "gt4sd_s3_secret_key_hub", py_key: Py( 0x00007f1ea54fae70, ), path: LookupPath( [ S( "gt4sd_s3_secret_key_hub", Py( 0x00007f1ea54faec0, ), ), ], ), }, name_py: Py( 0x00007f1ea60f1200, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60b2480, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secure_hub", lookup_key: Simple { key: "gt4sd_s3_secure_hub", py_key: Py( 0x00007f1ea54faf10, ), path: LookupPath( [ S( "gt4sd_s3_secure_hub", Py( 0x00007f1ea54faf60, ), ), ], ), }, name_py: Py( 0x00007f1ea60f1250, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000056185a4635a0, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_hub_algorithms", lookup_key: Simple { key: "gt4sd_s3_bucket_hub_algorithms", py_key: Py( 0x00007f1ea54fafb0, ), path: LookupPath( [ S( "gt4sd_s3_bucket_hub_algorithms", Py( 0x00007f1ea54fb000, ), ), ], ), }, name_py: Py( 0x00007f1ea60f12a0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60cdef0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_hub_properties", lookup_key: Simple { key: "gt4sd_s3_bucket_hub_properties", py_key: Py( 0x00007f1ea54fb050, ), path: LookupPath( [ S( "gt4sd_s3_bucket_hub_properties", Py( 0x00007f1ea54fb0a0, ), ), ], ), }, name_py: Py( 0x00007f1ea60f12f0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f1ea60cdf50, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f1ea71db950, ), }, ), frozen: false, }, ], model_name: "GT4SDConfiguration", extra_behavior: Forbid, extras_validator: None, strict: false, from_attributes: false, loc_by_alias: true, }, ), class: Py( 0x000056185cd24050, ), post_init: None, frozen: false, custom_init: true, root_model: false, undefined: Py( 0x00007f1ea71db950, ), name: "GT4SDConfiguration", }, ), definitions=[], cache_strings=True)¶: The pydantic-core SchemaValidator used to validate instances of the model.

__signature__: ClassVar[Signature] = <Signature (_case_sensitive: 'bool | None' = None, _nested_model_default_partial_update: 'bool | None' = None, _env_prefix: 'str | None' = None, _env_file: 'DotenvType | None' = PosixPath('.'), _env_file_encoding: 'str | None' = None, _env_ignore_empty: 'bool | None' = None, _env_nested_delimiter: 'str | None' = None, _env_parse_none_str: 'str | None' = None, _env_parse_enums: 'bool | None' = None, _cli_prog_name: 'str | None' = None, _cli_parse_args: 'bool | list[str] | tuple[str, ...] | None' = None, _cli_settings_source: 'CliSettingsSource[Any] | None' = None, _cli_parse_none_str: 'str | None' = None, _cli_hide_none_type: 'bool | None' = None, _cli_avoid_json: 'bool | None' = None, _cli_enforce_required: 'bool | None' = None, _cli_use_class_docs_for_groups: 'bool | None' = None, _cli_exit_on_error: 'bool | None' = None, _cli_prefix: 'str | None' = None, _cli_implicit_flags: 'bool | None' = None, _secrets_dir: 'PathType | None' = None, *, gt4sd_local_cache_path: str = '/home/runner/.gt4sd', gt4sd_local_cache_path_algorithms: str = 'algorithms', gt4sd_local_cache_path_properties: str = 'properties', gt4sd_max_number_of_stuck_calls: int = 50, gt4sd_max_number_of_samples: int = 1000000, gt4sd_max_runtime: int = 86400, gt4sd_create_unverified_ssl_context: bool = False, gt4sd_disable_cudnn: bool = False, gt4sd_s3_host: str = 's3.par01.cloud-object-storage.appdomain.cloud', gt4sd_s3_access_key: str = '6e9891531d724da89997575a65f4592e', gt4sd_s3_secret_key: str = '5997d63c4002cc04e13c03dc0c2db9dae751293dab106ac5', gt4sd_s3_secure: bool = True, gt4sd_s3_bucket_algorithms: str = 'gt4sd-cos-algorithms-artifacts', gt4sd_s3_bucket_properties: str = 'gt4sd-cos-properties-artifacts', gt4sd_s3_host_hub: str = 's3.par01.cloud-object-storage.appdomain.cloud', gt4sd_s3_access_key_hub: str = 'd9536662ebcf462f937efb9f58012830', gt4sd_s3_secret_key_hub: str = '934d1f3afdaea55ac586f6c2f729ac2ba2694bb8e975ee0b', gt4sd_s3_secure_hub: bool = True, gt4sd_s3_bucket_hub_algorithms: str = 'gt4sd-cos-hub-algorithms-artifacts', gt4sd_s3_bucket_hub_properties: str = 'gt4sd-cos-hub-properties-artifacts') -> None>¶: The synthesized __init__ [Signature][inspect.Signature] of the model.

_abc_impl = <_abc._abc_data object>¶

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}¶: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_fields: ClassVar[Dict[str, FieldInfo]] = {'gt4sd_create_unverified_ssl_context': FieldInfo(annotation=bool, required=False, default=False), 'gt4sd_disable_cudnn': FieldInfo(annotation=bool, required=False, default=False), 'gt4sd_local_cache_path': FieldInfo(annotation=str, required=False, default='/home/runner/.gt4sd'), 'gt4sd_local_cache_path_algorithms': FieldInfo(annotation=str, required=False, default='algorithms'), 'gt4sd_local_cache_path_properties': FieldInfo(annotation=str, required=False, default='properties'), 'gt4sd_max_number_of_samples': FieldInfo(annotation=int, required=False, default=1000000), 'gt4sd_max_number_of_stuck_calls': FieldInfo(annotation=int, required=False, default=50), 'gt4sd_max_runtime': FieldInfo(annotation=int, required=False, default=86400), 'gt4sd_s3_access_key': FieldInfo(annotation=str, required=False, default='6e9891531d724da89997575a65f4592e'), 'gt4sd_s3_access_key_hub': FieldInfo(annotation=str, required=False, default='d9536662ebcf462f937efb9f58012830'), 'gt4sd_s3_bucket_algorithms': FieldInfo(annotation=str, required=False, default='gt4sd-cos-algorithms-artifacts'), 'gt4sd_s3_bucket_hub_algorithms': FieldInfo(annotation=str, required=False, default='gt4sd-cos-hub-algorithms-artifacts'), 'gt4sd_s3_bucket_hub_properties': FieldInfo(annotation=str, required=False, default='gt4sd-cos-hub-properties-artifacts'), 'gt4sd_s3_bucket_properties': FieldInfo(annotation=str, required=False, default='gt4sd-cos-properties-artifacts'), 'gt4sd_s3_host': FieldInfo(annotation=str, required=False, default='s3.par01.cloud-object-storage.appdomain.cloud'), 'gt4sd_s3_host_hub': FieldInfo(annotation=str, required=False, default='s3.par01.cloud-object-storage.appdomain.cloud'), 'gt4sd_s3_secret_key': FieldInfo(annotation=str, required=False, default='5997d63c4002cc04e13c03dc0c2db9dae751293dab106ac5'), 'gt4sd_s3_secret_key_hub': FieldInfo(annotation=str, required=False, default='934d1f3afdaea55ac586f6c2f729ac2ba2694bb8e975ee0b'), 'gt4sd_s3_secure': FieldInfo(annotation=bool, required=False, default=True), 'gt4sd_s3_secure_hub': FieldInfo(annotation=bool, required=False, default=True)}¶

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class GT4SDArtifactManagementConfiguration(gt4sd_configuration)[source]¶

Bases: object

Artifact management configuration.

gt4sd_s3_modules: Set[str] = {'algorithms', 'properties'}¶

__init__(gt4sd_configuration)[source]¶

Initialize the artifact management configuration from the base one.

Parameters: gt4sd_configuration (GT4SDConfiguration) – GT4SD base configuration.

__annotations__ = {'gt4sd_s3_modules': typing.Set[str], 'local_cache_path': 'Dict[str, str]', 's3_bucket': 'Dict[str, str]', 's3_bucket_hub': 'Dict[str, str]'}¶

__dict__ = mappingproxy({'__module__': 'gt4sd.configuration', '__annotations__': {'gt4sd_s3_modules': typing.Set[str], 'local_cache_path': 'Dict[str, str]', 's3_bucket': 'Dict[str, str]', 's3_bucket_hub': 'Dict[str, str]'}, '__doc__': 'Artifact management configuration.', 'gt4sd_s3_modules': {'properties', 'algorithms'}, '__init__': <function GT4SDArtifactManagementConfiguration.__init__>, '__dict__': <attribute '__dict__' of 'GT4SDArtifactManagementConfiguration' objects>, '__weakref__': <attribute '__weakref__' of 'GT4SDArtifactManagementConfiguration' objects>})¶

__doc__ = 'Artifact management configuration.'¶

__module__ = 'gt4sd.configuration'¶

__weakref__¶: list of weak references to the object (if defined)

upload_to_s3(target_filepath, source_filepath, module='algorithms')[source]¶: Upload an algorithm in source_filepath in target_filepath on a bucket in the model hub. :type target_filepath: str :param target_filepath: path to save the objects in s3. :type source_filepath: str :param source_filepath: path to the file to sync. :type module: str :param module: the submodule of gt4sd that acts as a root for the bucket, defaults

to algorithms.

sync_algorithm_with_s3(prefix=None, module='algorithms')[source]¶

Sync an algorithm in the local cache using environment variables.

Parameters

prefix (Optional[str, None]) – the relative path in the bucket (both on S3 and locally) to match files to download. Defaults to None.
module (str) – the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Returns

local path using the prefix.

Return type

str

get_cached_algorithm_path(prefix=None, module='algorithms')[source]¶

Return type: str

get_algorithm_subdirectories_from_s3_coordinates(host, access_key, secret_key, bucket, secure=True, prefix=None)[source]¶

Wrapper to initialize a client and list the directories in a bucket.

Return type: Set[str]

get_algorithm_subdirectories_with_s3(prefix=None, module='algorithms')[source]¶

Get algorithms in the s3 buckets.

Parameters

prefix (Optional[str, None]) – the relative path in the bucket (both on S3 and locally) to match files to download. Defaults to None.
module (str) – the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Returns

set of available algorithms on s3 with that prefix.

Return type

Set

get_algorithm_subdirectories_in_cache(prefix=None, module='algorithms')[source]¶

Get algorithm subdirectories from the cache.

Parameters

prefix (Optional[str, None]) – prefix matching cache subdirectories. Defaults to None.
module (str) – the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Return type

Set[str]

Returns

a set of subdirectories.

reset_logging_root_logger()[source]¶: Reset the root logger from logging library.