gt4sd.configuration module¶

Module configuration.

Summary¶

Classes:

`GT4SDArtifactManagementConfiguration`	Artifact management configuration.
`GT4SDConfiguration`	GT4SDConfiguration settings from environment variables.

Functions:

`get_algorithm_subdirectories_from_s3_coordinates`	Wrapper to initialize a client and list the directories in a bucket.
`get_algorithm_subdirectories_in_cache`	Get algorithm subdirectories from the cache.
`get_algorithm_subdirectories_with_s3`	Get algorithms in the s3 buckets.
`get_cached_algorithm_path`	rtype `str`
`reset_logging_root_logger`	Reset the root logger from logging library.
`sync_algorithm_with_s3`	Sync an algorithm in the local cache using environment variables.
`upload_to_s3`	Upload an algorithm in source_filepath in target_filepath on a bucket in the model hub. :type target_filepath: `str` :param target_filepath: path to save the objects in s3. :type source_filepath: `str` :param source_filepath: path to the file to sync. :type module: `str` :param module: the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Reference¶

class GT4SDConfiguration(_case_sensitive=None, _nested_model_default_partial_update=None, _env_prefix=None, _env_file=PosixPath('.'), _env_file_encoding=None, _env_ignore_empty=None, _env_nested_delimiter=None, _env_parse_none_str=None, _env_parse_enums=None, _cli_prog_name=None, _cli_parse_args=None, _cli_settings_source=None, _cli_parse_none_str=None, _cli_hide_none_type=None, _cli_avoid_json=None, _cli_enforce_required=None, _cli_use_class_docs_for_groups=None, _cli_exit_on_error=None, _cli_prefix=None, _cli_flag_prefix_char=None, _cli_implicit_flags=None, _cli_ignore_unknown_args=None, _cli_kebab_case=None, _secrets_dir=None, **values)[source]¶

Bases: BaseSettings

GT4SDConfiguration settings from environment variables.

Default configurations for gt4sd including a read-only COS for algorithms’ artifacts. Default configuration for gt4sd hub including a read-write COS for algorithms’ artifacts uploaded by users.

gt4sd_local_cache_path: str¶

gt4sd_local_cache_path_algorithms: str¶

gt4sd_local_cache_path_properties: str¶

gt4sd_max_number_of_stuck_calls: int¶

gt4sd_max_number_of_samples: int¶

gt4sd_max_runtime: int¶

gt4sd_create_unverified_ssl_context: bool¶

gt4sd_disable_cudnn: bool¶

gt4sd_s3_host: str¶

gt4sd_s3_access_key: str¶

gt4sd_s3_secret_key: str¶

gt4sd_s3_secure: bool¶

gt4sd_s3_bucket_algorithms: str¶

gt4sd_s3_bucket_properties: str¶

gt4sd_s3_host_hub: str¶

gt4sd_s3_access_key_hub: str¶

gt4sd_s3_secret_key_hub: str¶

gt4sd_s3_secure_hub: bool¶

gt4sd_s3_bucket_hub_algorithms: str¶

gt4sd_s3_bucket_hub_properties: str¶

model_config: ClassVar[SettingsConfigDict] = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': '', 'extra': 'forbid', 'frozen': True, 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_file': None, 'yaml_file_encoding': None}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

static get_instance()[source]¶

Return type: GT4SDConfiguration

__dict__¶

__pydantic_fields_set__: set[str]¶: The names of fields explicitly set during instantiation.

__pydantic_extra__: dict[str, Any] | None¶: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_private__: dict[str, Any] | None¶: Values of private attributes set on the model instance.

__abstractmethods__ = frozenset({})¶

__annotations__ = {'__class_vars__': 'ClassVar[set[str]]', '__private_attributes__': 'ClassVar[Dict[str, ModelPrivateAttr]]', '__pydantic_complete__': 'ClassVar[bool]', '__pydantic_computed_fields__': 'ClassVar[Dict[str, ComputedFieldInfo]]', '__pydantic_core_schema__': 'ClassVar[CoreSchema]', '__pydantic_custom_init__': 'ClassVar[bool]', '__pydantic_decorators__': 'ClassVar[_decorators.DecoratorInfos]', '__pydantic_extra__': 'dict[str, Any] | None', '__pydantic_fields__': 'ClassVar[Dict[str, FieldInfo]]', '__pydantic_fields_set__': 'set[str]', '__pydantic_generic_metadata__': 'ClassVar[_generics.PydanticGenericMetadata]', '__pydantic_parent_namespace__': 'ClassVar[Dict[str, Any] | None]', '__pydantic_post_init__': "ClassVar[None | Literal['model_post_init']]", '__pydantic_private__': 'dict[str, Any] | None', '__pydantic_root_model__': 'ClassVar[bool]', '__pydantic_serializer__': 'ClassVar[SchemaSerializer]', '__pydantic_validator__': 'ClassVar[SchemaValidator | PluggableSchemaValidator]', '__signature__': 'ClassVar[Signature]', 'gt4sd_create_unverified_ssl_context': <class 'bool'>, 'gt4sd_disable_cudnn': <class 'bool'>, 'gt4sd_local_cache_path': <class 'str'>, 'gt4sd_local_cache_path_algorithms': <class 'str'>, 'gt4sd_local_cache_path_properties': <class 'str'>, 'gt4sd_max_number_of_samples': <class 'int'>, 'gt4sd_max_number_of_stuck_calls': <class 'int'>, 'gt4sd_max_runtime': <class 'int'>, 'gt4sd_s3_access_key': <class 'str'>, 'gt4sd_s3_access_key_hub': <class 'str'>, 'gt4sd_s3_bucket_algorithms': <class 'str'>, 'gt4sd_s3_bucket_hub_algorithms': <class 'str'>, 'gt4sd_s3_bucket_hub_properties': <class 'str'>, 'gt4sd_s3_bucket_properties': <class 'str'>, 'gt4sd_s3_host': <class 'str'>, 'gt4sd_s3_host_hub': <class 'str'>, 'gt4sd_s3_secret_key': <class 'str'>, 'gt4sd_s3_secret_key_hub': <class 'str'>, 'gt4sd_s3_secure': <class 'bool'>, 'gt4sd_s3_secure_hub': <class 'bool'>, 'model_computed_fields': 'ClassVar[dict[str, ComputedFieldInfo]]', 'model_config': 'ClassVar[SettingsConfigDict]', 'model_fields': 'ClassVar[dict[str, FieldInfo]]'}¶

__class_vars__: ClassVar[set[str]] = {}¶: The names of the class variables defined on the model.

__doc__ = "GT4SDConfiguration settings from environment variables.\n\n Default configurations for gt4sd including a read-only COS for algorithms' artifacts.\n Default configuration for gt4sd hub including a read-write COS for algorithms' artifacts uploaded by users.\n "¶

__hash__() → int¶

Return hash(self).

Return type: int

__module__ = 'gt4sd.configuration'¶

__private_attributes__: ClassVar[Dict[str, ModelPrivateAttr]] = {}¶: Metadata about the private attributes of the model.

__pydantic_complete__: ClassVar[bool] = True¶: Whether model building is completed, or if there are still undefined fields.

__pydantic_computed_fields__: ClassVar[Dict[str, ComputedFieldInfo]] = {}¶: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_core_schema__: ClassVar[CoreSchema] = {'cls': <class 'gt4sd.configuration.GT4SDConfiguration'>, 'config': {'extra_fields_behavior': 'forbid', 'title': 'GT4SDConfiguration', 'validate_default': True}, 'custom_init': True, 'metadata': {'pydantic_js_functions': [<bound method BaseModel.__get_pydantic_json_schema__ of <class 'gt4sd.configuration.GT4SDConfiguration'>>]}, 'ref': 'gt4sd.configuration.GT4SDConfiguration:94427755927392', 'root_model': False, 'schema': {'computed_fields': [], 'fields': {'gt4sd_create_unverified_ssl_context': {'metadata': {}, 'schema': {'default': False, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_disable_cudnn': {'metadata': {}, 'schema': {'default': False, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_local_cache_path': {'metadata': {}, 'schema': {'default': '/home/runner/.gt4sd', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_local_cache_path_algorithms': {'metadata': {}, 'schema': {'default': 'algorithms', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_local_cache_path_properties': {'metadata': {}, 'schema': {'default': 'properties', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_max_number_of_samples': {'metadata': {}, 'schema': {'default': 1000000, 'schema': {'type': 'int'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_max_number_of_stuck_calls': {'metadata': {}, 'schema': {'default': 50, 'schema': {'type': 'int'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_max_runtime': {'metadata': {}, 'schema': {'default': 86400, 'schema': {'type': 'int'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_access_key': {'metadata': {}, 'schema': {'default': 'b087e6810a5d4246a64e07e36ace338f', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_access_key_hub': {'metadata': {}, 'schema': {'default': '1168c1d9ce664e75a8a151e6f4a29078', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_algorithms': {'metadata': {}, 'schema': {'default': 'gt4sd-cos-algorithms-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_hub_algorithms': {'metadata': {}, 'schema': {'default': 'gt4sd-cos-hub-algorithms-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_hub_properties': {'metadata': {}, 'schema': {'default': 'gt4sd-cos-hub-properties-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_bucket_properties': {'metadata': {}, 'schema': {'default': 'gt4sd-cos-properties-artifacts', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_host': {'metadata': {}, 'schema': {'default': 's3.par01.cloud-object-storage.appdomain.cloud', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_host_hub': {'metadata': {}, 'schema': {'default': 's3.par01.cloud-object-storage.appdomain.cloud', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secret_key': {'metadata': {}, 'schema': {'default': 'ba4a1db5647a32c6109b58714befb7ea7145b983143e0836', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secret_key_hub': {'metadata': {}, 'schema': {'default': '4996c6cc737828213a7afcc7e27450e1af2daf027af95c1d', 'schema': {'type': 'str'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secure': {'metadata': {}, 'schema': {'default': True, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}, 'gt4sd_s3_secure_hub': {'metadata': {}, 'schema': {'default': True, 'schema': {'type': 'bool'}, 'type': 'default'}, 'type': 'model-field'}}, 'model_name': 'GT4SDConfiguration', 'type': 'model-fields'}, 'type': 'model'}¶: The core schema of the model.

__pydantic_custom_init__: ClassVar[bool] = True¶: Whether the model has a custom __init__ method.

__pydantic_decorators__: ClassVar[_decorators.DecoratorInfos] = DecoratorInfos(validators={}, field_validators={}, root_validators={}, field_serializers={}, model_serializers={}, model_validators={}, computed_fields={})¶: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_fields__: ClassVar[Dict[str, FieldInfo]] = {'gt4sd_create_unverified_ssl_context': FieldInfo(annotation=bool, required=False, default=False), 'gt4sd_disable_cudnn': FieldInfo(annotation=bool, required=False, default=False), 'gt4sd_local_cache_path': FieldInfo(annotation=str, required=False, default='/home/runner/.gt4sd'), 'gt4sd_local_cache_path_algorithms': FieldInfo(annotation=str, required=False, default='algorithms'), 'gt4sd_local_cache_path_properties': FieldInfo(annotation=str, required=False, default='properties'), 'gt4sd_max_number_of_samples': FieldInfo(annotation=int, required=False, default=1000000), 'gt4sd_max_number_of_stuck_calls': FieldInfo(annotation=int, required=False, default=50), 'gt4sd_max_runtime': FieldInfo(annotation=int, required=False, default=86400), 'gt4sd_s3_access_key': FieldInfo(annotation=str, required=False, default='b087e6810a5d4246a64e07e36ace338f'), 'gt4sd_s3_access_key_hub': FieldInfo(annotation=str, required=False, default='1168c1d9ce664e75a8a151e6f4a29078'), 'gt4sd_s3_bucket_algorithms': FieldInfo(annotation=str, required=False, default='gt4sd-cos-algorithms-artifacts'), 'gt4sd_s3_bucket_hub_algorithms': FieldInfo(annotation=str, required=False, default='gt4sd-cos-hub-algorithms-artifacts'), 'gt4sd_s3_bucket_hub_properties': FieldInfo(annotation=str, required=False, default='gt4sd-cos-hub-properties-artifacts'), 'gt4sd_s3_bucket_properties': FieldInfo(annotation=str, required=False, default='gt4sd-cos-properties-artifacts'), 'gt4sd_s3_host': FieldInfo(annotation=str, required=False, default='s3.par01.cloud-object-storage.appdomain.cloud'), 'gt4sd_s3_host_hub': FieldInfo(annotation=str, required=False, default='s3.par01.cloud-object-storage.appdomain.cloud'), 'gt4sd_s3_secret_key': FieldInfo(annotation=str, required=False, default='ba4a1db5647a32c6109b58714befb7ea7145b983143e0836'), 'gt4sd_s3_secret_key_hub': FieldInfo(annotation=str, required=False, default='4996c6cc737828213a7afcc7e27450e1af2daf027af95c1d'), 'gt4sd_s3_secure': FieldInfo(annotation=bool, required=False, default=True), 'gt4sd_s3_secure_hub': FieldInfo(annotation=bool, required=False, default=True)}¶: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects. This replaces Model.__fields__ from Pydantic V1.

__pydantic_generic_metadata__: ClassVar[_generics.PydanticGenericMetadata] = {'args': (), 'origin': None, 'parameters': ()}¶: Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: ClassVar[Dict[str, Any] | None] = None¶: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: ClassVar[None | Literal['model_post_init']] = None¶: The name of the post-init method for the model, if defined.

__pydantic_serializer__: ClassVar[SchemaSerializer] = SchemaSerializer(serializer=Model( ModelSerializer { class: Py( 0x000055e1acd95760, ), serializer: Fields( GeneralFieldsSerializer { fields: { "gt4sd_s3_access_key_hub": SerField { key_py: Py( 0x00007f9e9a42b500, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a415f50, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secret_key_hub": SerField { key_py: Py( 0x00007f9e9a42b550, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a5ee020, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_hub_properties": SerField { key_py: Py( 0x00007f9e9a42b640, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a416010, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_access_key": SerField { key_py: Py( 0x00007f9e9a42b2d0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a415ef0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_host": SerField { key_py: Py( 0x00007f9e9a433430, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a415e90, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secure": SerField { key_py: Py( 0x00007f9e9a433470, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000055e1a1fe15a0, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secure_hub": SerField { key_py: Py( 0x00007f9e9a42b5a0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000055e1a1fe15a0, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_local_cache_path_properties": SerField { key_py: Py( 0x00007f9e9a415e30, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9cb011f0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_hub_algorithms": SerField { key_py: Py( 0x00007f9e9a42b5f0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a415fb0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_s3_secret_key": SerField { key_py: Py( 0x00007f9e9a42b320, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a5edfb0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_local_cache_path_algorithms": SerField { key_py: Py( 0x00007f9e9a415dd0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9b1fd8f0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_max_runtime": SerField { key_py: Py( 0x00007f9e9a42b230, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a599610, ), ), serializer: Int( IntSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_algorithms": SerField { key_py: Py( 0x00007f9e9a42b3c0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a42b370, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_max_number_of_stuck_calls": SerField { key_py: Py( 0x00007f9e9a42b190, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9d500710, ), ), serializer: Int( IntSerializer, ), }, ), ), required: true, }, "gt4sd_s3_host_hub": SerField { key_py: Py( 0x00007f9e9a42b4b0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a415e90, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_max_number_of_samples": SerField { key_py: Py( 0x00007f9e9a42b1e0, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a599690, ), ), serializer: Int( IntSerializer, ), }, ), ), required: true, }, "gt4sd_create_unverified_ssl_context": SerField { key_py: Py( 0x00007f9e9a415d70, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000055e1a1fe1580, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_local_cache_path": SerField { key_py: Py( 0x00007f9e9a42b140, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e998753e0, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, "gt4sd_disable_cudnn": SerField { key_py: Py( 0x00007f9e9a42b280, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x000055e1a1fe1580, ), ), serializer: Bool( BoolSerializer, ), }, ), ), required: true, }, "gt4sd_s3_bucket_properties": SerField { key_py: Py( 0x00007f9e9a42b460, ), alias: None, alias_py: None, serializer: Some( WithDefault( WithDefaultSerializer { default: Default( Py( 0x00007f9e9a42b410, ), ), serializer: Str( StrSerializer, ), }, ), ), required: true, }, }, computed_fields: Some( ComputedFields( [], ), ), mode: SimpleDict, extra_serializer: None, filter: SchemaFilter { include: None, exclude: None, }, required_fields: 20, }, ), has_extra: false, root_model: false, name: "GT4SDConfiguration", }, ), definitions=[])¶: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: ClassVar[SchemaValidator | PluggableSchemaValidator] = SchemaValidator(title="GT4SDConfiguration", validator=Model( ModelValidator { revalidate: Never, validator: ModelFields( ModelFieldsValidator { fields: [ Field { name: "gt4sd_local_cache_path", lookup_key: Simple { key: "gt4sd_local_cache_path", py_key: Py( 0x00007f9e9ba02650, ), path: LookupPath( [ S( "gt4sd_local_cache_path", Py( 0x00007f9e99875570, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b140, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e998753e0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_local_cache_path_algorithms", lookup_key: Simple { key: "gt4sd_local_cache_path_algorithms", py_key: Py( 0x00007f9e9986c270, ), path: LookupPath( [ S( "gt4sd_local_cache_path_algorithms", Py( 0x00007f9e9986c2d0, ), ), ], ), }, name_py: Py( 0x00007f9e9a415dd0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9b1fd8f0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_local_cache_path_properties", lookup_key: Simple { key: "gt4sd_local_cache_path_properties", py_key: Py( 0x00007f9e9986c330, ), path: LookupPath( [ S( "gt4sd_local_cache_path_properties", Py( 0x00007f9e9986c390, ), ), ], ), }, name_py: Py( 0x00007f9e9a415e30, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9cb011f0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_max_number_of_stuck_calls", lookup_key: Simple { key: "gt4sd_max_number_of_stuck_calls", py_key: Py( 0x00007f9e998760b0, ), path: LookupPath( [ S( "gt4sd_max_number_of_stuck_calls", Py( 0x00007f9e99876060, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b190, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9d500710, ), ), on_error: Raise, validator: Int( IntValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[int]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_max_number_of_samples", lookup_key: Simple { key: "gt4sd_max_number_of_samples", py_key: Py( 0x00007f9e99876010, ), path: LookupPath( [ S( "gt4sd_max_number_of_samples", Py( 0x00007f9e99876100, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b1e0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a599690, ), ), on_error: Raise, validator: Int( IntValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[int]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_max_runtime", lookup_key: Simple { key: "gt4sd_max_runtime", py_key: Py( 0x00007f9e99876150, ), path: LookupPath( [ S( "gt4sd_max_runtime", Py( 0x00007f9e998761a0, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b230, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a599610, ), ), on_error: Raise, validator: Int( IntValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[int]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_create_unverified_ssl_context", lookup_key: Simple { key: "gt4sd_create_unverified_ssl_context", py_key: Py( 0x00007f9e9986c3f0, ), path: LookupPath( [ S( "gt4sd_create_unverified_ssl_context", Py( 0x00007f9e9986c450, ), ), ], ), }, name_py: Py( 0x00007f9e9a415d70, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000055e1a1fe1580, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_disable_cudnn", lookup_key: Simple { key: "gt4sd_disable_cudnn", py_key: Py( 0x00007f9e998761f0, ), path: LookupPath( [ S( "gt4sd_disable_cudnn", Py( 0x00007f9e99876240, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b280, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000055e1a1fe1580, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_host", lookup_key: Simple { key: "gt4sd_s3_host", py_key: Py( 0x00007f9e9985c5b0, ), path: LookupPath( [ S( "gt4sd_s3_host", Py( 0x00007f9e9985c570, ), ), ], ), }, name_py: Py( 0x00007f9e9a433430, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a415e90, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_access_key", lookup_key: Simple { key: "gt4sd_s3_access_key", py_key: Py( 0x00007f9e99876290, ), path: LookupPath( [ S( "gt4sd_s3_access_key", Py( 0x00007f9e998762e0, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b2d0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a415ef0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secret_key", lookup_key: Simple { key: "gt4sd_s3_secret_key", py_key: Py( 0x00007f9e99876330, ), path: LookupPath( [ S( "gt4sd_s3_secret_key", Py( 0x00007f9e99876380, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b320, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a5edfb0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secure", lookup_key: Simple { key: "gt4sd_s3_secure", py_key: Py( 0x00007f9e9985c530, ), path: LookupPath( [ S( "gt4sd_s3_secure", Py( 0x00007f9e9985c4f0, ), ), ], ), }, name_py: Py( 0x00007f9e9a433470, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000055e1a1fe15a0, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_algorithms", lookup_key: Simple { key: "gt4sd_s3_bucket_algorithms", py_key: Py( 0x00007f9e998763d0, ), path: LookupPath( [ S( "gt4sd_s3_bucket_algorithms", Py( 0x00007f9e99876420, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b3c0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a42b370, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_properties", lookup_key: Simple { key: "gt4sd_s3_bucket_properties", py_key: Py( 0x00007f9e99876470, ), path: LookupPath( [ S( "gt4sd_s3_bucket_properties", Py( 0x00007f9e998764c0, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b460, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a42b410, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_host_hub", lookup_key: Simple { key: "gt4sd_s3_host_hub", py_key: Py( 0x00007f9e99876510, ), path: LookupPath( [ S( "gt4sd_s3_host_hub", Py( 0x00007f9e99876560, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b4b0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a415e90, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_access_key_hub", lookup_key: Simple { key: "gt4sd_s3_access_key_hub", py_key: Py( 0x00007f9e998765b0, ), path: LookupPath( [ S( "gt4sd_s3_access_key_hub", Py( 0x00007f9e99876600, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b500, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a415f50, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secret_key_hub", lookup_key: Simple { key: "gt4sd_s3_secret_key_hub", py_key: Py( 0x00007f9e99876650, ), path: LookupPath( [ S( "gt4sd_s3_secret_key_hub", Py( 0x00007f9e998766a0, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b550, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a5ee020, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_secure_hub", lookup_key: Simple { key: "gt4sd_s3_secure_hub", py_key: Py( 0x00007f9e998766f0, ), path: LookupPath( [ S( "gt4sd_s3_secure_hub", Py( 0x00007f9e99876740, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b5a0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x000055e1a1fe15a0, ), ), on_error: Raise, validator: Bool( BoolValidator { strict: false, }, ), validate_default: true, copy_default: false, name: "default[bool]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_hub_algorithms", lookup_key: Simple { key: "gt4sd_s3_bucket_hub_algorithms", py_key: Py( 0x00007f9e99876790, ), path: LookupPath( [ S( "gt4sd_s3_bucket_hub_algorithms", Py( 0x00007f9e998767e0, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b5f0, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a415fb0, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, Field { name: "gt4sd_s3_bucket_hub_properties", lookup_key: Simple { key: "gt4sd_s3_bucket_hub_properties", py_key: Py( 0x00007f9e99876830, ), path: LookupPath( [ S( "gt4sd_s3_bucket_hub_properties", Py( 0x00007f9e99876880, ), ), ], ), }, name_py: Py( 0x00007f9e9a42b640, ), validator: WithDefault( WithDefaultValidator { default: Default( Py( 0x00007f9e9a416010, ), ), on_error: Raise, validator: Str( StrValidator { strict: false, coerce_numbers_to_str: false, }, ), validate_default: true, copy_default: false, name: "default[str]", undefined: Py( 0x00007f9e9b5139a0, ), }, ), frozen: false, }, ], model_name: "GT4SDConfiguration", extra_behavior: Forbid, extras_validator: None, strict: false, from_attributes: false, loc_by_alias: true, }, ), class: Py( 0x000055e1acd95760, ), generic_origin: None, post_init: None, frozen: false, custom_init: true, root_model: false, undefined: Py( 0x00007f9e9b5139a0, ), name: "GT4SDConfiguration", }, ), definitions=[], cache_strings=True)¶: The pydantic-core SchemaValidator used to validate instances of the model.

__signature__: ClassVar[Signature] = <Signature (_case_sensitive: 'bool | None' = None, _nested_model_default_partial_update: 'bool | None' = None, _env_prefix: 'str | None' = None, _env_file: 'DotenvType | None' = PosixPath('.'), _env_file_encoding: 'str | None' = None, _env_ignore_empty: 'bool | None' = None, _env_nested_delimiter: 'str | None' = None, _env_parse_none_str: 'str | None' = None, _env_parse_enums: 'bool | None' = None, _cli_prog_name: 'str | None' = None, _cli_parse_args: 'bool | list[str] | tuple[str, ...] | None' = None, _cli_settings_source: 'CliSettingsSource[Any] | None' = None, _cli_parse_none_str: 'str | None' = None, _cli_hide_none_type: 'bool | None' = None, _cli_avoid_json: 'bool | None' = None, _cli_enforce_required: 'bool | None' = None, _cli_use_class_docs_for_groups: 'bool | None' = None, _cli_exit_on_error: 'bool | None' = None, _cli_prefix: 'str | None' = None, _cli_flag_prefix_char: 'str | None' = None, _cli_implicit_flags: 'bool | None' = None, _cli_ignore_unknown_args: 'bool | None' = None, _cli_kebab_case: 'bool | None' = None, _secrets_dir: 'PathType | None' = None, *, gt4sd_local_cache_path: str = '/home/runner/.gt4sd', gt4sd_local_cache_path_algorithms: str = 'algorithms', gt4sd_local_cache_path_properties: str = 'properties', gt4sd_max_number_of_stuck_calls: int = 50, gt4sd_max_number_of_samples: int = 1000000, gt4sd_max_runtime: int = 86400, gt4sd_create_unverified_ssl_context: bool = False, gt4sd_disable_cudnn: bool = False, gt4sd_s3_host: str = 's3.par01.cloud-object-storage.appdomain.cloud', gt4sd_s3_access_key: str = 'b087e6810a5d4246a64e07e36ace338f', gt4sd_s3_secret_key: str = 'ba4a1db5647a32c6109b58714befb7ea7145b983143e0836', gt4sd_s3_secure: bool = True, gt4sd_s3_bucket_algorithms: str = 'gt4sd-cos-algorithms-artifacts', gt4sd_s3_bucket_properties: str = 'gt4sd-cos-properties-artifacts', gt4sd_s3_host_hub: str = 's3.par01.cloud-object-storage.appdomain.cloud', gt4sd_s3_access_key_hub: str = '1168c1d9ce664e75a8a151e6f4a29078', gt4sd_s3_secret_key_hub: str = '4996c6cc737828213a7afcc7e27450e1af2daf027af95c1d', gt4sd_s3_secure_hub: bool = True, gt4sd_s3_bucket_hub_algorithms: str = 'gt4sd-cos-hub-algorithms-artifacts', gt4sd_s3_bucket_hub_properties: str = 'gt4sd-cos-hub-properties-artifacts') -> None>¶: The synthesized __init__ [Signature][inspect.Signature] of the model.

_abc_impl = <_abc._abc_data object>¶

class GT4SDArtifactManagementConfiguration(gt4sd_configuration)[source]¶

Bases: object

Artifact management configuration.

gt4sd_s3_modules: Set[str] = {'algorithms', 'properties'}¶

__init__(gt4sd_configuration)[source]¶

Initialize the artifact management configuration from the base one.

Parameters: gt4sd_configuration (GT4SDConfiguration) – GT4SD base configuration.

__annotations__ = {'gt4sd_s3_modules': typing.Set[str], 'local_cache_path': 'Dict[str, str]', 's3_bucket': 'Dict[str, str]', 's3_bucket_hub': 'Dict[str, str]'}¶

__dict__ = mappingproxy({'__module__': 'gt4sd.configuration', '__annotations__': {'gt4sd_s3_modules': typing.Set[str], 'local_cache_path': 'Dict[str, str]', 's3_bucket': 'Dict[str, str]', 's3_bucket_hub': 'Dict[str, str]'}, '__doc__': 'Artifact management configuration.', 'gt4sd_s3_modules': {'algorithms', 'properties'}, '__init__': <function GT4SDArtifactManagementConfiguration.__init__>, '__dict__': <attribute '__dict__' of 'GT4SDArtifactManagementConfiguration' objects>, '__weakref__': <attribute '__weakref__' of 'GT4SDArtifactManagementConfiguration' objects>})¶

__doc__ = 'Artifact management configuration.'¶

__module__ = 'gt4sd.configuration'¶

__weakref__¶: list of weak references to the object (if defined)

upload_to_s3(target_filepath, source_filepath, module='algorithms')[source]¶: Upload an algorithm in source_filepath in target_filepath on a bucket in the model hub. :type target_filepath: str :param target_filepath: path to save the objects in s3. :type source_filepath: str :param source_filepath: path to the file to sync. :type module: str :param module: the submodule of gt4sd that acts as a root for the bucket, defaults

to algorithms.

sync_algorithm_with_s3(prefix=None, module='algorithms')[source]¶

Sync an algorithm in the local cache using environment variables.

Parameters

prefix (Optional[str, None]) – the relative path in the bucket (both on S3 and locally) to match files to download. Defaults to None.
module (str) – the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Returns

local path using the prefix.

Return type

str

get_cached_algorithm_path(prefix=None, module='algorithms')[source]¶

Return type: str

get_algorithm_subdirectories_from_s3_coordinates(host, access_key, secret_key, bucket, secure=True, prefix=None)[source]¶

Wrapper to initialize a client and list the directories in a bucket.

Return type: Set[str]

get_algorithm_subdirectories_with_s3(prefix=None, module='algorithms')[source]¶

Get algorithms in the s3 buckets.

Parameters

prefix (Optional[str, None]) – the relative path in the bucket (both on S3 and locally) to match files to download. Defaults to None.
module (str) – the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Returns

set of available algorithms on s3 with that prefix.

Return type

Set

get_algorithm_subdirectories_in_cache(prefix=None, module='algorithms')[source]¶

Get algorithm subdirectories from the cache.

Parameters

prefix (Optional[str, None]) – prefix matching cache subdirectories. Defaults to None.
module (str) – the submodule of gt4sd that acts as a root for the bucket, defaults to algorithms.

Return type

Set[str]

Returns

a set of subdirectories.

reset_logging_root_logger()[source]¶: Reset the root logger from logging library.