GT4SD server upload

Here we report an example of how you can setup a custom minio server on localhost where you can upload your algorithms. Keep in mind that the same procedure can be used with a pre-existing COS simply setting the environment variables to the appropriate values.

---

Requirements

• docker

• minio

• gt4sd requirements

---

Run a local minio server

1) Set environment variables

export GT4SD_S3_SECRET_KEY_HUB=''
export GT4SD_S3_ACCESS_KEY_HUB=''
export GT4SD_S3_HOST_HUB='127.0.0.1:9000'
export GT4SD_S3_SECURE_HUB=False
export GT4SD_S3_BUCKET_HUB='gt4sd-cos-algorithms-artifacts'
export GT4SD_S3_BUCKET_MODELS_HUB='gt4sd-cos-algorithms-models'
export GT4SD_S3_BUCKET_DATA_HUB='gt4sd-cos-algorithms-data'

set GT4SD_S3_SECURE True or False if https/http server.

2) Create a docker container with a minio server

cd ~/
mkdir localhost-server
cd localhost-server
mkdir env/
echo >> docker-compose.yml

copy this configuration script in docker-compose.yml:

version: '10'
services:
  cos:
    image: minio/minio:RELEASE.2022-06-07T00-33-41Z
    ports:
      - 9000:9000
    env_file:
     - env/.env.dev 
    environment:
      MINIO_ACCESS_KEY: "${GT4SD_S3_ACCESS_KEY_HUB}"
      MINIO_SECRET_KEY: "${GT4SD_S3_SECRET_KEY_HUB}"
    command: server /export
  createbuckets:
    image: minio/mc
    depends_on:
      - cos
    env_file:
     - env/.env.dev
    # ensure there is a file in the artifacts bucket
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add myminio http://cos:9000 ${GT4SD_S3_ACCESS_KEY_HUB} ${GT4SD_S3_SECRET_KEY_HUB};
      /usr/bin/mc mb myminio/${GT4SD_S3_BUCKET_HUB};
      /usr/bin/mc mb myminio/${GT4SD_S3_BUCKET_DATA_HUB};
      /usr/bin/mc mb myminio/${GT4SD_S3_BUCKET_MODELS_HUB};
      echo 'this is an artifact' >> a_file.txt;
      /usr/bin/mc cp a_file.txt myminio/${GT4SD_S3_BUCKET_HUB}/a_file.txt;
      exit 0;
      "

You can store default environment variables in .env.dev.

3) MinIO server configuration

Add the new server to the minio configuration file (~/.mc/config.json):

{
	"version": "10",
	"aliases": {
                "myminio": {
                        "url": "${GT4SD_S3_HOST_HUB}",
                        "accessKey": "${GT4SD_S3_ACCESS_KEY_HUB}",
                        "secretKey": "${GT4SD_S3_SECRET_KEY_HUB}",
                        "api": "s3v4",
                        "path": "auto"
                },
                ...
                }
}

and add myminio to the list of servers:

mc alias set myminio $GT4SD_S3_HOST_HUB $GT4SD_S3_ACCESS_KEY_HUB $GT4SD_S3_SECRET_KEY_HUB

4) run docker

After running docker compose up inside localhost-server the script creates a local minio server and the bucket structure on myminio. If everything is working you should be able to see a_file.txt running:

mc ls myminio/gt4sd-cos-algorithms-artifacts/

---

Upload models

After setting th environment variables appropriately and following steps 1-4), you can now upload your model on the server:

gt4sd-upload --training_pipeline_name paccmann-vae-trainer --model_path /tmp/gt4sd-paccmann-gp --training_name fast-example --target_version fast-example-v0 --algorithm_application PaccMannGPGenerator

You should be able to see the model and uploaded files running:

mc ls myminio/gt4sd-algorithms-artifacts/controlled_sampling/PaccMannGP/PaccMannGPGenerator/fast-example-v0/