# About The Oracle Private AI Services Container The Private AI Services Container is a lightweight, containerized web service that provides a REST API for performing inference on AI models. Using the AI Services Container allows you to offload expensive AI computation, such as vector embedding generation, outside of the database. This can free up database compute resources that can be used for indexing and similarity search. # Using This Image The following sections give basic instructions and examples for configuring and running the container. For more detailed information, refer to the [user documentation](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/index.html). ## Running A Container With Default Models (HTTP) This is the quickest way to get up and running. NOTE: these instructions will run the container with no security enabled and is not recommended for production environments. Use the following command, replacing `` with an available port on your host (e.g. 9091): ``` podman run -it -e PRIVATE_AI_HTTPS_ENABLED=false -p :8080 container-registry.oracle.com/database/private-ai:latest ``` **NOTE**: In this example, very little information will be outputted to the terminal. By default the container does not log to `stdout` since it can result in high disk usage and lead to the potential of running out of disk space. You can choose instead to configure external logging (see logging in the Advanced Configuration section), or you can enable logging to stdout by adding `-e PRIVATE_AI_LOG_STDOUT_ENABLED=true` to the command. `-e PRIVATE_AI_LOG_STDOUT_ENABLED=true` is not recommended in production environments. Now that the container is up and running you can perform a quick test by listing the models. Using curl you can list models with the following command - `` should be the same port you used when starting the container: ``` curl http://localhost:/v1/models ``` ## Running A Container With Default Models In Secure Mode (HTTPS + Authentication) To run the container in secure mode you will need to complete the following tasks before starting the container. The tools for these tasks should be available in any standard Linux distribution. * Create a self-signed certificate * Create an API key * Create a keystore for the certificate * Create podman secrets ### Creating A Self-Signed Certificate and Keystore You should run the following commands in a directory of your choice. This directory will need to have permissions that allow the container user (**NOTE**: the container user is not the same as the user that launches the container) to read these files. In non production environments you can use `chmod a+r -R ` or `chmod 777 -R `, but in a production environment you should provide the least amount of permissions possible. First generate a private key : ``` openssl genrsa -out key.pem ``` Next generate the self-signed certificate. You will be prompted with some questions before generation such as the country, and state for the certificate. You can skip or enter fake information for most questions however on the prompt for "Common Name", be sure to enter the ip address of your host (typically `hostname -i` will give you this information). Please replace `` in the example below with the number of days the certificate should be valid for - e.g. 365: ``` openssl req -new -x509 -key key.pem -out cert.pem -days ``` Now that the private key and certificate have been created, it's time to generate the keystore. When generating the keystore you will be prompted to enter a password. It is crucial that you remember this password as it will be used later on when secrets are created so that the container can actually access the keystore. Replace `` with the name of the actual keystore file you wish to use. For simplicity you can use `keystore` as the name of the file: ``` openssl pkcs12 -export -inkey key.pem -in cert.pem -name mykey -out ``` ### Creating An API Key You can generate an API key that can be used to authenticate clients accessing the service. The following step is but one way to generate a unique, randomized string that can be used as an API key. In this example the API key will be written to a file named api-key.txt. If you choose a different file name you will need to also update subsequent steps:

head -c 32 /dev/urandom | xxd -p | tr -d '\n' | head -c 64 > api-key.txt

### Creating The Podman Secrets The service uses podman secrets to access the previously created security artifacts. You will need to create three distinct secrets (order does not matter): * A keystore secret * A keystore password secret * An API Key secret You can verify the secrets exist in podman using the `podman ls` command. You can also delete individual secrets using `podman secret rm ` #### **Creating The Keystore Secret** Assuming you chose `keystore` as the name of your keystore file (this is the final argument of the command), you can create the keystore secret with the following command: ``` podman secret create keystore keystore ``` #### **Creating The Keystore Password Secret** You'll now store the password you used earlier for creating the keystore as a secret. First you'll need to save the password to a file. Assuming the password is in a file named `privateai-ssl-pwd.txt`, you can create a secret for the password with the following command: ``` podman secret create privateai-ssl-pwd privateai-ssl-pwd.txt ``` #### **Creating The API Key Secret** Assuming you saved the API Key to a file named `api-key.txt` you can create a secret for the key with the following command: ``` podman secret create api-key api-key.txt ``` ### Running The Container In Secure Mode With all the secrets created you can now run the container in secure mode: ``` podman run -it \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ -p :8443 \ container-registry.oracle.com/database/private-ai:latest ``` You will need to replace `` in the command above with an available port on your host (e.g 9092 or 9443). Now that the container is up and running you can test that the service is operable in secure mode. An easy way to do this is using the `curl` command with the previously created certificate and api key you created previously. Assuming these are named `cert.pem` and `api-key.txt` respectively: ``` curl --cacert cert.pem --header "Authorization: Bearer `cat api-key.txt`" https://:/v1/models ``` You'll need to replace `` with the same ip address or hostname you used when you created the certificate (answer to the "Common Name" question), and the port should be the same port you used when starting the container (refer to the port mapping `-p :8443` in the previous step). ## Scripts To Configure And Run The Container As of version `25.1.2.0.0`, there are scripts included in the image that allow you to configure and run the container. These scripts simplify the configuration process for both the secure and non-secure modes of running. To use the scripts you first need to copy them from an image. The following commands will copy the scripts to the current directory: ``` imageid=$(podman create container-registry.oracle.com/database/private-ai:) podman cp $imageid:/privateai/scripts/privateai-setup-.zip . ``` In the commands above you need to replace with the actual version of the image you are using (**NOTE**: you must specify an explicit version, not `latest`). Once you have successfully copied the zip you can unzip it and follow the directions in the README.md file. More detailed instructions on using the scripts are also provided in the [user documentation](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/index.html). ## Advanced Configurations ### Configuring Logging The service can write logs to files instead of the terminal. This is convenient for saving logs and also is more efficient with disk usage as the logs will roll after reaching a preconfigured size total. Since the container needs write logs to a directory on your host system, you need to set permissions on the log directory before starting the container. For example, you can give write permissions for other users to the log directory. **NOTE** even though you will be running the container it will not be using your user id to access files. ``` mkdir logs chmod a+rw logs ``` Now launch the container and mount the logs directory. Previously we enabled logging to stdout with `-e PRIVATE_AI_LOG_STDOUT_ENABLED=true`. While this is still a valid setting, with logging enabled this is no longer necessary so can be omitted. We can also launch the container in daemon mode with `-d` instead of `-it`. Finally, if running on an SELinux system, the volume mount command is suffixed with `:z` to enable the container to write to the logs folder even with permissions set. This suffix may not be needed on non SELinux systems. ``` podman run -d --name \ -v ./logs:/privateai/logs:z \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ -p :8443 \ container-registry.oracle.com/database/private-ai:latest ``` Replace `` with a unique name for the container (e.g. mycontainer), and `` with an available port on your host. Now that the container is running you should see new directories appear when you send a request to the host. Try any of the list model examples from previous sections. **NOTE**: logging to directories works both in secure and non-secure mode. ### Providing Models **NOTE**: versions `25.x.x.x.x` of the service only support ONNX embedding pipeline models. To generate these models you can use OML4Py Client 2.1 which downloads embedding models from huggingface and converts them to the ONNX format with embedded pre-processors (e.g. tokenizer) and post-processors. See [these instructions](https://docs.oracle.com/en/database/oracle/machine-learning/oml4py/2-23ai/mlpug/convert-pretrained-models-onnx-model-end-end-instructions.html) for more details. **NOTE**: you must use OML4Py to perform the conversion to ONNX. The service does not support ONNX files downloaded directly from huggingface. Once you have generated your embedding model ONNX file you can copy it to a directory of your choosing. You will need to mount this directory to the container. You will then create a file `config.json` that will contain metadata for your ONNX format model: ``` { "models": [ { "modelname":"", "modelfile":".onnx", "modelfunction":"EMBEDDING", "cache_on_startup":true } ] } ``` Replace `` with a unique name for the model, and `` with the simplename of the model file along with the `.onnx` prefix. **NOTE**: do not include the full path to the model as it will be loaded from within the container relative to the volume mount point. Save this file as config.json. Now run the container and mount both the directory containing the configuration file as well as the directory containing the model: ``` podman run -it -e PRIVATE_AI_LOG_STDOUT_ENABLED=true \ -e PRIVATE_AI_CONFIG_FILE=/privateai/config/config.json \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ -v :/privateai/models \ -v :/privateai/config \ -p :8443 \ container-registry.oracle.com/database/private-ai:latest ``` The `-e PRIVATE_AI_CONFIG_FILE=/privateai/config/config.json` instructs the container to read models using the config.json file. Note that the path is internal to the container **not** to your host system. Replace `` and `` with the full paths of the directory where you stored the ONNX format model and the directory where you stored the config.json file respectively. Replace `` with an available port on your host. #More Info [ Documentation](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/index.html)