# About The Oracle Private AI Services Container The Private AI Services Container is a lightweight, containerized web service that provides a REST API for performing inference on AI models and HNSW Index creation. Using the AI Services Container allows you to offload expensive AI computation, such as vector embedding generation, outside of the database. This can free up database compute resources that can be used for indexing and similarity search. # Using The Images As of version 26.1.0.0.0 there are several variations of the Oracle Private AI Services Container. Choosing a variation depends on your specific hardware configuration, system resources, and what specific AI tasks you wish to perform. A new naming tagging pattern has been created to facilitate these new variations: [registry]/[repo]:[qualifiers]-[version] Where: * **[registry]**: Registry host - container-registry.oracle.com * **[repo]**: Repository path - database/private-ai * **[qualifiers]**: One or more hyphen-delimited qualifiers in the fixed order [size][platform][variant][specialty] * **[size]**: small | medium | large * **[platform]** cpu | gpu * **[variant]** _(optional)_: avx512 * **[specialty]**: specialty function. infer | index * **[version]**: Product/container version (e.g., 26.2.0.0.0 or latest) The following table shows the list of new images by image qualifiers. The image qualifiers come after the registry and repo but before the version. For example in container-registry.oracle.com/database/private-ai:medium-cpu-avx512-infer-26.2.0.0.0 (the qualifiers are **medium-cpu-avx512-infer**). | Image Qualifiers | Hardware Support | AI Tasks | Notes | | ----------------------- | ----------------------- | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | small-cpu-infer | CPU | Embedding Generation, Reranking, Text Classification | This image is identical to container-registry.oracle.com/database/private-ai:25.1.5.0.0 , supporting ONNX models only, with the addition of reranking and text classification support in ONNX. | | medium-cpu-infer | CPU | Embedding Generation, Reranking, Chat Completion, Text Classification | This image runs vLLM and Llama.cpp on CPU. It contains ONNX models but does not contain an LLM model. | | medium-cpu-avx512-infer | CPU with AVX512 support | Embedding Generation, Reranking, Chat Completion, Text Classification | This image supports all the features of medium-cpu-infer but adds support for the AVX512 architecture. If AVX512 is available, this image should be chosen as LLM chat completion has been proven to be more performant on these systems. | | medium-gpu-infer | GPU | Embedding Generation, Reranking, Chat Completion, Text Classification | This image must be run on a host with a GPU. It supports all the features of medium-cpu-infer but benefits from the acceleration that GPUs provide. | | large-cpu-infer | CPU | Embedding Generation, Reranking, Chat Completion, Text Classification | This image is identical to medium-cpu-infer but includes the model **Ministral-3-3B-Reasoning-2512** . On CPU, this model will be loaded in Llama.cpp by default. To load this provided model in vLLM you must provide specific configuration. | | large-cpu-avx512-infer | CPU with AVX512 support | Embedding Generation, Reranking, Chat Completion, Text Classification | This image supports all the features of large-cpu-infer but adds support for the AVX512 architecture. | | large-gpu-infer | GPU | Embedding Generation, Reranking, Chat Completion, Text Classification | This image is identical to medium-gpu-infer but includes the model **Ministral-3-3B-Reasoning-2512**. On GPU, this model will be loaded in vLLM. There is no GPU support for Llama.cpp in this release. | | gpu-index | GPU | HNSW Index Creation | Allows you to offload the creation of Hierarchical Navigable Small World (HNSW) indexes to the container running on a host equipped with a powerful NVIDIA GPU.
This offloading is completely transparent, maintaining simplicity while delivering enhanced performance. | The following sections give basic instructions and examples for configuring and running the container. For more detailed information, refer to the [user documentation](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/index.html). ## Included Models With the exception of the `gpu-index` images, all Private AI Service Container images are shipped with a list of models. These models are sometimes referred to as default models since no configuration is required to use them. Simply starting a container without a reference to `PRIVATE_AI_CONFIG_FILE` will make all the models shipped with the image available. **NOTE** you can mix your own provided models with default models, however in this case `PRIVATE_AI_CONFIG_FILE` must be set to your config file that specifies **both** your provided models and the specific default models you wish to use. You will also need to mount a directory with your models to the container (this is not necessary when purely using default models). ### Included ONNX Format Models With the exception of the `gpu-index` images, all Private AI Service Container images contain the following ONNX Pipeline Format Models. Note, the **model path** is only required when selecting specific default models. This path must match exactly with what is shown in the table. The model name can be changed in your configuration. | Model Name | Model Path | Description | | ------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | all-minilm-l12-v2 | all-MiniLM-L12-v2.onnx | all-MiniLM-L12-v2 (based on `sentence-transformers/all-MiniLM-L12-v2`) is a lightweight, general-purpose sentence and short-paragraph embedding model. **Note:** pay attention to the case in the model file. Linux filesystems are case-sensitive so the case must match. | | multilingual-e5-base | multilingual-e5-base.zip | Multilingual E5 Base (based on `intfloat/multilingual-e5-base`) is a highly performant, open-source text embedding model designed to map sentences and paragraphs into a 768-dimensional vector space. **NOTE:** this model is distributed as a zip, not a single ONNX file. | | multilingual-e5-large | multilingual-e5-large.zip | Multilingual E5 Large (based on `multilingual-e5-large`) is a state-of-the-art text embedding model that maps sentences and documents into a high-dimensional vector space. **NOTE:** this model is distributed as a zip, not a single ONNX file. | | clip-vit-base-patch32-txt | clip-vit-base-patch32-txt.onnx | CLIP-ViT-Base-Patch32 (based on `openai/clip-vit-base-patch32`)is a popular, pretrained multimodal model created by OpenAI that connects images and text into a shared vector space. This model is the text version. | | clip-vit-base-patch32-img | clip-vit-base-patch32-img.onnx | CLIP-ViT-Base-Patch32 (based on `openai/clip-vit-base-patch32`)is a popular, pretrained multimodal model created by OpenAI that connects images and text into a shared vector space. This model is the image version. | | all-mpnet-base-v2 | all-mpnet-base-v2.onnx | all-mpnet-base-v2 (based on `sentence-transformers/all-mpnet-base-v2`) is a highly popular open-source embedding model that maps sentences and paragraphs into a 768-dimensional dense vector space | | Llama-Prompt-Guard-2-86M | N/A | Llama Prompt Guard 2 (based on `meta-llama/Llama-Prompt-Guard-2-86M` ) is a specialized, open-source text classification model designed by Meta to protect Large Language Model (LLM) applications by intercepting and blocking **prompt injection and jailbreaking attacks** in real time. **NOTE:** this model is not available for direct use (hence no path provided). It is only loaded and used as a guardrail. | ### Included Large Language Models In addition to the aforementioned ONNX models, Large Language Models are shipped with images that have the following qualifiers: * large-cpu-infer * large-cpu-avx512-infer * large-gpu-infer These LLMs are loaded be the backend runtime engines - vLLM or Llama.cpp - by default. For CPU based images (large-cpu-infer and large-cpu-avx512-infer), Llama.cpp will be the default runtime engine. | Model Name | Model Path | Description | | ---------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Ministral-3-3B-Reasoning-2512-Q8_0 | Ministral-3-3B-Reasoning-2512-Q8_0.gguf | Ministral-3-3B-Reasoning-2512 is a compact, open-weight language model developed by [Mistral AI](https://docs.mistral.ai/models/model-cards/ministral-3-3b-25-12). Released in December 2025, it is specifically post-trained for complex, multi-step reasoning and dynamic problem-solving. **NOTE:** this model is the GGUF version loaded by Llama.cpp on CPU only. | | Ministral-3-3B-Reasoning-2512 | Ministral-3-3B-Reasoning-2512 | Ministral-3-3B-Reasoning-2512 is a compact, open-weight language model developed by [Mistral AI](https://docs.mistral.ai/models/model-cards/ministral-3-3b-25-12). Released in December 2025, it is specifically post-trained for complex, multi-step reasoning and dynamic problem-solving. **NOTE:** this model is the vLLM ready version that can run on either CPU or GPU. For CPU usage you must specify this model in your config.json and set the runtime to `vllm`. The model path is a directory | ## Running A Container With Default Models (HTTP) This is the quickest way to get up and running. NOTE: these instructions will run the container with no security enabled and is not recommended for production environments. Use the following command, replacing `` with an available port on your host (e.g. 9091): ``` podman run -it -e PRIVATE_AI_HTTPS_ENABLED=false -p :8080 container-registry.oracle.com/database/private-ai:large-cpu-infer-latest ``` **NOTE**: In this example, very little information will be outputted to the terminal. By default the container does not log to `stdout` since it can result in high disk usage and lead to the potential of running out of disk space. You can choose instead to configure external logging (see logging in the Advanced Configuration section), or you can enable logging to stdout by adding `-e PRIVATE_AI_LOG_STDOUT_ENABLED=true` to the command. `-e PRIVATE_AI_LOG_STDOUT_ENABLED=true` is not recommended in production environments. Now that the container is up and running you can perform a quick test by listing the models. Using curl you can list models with the following command - `` should be the same port you used when starting the container: ``` curl http://localhost:/v1/models ``` ## Running A Container With Default Models In Secure Mode (HTTPS + Authentication) To run the container in secure mode you will need to complete the following tasks before starting the container. The tools for these tasks should be available in any standard Linux distribution. * Create a self-signed certificate * Create an API key * Create a keystore for the certificate * Create podman secrets ### Creating A Self-Signed Certificate and Keystore You should run the following commands in a directory of your choice. This directory will need to have permissions that allow the container user (**NOTE**: the container user is not the same as the user that launches the container) to read these files. In non production environments you can use `chmod a+r -R ` or `chmod 777 -R `, but in a production environment you should provide the least amount of permissions possible. First generate a private key : ``` openssl genrsa -out key.pem ``` Next generate the self-signed certificate. You will be prompted with some questions before generation such as the country, and state for the certificate. You can skip or enter fake information for most questions however on the prompt for "Common Name", be sure to enter the ip address of your host (typically `hostname -i` will give you this information). Please replace `` in the example below with the number of days the certificate should be valid for - e.g. 365: ``` openssl req -new -x509 -key key.pem -out cert.pem -days ``` Now that the private key and certificate have been created, it's time to generate the keystore. When generating the keystore you will be prompted to enter a password. It is crucial that you remember this password as it will be used later on when secrets are created so that the container can actually access the keystore. Replace `` with the name of the actual keystore file you wish to use. For simplicity you can use `keystore` as the name of the file: ``` openssl pkcs12 -export -inkey key.pem -in cert.pem -name mykey -out ``` ### Creating An API Key **NOTE:** as of version 26.2.0.0.0 the service supports multiple API keys. Single keys are still supported and shown in this example. Please refer to the documentation for multi-key support. You can generate an API key that can be used to authenticate clients accessing the service. The following step is but one way to generate a unique, randomized string that can be used as an API key. In this example the API key will be written to a file named api-key.txt. If you choose a different file name you will need to also update subsequent steps:

head -c 32 /dev/urandom | xxd -p | tr -d '\n' | head -c 64 > api-key.txt

### Creating The Podman Secrets The service uses podman secrets to access the previously created security artifacts. You will need to create three distinct secrets (order does not matter): * A keystore secret * A keystore password secret * An API Key secret You can verify the secrets exist in podman using the `podman ls` command. You can also delete individual secrets using `podman secret rm ` #### **Creating The Keystore Secret** Assuming you chose `keystore` as the name of your keystore file (this is the final argument of the command), you can create the keystore secret with the following command: ``` podman secret create keystore keystore ``` #### **Creating The Keystore Password Secret** You'll now store the password you used earlier for creating the keystore as a secret. First you'll need to save the password to a file. Assuming the password is in a file named `privateai-ssl-pwd.txt`, you can create a secret for the password with the following command: ``` podman secret create privateai-ssl-pwd privateai-ssl-pwd.txt ``` #### **Creating The API Key Secret** Assuming you saved the API Key to a file named `api-key.txt` you can create a secret for the key with the following command: ``` podman secret create api-key api-key.txt ``` ### Running The Container In Secure Mode With all the secrets created you can now run the container in secure mode: ``` podman run -it \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ -p :8443 \ container-registry.oracle.com/database/private-ai:large-cpu-infer-latest ``` You will need to replace `` in the command above with an available port on your host (e.g 9092 or 9443). Now that the container is up and running you can test that the service is operable in secure mode. An easy way to do this is using the `curl` command with the previously created certificate and api key you created previously. Assuming these are named `cert.pem` and `api-key.txt` respectively: ``` curl --cacert cert.pem --header "Authorization: Bearer `cat api-key.txt`" https://:/v1/models ``` You'll need to replace `` with the same ip address or hostname you used when you created the certificate (answer to the "Common Name" question), and the port should be the same port you used when starting the container (refer to the port mapping `-p :8443` in the previous step). ## Running The Container On A GPU When using any of the GPU container variants, you should take the following steps: 1. Ensure that you have a recent NVIDIA Container Toolkit installed on a GPU machine, 2. Check that the GPUs are visible in `nvidia-ctk cdi list` and meet the [minimum requirements](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/install-private-ai-services-container.html), 3. (especially for `gpu-index` variants) ensure that the GPU machine is reachable by your Oracle AI Database 26ai 4. select one of the `gpu-index-` , `medium-gpu-infer-` or `large-gpu-infer-`tags 5. launch the container in the GPU machine and bind a GPU to it using `--devices nvidia.com/gpu=` as part of the aforementioned podman run. For example, to launch the `gpu-index` container with GPU 0: ``` podman run -it \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ --devices nvidia.com/gpu=0 \ -p :8443 \ container-registry.oracle.com/database/private-ai:gpu-index-26.1.0.0.0 ``` Or similarly for the large-gpu-infer container: ``` podman run -it \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ --devices nvidia.com/gpu=0 \ -p :8443 \ container-registry.oracle.com/database/private-ai:large-gpu-infer-latest ``` ## Scripts To Configure And Run The Container As of version `25.1.2.0.0`, there are scripts included in the image that allow you to configure and run the container. These scripts simplify the configuration process for both the secure and non-secure modes of running. To use the scripts you first need to copy them from an image. The following commands will copy the scripts to the current directory: ``` imageid=$(podman create container-registry.oracle.com/database/private-ai:) podman cp $imageid:/privateai/scripts/privateai-setup-.zip . ``` In the commands above you need to replace `` with the actual version of the image you are using (**NOTE**: you must specify an explicit version, not `latest`). Once you have successfully copied the zip you can unzip it and follow the directions in the README.md file. More detailed instructions on using the scripts are also provided in the [user documentation](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/index.html). ## Advanced Configurations ### Configuring Logging The service can write logs to files instead of the terminal. This is convenient for saving logs and also is more efficient with disk usage as the logs will roll after reaching a preconfigured size total. Since the container needs write logs to a directory on your host system, you need to set permissions on the log directory before starting the container. For example, you can give write permissions for other users to the log directory. **NOTE** even though you will be running the container it will not be using your user id to access files. ``` mkdir logs chmod a+rw logs ``` Now launch the container and mount the logs directory. Previously we enabled logging to stdout with `-e PRIVATE_AI_LOG_STDOUT_ENABLED=true`. While this is still a valid setting, with logging enabled this is no longer necessary so can be omitted. We can also launch the container in daemon mode with `-d` instead of `-it`. Finally, if running on an SELinux system, the volume mount command is suffixed with `:z` to enable the container to write to the logs folder even with permissions set. This suffix may not be needed on non SELinux systems. ``` podman run -d --name \ -v ./logs:/privateai/logs:z \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ -p :8443 \ container-registry.oracle.com/database/private-ai:large-cpu-infer-latest ``` Replace `` with a unique name for the container (e.g. mycontainer), and `` with an available port on your host. Now that the container is running you should see new directories appear when you send a request to the host. Try any of the list model examples from previous sections. **NOTE**: logging to directories works both in secure and non-secure mode. ### Providing Models In version 26.2.x.x.x, models can be provided from the local filesystem or via remote download from huggingface. The latter is far more convenient if the container has access to the internet. The following example shows a configuration for downloading the **microsoft/Phi-3.5-mini-instruct** from huggingface. ```json { "models":[ { "name":"phi-mini-instruct", "path":"microsoft/Phi-3.5-mini-instruct", "runtime":"vllm", "capabilities":["TEXT_GENERATION"] } ] } ``` When downloading from huggingface be sure to provide the correct path to the model which typically comes right after the huggingface domain (huggingface.co). When downloading models you do not need to mount a models directory. This is only required when loading models from the local filesystem. **NOTE:** additional configuration may be needed for some models. Please see the docs for more information on downloading and configuring models. **NOTE**: for ONNX support (since version 25.x.x.x.x) only ONNX pipeline models are supported. To generate these models you must use OML4Py Client 2.1 which downloads embedding models from huggingface and converts them to the ONNX format with embedded pre-processors (e.g. tokenizer) and post-processors. See [these instructions](https://docs.oracle.com/en/database/oracle/machine-learning/oml4py/2-23ai/mlpug/convert-pretrained-models-onnx-model-end-end-instructions.html) for more details. **NOTE**: you must use OML4Py to perform the conversion to ONNX. The service does not support ONNX files downloaded directly from huggingface. Once you have generated your embedding model ONNX file you can copy it to a directory of your choosing. You will need to mount this directory to the container. You will then create a file `config.json` that will contain metadata for your ONNX format model: ``` { "models": [ { "name":"", "path":".onnx", "function":"EMBEDDING", "cache_on_startup":true } ] } ``` Replace `` with a unique name for the model, and `` with the simplename of the model file along with the `.onnx` prefix. **NOTE**: do not include the full path to the model as it will be loaded from within the container relative to the volume mount point. Save this file as config.json. Now run the container and mount both the directory containing the configuration file as well as the directory containing the model: ``` podman run -it -e PRIVATE_AI_LOG_STDOUT_ENABLED=true \ -e PRIVATE_AI_CONFIG_FILE=/privateai/config/config.json \ --secret keystore \ --secret privateai-ssl-pwd \ --secret api-key \ -v :/privateai/models \ -v :/privateai/config \ -p :8443 \ container-registry.oracle.com/database/private-ai:large-cpu-infer-latest ``` The `-e PRIVATE_AI_CONFIG_FILE=/privateai/config/config.json` instructs the container to read models using the config.json file. Note that the path is internal to the container **not** to your host system. Replace `` and `` with the full paths of the directory where you stored the ONNX format model and the directory where you stored the config.json file respectively. Replace `` with an available port on your host. #More Info [ Documentation](https://docs.oracle.com/en/database/oracle/oracle-database/26/prvai/index.html)