TensorFlow Serving 2

sudo systemctl is-active docker.service tfserving.service tfserving-firstboot.service
sudo docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
sudo docker exec tfserving tensorflow_model_server --version

All three services report active. The tfserving container shows image tensorflow/serving:2.19.1 and an Up status. tensorflow_model_server --version reports the embedded TensorFlow build banner.

sudo cat /stage/scripts/tfserving-credentials.log

The file defines the following values.

• TFSERVING_REST_URL (default http://127.0.0.1:8501)
• TFSERVING_GRPC_URL (default 127.0.0.1:8500)
• TFSERVING_SAMPLE_PREDICT_URL, the ready-made POST URL for the sample model
• TFSERVING_API_KEY, a per-VM string for use with a downstream nginx auth layer. TF Serving itself does not enforce it.

The file is root:root 0600.

curl -sf http://127.0.0.1:8501/v1/models/half_plus_two | jq

The response shows model_version_status[0].state = "AVAILABLE" and version = "1". This shows that the SavedModel at /data/models/half_plus_two/1/ is loaded and ready to serve predictions. The status.error_code = "OK" field confirms that there were no load-time errors.

The sample model implements y = 0.5x + 2, so inputs [1.0, 2.0, 5.0] return [2.5, 3.0, 4.5].

curl -sf -X POST -H 'Content-Type: application/json' \
  -d '{"instances": [1.0, 2.0, 5.0]}' \
  http://127.0.0.1:8501/v1/models/half_plus_two:predict | jq

The response is {"predictions": [2.5, 3.0, 4.5]}. This confirms the full inference path: the request reaches the container, the SavedModel runs the math, and the response comes back over REST. Once you drop a SavedModel into /data/models/, replace half_plus_two with your own model name.

curl -sf http://127.0.0.1:8501/v1/models/half_plus_two/metadata | jq

The signature_def block is the contract every client needs to call the model. It lists the named input tensors (with dtype and shape), the named output tensors and the signature method name, which is typically tensorflow/serving/predict or tensorflow/serving/classify. When you build a Python or Go client, query this endpoint first to discover the input and output tensor names without reading the SavedModel by hand.

TensorFlow Serving expects the on-disk layout <model_root>/<model_name>/<version>/saved_model.pb plus the variables/ and assets/ directories that tf.saved_model.save() produces. Copy your SavedModel into /data/models/.

sudo mkdir -p /data/models/my_model/1
sudo cp -r /path/to/saved_model/* /data/models/my_model/1/
sudo chown -R root:root /data/models/my_model
sudo systemctl restart tfserving.service sleep 5 curl -sf http://127.0.0.1:8501/v1/models/my_model | jq

TensorFlow Serving detects new versions automatically. Add a higher-numbered version directory (for example /data/models/my_model/2/) to the same model root, and TF Serving hot-swaps to the new version on its next file-system scan, with no restart. The previous version is unloaded once the new one is fully ready, so in-flight requests are not dropped.

To serve more than one model at once, edit /opt/tfserving/models.config and list every model in the model_config_list block.

sudo tee /opt/tfserving/models.config <<'EOF' model_config_list { config { name: "half_plus_two" base_path: "/models/half_plus_two" model_platform: "tensorflow" } config { name: "my_model" base_path: "/models/my_model" model_platform: "tensorflow" } } EOF
sudo systemctl restart tfserving.service curl -sf http://127.0.0.1:8501/v1/models/my_model | jq

Mount the config file into the container by adding a --model_config_file=/models/models.config flag to the command: entry in /opt/tfserving/docker-compose.yml, replacing the single-model --model_name and --model_base_path flags. Then run docker compose up -d to apply the change.

For lower-latency, higher-throughput access with binary protobuf instead of JSON, use the gRPC endpoint on :8500.

import grpc import tensorflow as tf from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc channel = grpc.insecure_channel('<vm-ip>:8500') stub = prediction_service_pb2_grpc.PredictionServiceStub(channel) request = predict_pb2.PredictRequest() request.model_spec.name = 'half_plus_two' request.model_spec.signature_name = 'serving_default' request.inputs['x'].CopyFrom( tf.make_tensor_proto([1.0, 2.0, 5.0], dtype=tf.float32) ) print(stub.Predict(request, 5.0))

Install the client dependencies with pip install tensorflow tensorflow-serving-api grpcio. The gRPC API supports the same Predict, Classify, Regress, MultiInference and GetModelMetadata calls as REST. It uses proto-encoded tensors, which are roughly 30-60% smaller on the wire and avoid JSON parsing overhead on both ends.

For throughput-sensitive workloads, enable server-side batching so that concurrent client requests are coalesced into a single GPU or CPU forward pass.

sudo tee /opt/tfserving/batching.config <<'EOF' max_batch_size { value: 32 } batch_timeout_micros { value: 5000 } num_batch_threads { value: 4 } max_enqueued_batches { value: 100 } EOF

Then extend the command: array in /opt/tfserving/docker-compose.yml to include the following.

- --enable_batching=true - --batching_parameters_file=/models/batching.config

Make the batching config available to the container by placing the file in /data/models, which maps to /models in the container. Apply the change with sudo systemctl restart tfserving.service. You can tune CPU resources per container with the cpus: and mem_limit: keys in docker-compose.yml.

For GPU inference, swap the image for tensorflow/serving:2.19.1-gpu and add the runtime: nvidia key. This requires the NVIDIA Container Toolkit on a GPU-equipped VM such as Standard_NC6s_v3.

TensorFlow Serving has no built-in auth, so a fronting nginx is the conventional way to add HTTP basic auth and TLS. You can also add mTLS, JWT or OAuth through a sidecar.

sudo apt-get update
sudo apt-get install -y nginx apache2-utils
sudo htpasswd -bc /etc/nginx/.tfserving.htpasswd tfclient "$(openssl rand -base64 24)"
sudo tee /etc/nginx/sites-available/tfserving <<'EOF' server { listen 80; server_name _; location / { auth_basic "TF Serving"; auth_basic_user_file /etc/nginx/.tfserving.htpasswd; proxy_pass http://127.0.0.1:8501; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } } EOF
sudo ln -sf /etc/nginx/sites-available/tfserving /etc/nginx/sites-enabled/tfserving
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

Once nginx is in front, lock down the NSG. Make 8500/tcp and 8501/tcp reachable only from 127.0.0.1 (or your private VNet), and make only 80/tcp and 443/tcp publicly reachable.

sudo apt-get install -y python3-certbot-nginx
sudo certbot --nginx -d apps.example.com -m you@example.com --agree-tos --non-interactive --redirect

Point your domain's A record at this VM first, then run certbot. The plugin edits the nginx server block in place to add listen 443 ssl;, the certificate paths and a 301 redirect from port 80. The systemd timer certbot.timer, which the Ubuntu package installs by default, handles renewal, so no further work is needed.

TensorFlow Serving is stateless, and its only persistent state is the SavedModel files on disk. A backup is a tarball of /data/models.

sudo tar -czf /var/backups/tfserving-models-$(date +%F).tgz -C /data models
sudo cp /opt/tfserving/docker-compose.yml /var/backups/docker-compose.yml-$(date +%F)

For off-VM retention, periodically copy /var/backups to Azure Blob Storage with az storage blob upload-batch. You do not need to back up the container itself. The image is pulled from Docker Hub on demand, and the wrapper config lives in /opt/tfserving/, which is small enough to keep in version control.

sudo journalctl -u tfserving.service --no-pager -n 80
sudo journalctl -u tfserving-firstboot.service --no-pager -n 30
sudo docker logs tfserving --tail 80
sudo docker exec tfserving tensorflow_model_server --version

docker logs tfserving -f follows the live server log. Every request hits the access log at INFO level, every load and unload is reported, and any malformed input tensor surfaces here as a structured error.

Common issues:

• Could not find SavedModel: The on-disk layout is wrong. The required layout is /data/models/<model_name>/<version>/saved_model.pb plus the variables/ directory next to it. The <version> segment must be a positive integer.
• No versions of servable <name> found: The model directory is empty or contains only a non-numeric subdirectory. Rename the subdirectory to 1/, 2/ and so on.
• HTTP 400 Could not parse request body: The JSON shape is wrong. The REST API expects {"instances": [...]} for predict and {"signature_name": "serving_default", "examples": [...]} for classify and regress.
• Container restarts on boot: Confirm that /var/lib/sac/data/tfserving-firstboot.done exists. If it does not, run sudo systemctl start tfserving-firstboot.service and check again.