The APAC Last Mile of ML: Getting Models from Training to Production
APAC data science and ML engineering teams that train APAC ML models successfully but struggle to deploy them into APAC production reliably encounter the "last mile" problem of ML: the gap between a APAC model artifact (.pkl, .pt, .savedmodel) on a data scientist's laptop and a production-grade APAC API that APAC applications can query at scale.
The APAC last mile involves:
- APAC packaging: wrapping the APAC model + preprocessing + postprocessing as a deployable APAC artifact
- APAC serving infrastructure: REST/gRPC API server with APAC request routing and batching
- APAC versioning: running APAC multiple model versions for APAC A/B testing and canary rollouts
- APAC observability: APAC latency, throughput, and error rate metrics for deployed APAC models
- APAC scaling: handling APAC traffic bursts without over-provisioning APAC idle GPU capacity
Three frameworks address different APAC serving profiles:
BentoML — Python-native, framework-agnostic APAC model packaging for APAC data scientists and ML engineers who want to own APAC production deployment.
TorchServe — PyTorch-specific APAC production server from Meta and AWS for APAC engineering teams deeply committed to the PyTorch ecosystem.
KServe — Kubernetes-native APAC InferenceService platform for APAC ML platform teams building shared APAC model serving infrastructure.
APAC ML Serving Fundamentals
APAC inference serving architecture patterns
Pattern 1: APAC Direct serving (small team, single model)
APAC Model → BentoML Service → APAC Docker container → Cloud Run / ECS
APAC data scientist owns full APAC stack
APAC scaling: Cloud Run auto-scale on APAC requests
Pattern 2: APAC Multi-model server (APAC engineering team, managed GPU)
APAC Models (n) → TorchServe instance (shared APAC GPU)
APAC ML engineer manages APAC model registration/lifecycle
APAC scaling: Horizontal APAC TorchServe pod scaling on K8s
Pattern 3: APAC ML platform (APAC platform team, many teams/models)
APAC Team A model → InferenceService A (KServe) ──┐
APAC Team B model → InferenceService B (KServe) ──┤── APAC K8s cluster
APAC Team C model → InferenceService C (KServe) ──┘ (shared APAC infra)
APAC ML platform team owns serving APAC infrastructure
APAC ML teams own InferenceService APAC manifests
APAC scaling: Knative serverless scale-to-zero per APAC service
Key APAC model serving metrics
APAC Metric Target Tooling
APAC Latency p95 < 200ms (realtime) Prometheus + Grafana
< 2s (async)
APAC Throughput Model-dependent APAC requests/second
APAC GPU utilization > 60% (efficient) DCGM Exporter
APAC Batch fill rate > 80% (GPU serving) TorchServe / BentoML metrics
APAC Model error rate < 0.1% APAC custom metrics
APAC Queue depth < 10 (realtime) APAC serving framework metrics
BentoML: APAC Python-Native Model Packaging
BentoML Service definition — APAC fraud detection model
# apac_fraud_service.py — BentoML Service for APAC payment fraud detection
import bentoml
import numpy as np
from bentoml.io import NumpyNdarray, JSON
# Load APAC model from BentoML model store
apac_fraud_runner = bentoml.sklearn.get("apac-fraud-detector:latest").to_runner()
# Define APAC BentoML service
svc = bentoml.Service("apac_fraud_detector", runners=[apac_fraud_runner])
@svc.api(input=JSON(), output=JSON())
async def predict_fraud(apac_transaction: dict) -> dict:
"""APAC fraud prediction endpoint for payment transactions."""
# APAC preprocessing: extract features from APAC transaction dict
apac_features = np.array([[
apac_transaction["amount"],
apac_transaction["hour_of_day"],
apac_transaction["days_since_last_tx"],
apac_transaction["apac_merchant_category_code"],
apac_transaction["apac_country_risk_score"],
]])
# APAC inference via runner (handles APAC batching automatically)
apac_fraud_probability = await apac_fraud_runner.predict.async_run(apac_features)
return {
"apac_transaction_id": apac_transaction["id"],
"apac_fraud_probability": float(apac_fraud_probability[0]),
"apac_risk_level": "HIGH" if apac_fraud_probability[0] > 0.7 else "LOW",
}
BentoML — build and deploy APAC Bento
# Save APAC trained model to BentoML model store
# (run after APAC model training)
bentoml.sklearn.save_model(
"apac-fraud-detector",
trained_model,
signatures={"predict": {"batchable": True, "batch_dim": 0}},
labels={"framework": "sklearn", "apac_region": "sea", "version": "2.1.0"},
)
# Build APAC Bento (Docker image with model + deps + serving infra)
bentoml build apac_fraud_service.py --do-not-track
# Output:
# Successfully built Bento(tag="apac_fraud_detector:abc123")
# → Image contains: APAC model artifact + sklearn + numpy + BentoML server
# Containerize APAC Bento for deployment
bentoml containerize apac_fraud_detector:abc123 \
--image-tag apac-ecr.region.amazonaws.com/fraud-service:2.1.0
# Push and deploy to APAC Kubernetes:
docker push apac-ecr.region.amazonaws.com/fraud-service:2.1.0
kubectl apply -f apac-fraud-deployment.yaml
TorchServe: APAC PyTorch Production Server
TorchServe — APAC model archive and registration
# Package APAC PyTorch model as MAR (Model Archive)
torch-model-archiver \
--model-name apac-image-classifier \
--version 3.0 \
--model-file apac_model.py \
--serialized-file apac_model_weights.pt \
--handler apac_image_classification_handler.py \
--extra-files apac_class_labels.json \
--export-path ./apac-model-store/
# Start TorchServe with APAC model store
torchserve --start \
--model-store ./apac-model-store/ \
--models apac-image-classifier=apac-image-classifier.mar \
--ts-config apac-config.properties
# Register additional APAC model without restart (management API)
curl -X POST "http://localhost:8081/models?model_name=apac-fraud-v2&url=apac-fraud-v2.mar&batch_size=8&max_batch_delay=100"
# APAC A/B: set traffic split between APAC model versions
curl -X PUT "http://localhost:8081/models/apac-fraud-detector/1.0?min_worker=2"
curl -X PUT "http://localhost:8081/models/apac-fraud-detector/2.0?min_worker=1"
# → 75% APAC traffic to v1.0, 25% to v2.0 (3:1 APAC worker ratio)
TorchServe config.properties — APAC production settings
# apac-config.properties
inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
# APAC batch inference settings
batch_size=16
max_batch_delay=100
# APAC GPU settings
number_of_gpu=2
# APAC metrics (Prometheus scrape target at :8082/metrics)
metrics_mode=prometheus
KServe: APAC Kubernetes-Native InferenceService
KServe InferenceService — APAC multi-framework serving
# apac-sklearn-fraud.yaml — KServe InferenceService for APAC sklearn model
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: apac-fraud-classifier
namespace: apac-ml-serving
annotations:
serving.kserve.io/deploymentMode: "Serverless" # APAC Knative scale-to-zero
spec:
predictor:
sklearn:
storageUri: "s3://apac-models/fraud-classifier/v3/"
resources:
requests:
memory: "2Gi"
cpu: "1"
limits:
memory: "4Gi"
---
# apac-pytorch-canary.yaml — APAC canary rollout for new PyTorch model version
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: apac-image-classifier
namespace: apac-ml-serving
spec:
predictor:
canaryTrafficPercent: 20 # APAC 20% to new version, 80% to stable
pytorch:
storageUri: "s3://apac-models/image-classifier/v4/"
resources:
requests:
cpu: "2"
memory: "8Gi"
nvidia.com/gpu: "1" # APAC GPU request
KServe InferenceGraph — APAC pipeline orchestration
# Compose APAC preprocessing + classifier + postprocessing models
apiVersion: "serving.kserve.io/v1alpha1"
kind: "InferenceGraph"
metadata:
name: apac-payment-risk-pipeline
namespace: apac-ml-serving
spec:
nodes:
root:
routerType: Sequence # APAC sequential pipeline
steps:
- serviceName: apac-feature-extractor # Step 1: APAC feature extraction
name: apac-extract-features
- serviceName: apac-fraud-classifier # Step 2: APAC fraud score
name: apac-score-transaction
- serviceName: apac-risk-aggregator # Step 3: APAC risk aggregation
name: apac-aggregate-risk
APAC ML Serving Tool Selection
APAC Serving Need → Tool → Why
APAC data scientist owns deployment → BentoML Python-native; APAC minimal
(single model, Python team) → DevOps overhead; BentoCloud
managed option available
APAC PyTorch production team → TorchServe PyTorch-native MAR format;
(AWS/SageMaker ecosystem) → APAC multi-model GPU sharing;
SageMaker APAC integration
APAC ML platform (shared K8s infra) → KServe InferenceService CRD; APAC
(many teams, many APAC models) → serverless scale-to-zero;
canary rollouts; multi-framework
APAC LLM / large model serving → vLLM Continuous batching; APAC
(GPT-scale APAC transformer serving) → PagedAttention; APAC OpenAI
compatible API; APAC GPU efficient
APAC simple REST wrapper needed → BentoML Fastest APAC path from APAC
(prototype to APAC API quickly) → notebook to APAC HTTP endpoint
Related APAC AI and ML Resources
For the ML pipeline orchestration tools (Kubeflow, Ray, Spark) that produce the APAC trained models these serving frameworks deploy, see the APAC ML infrastructure guide.
For the LLM-specific serving frameworks (vLLM, Ollama, LiteLLM) that handle APAC large language model inference at scale, see the APAC LLM inference guide.
For the feature store platforms (Feast, Tecton, Hopsworks) that supply APAC real-time features to these inference endpoints, see the APAC feature store guide.
Beyond this insight
Cross-reference our practice depth.
If this article matches your stage of thinking, the underlying capabilities ship across all six pillars, ten verticals, and nine Asian markets.
Other service pillars
All industries
Asian markets