AI orchestration and model routing
Responsibilities
The AI/runtime orchestration plane tracks jobs, endpoint capabilities, worker heartbeats, routing, build-spec sessions, app seeding, cloud-burst paths, and tenant-container health.
Execution patterns
- Live embedding/reranking can resolve synchronously to a healthy endpoint.
- Background work can be queued and leased by workers.
- Different capabilities can route to different hardware/model classes.
- Managed, local, or burst capacity can sit behind the same application contract.
Design goal
Application logic should not be coupled to one GPU vendor, public API, or inference host. Provider/model identity remains observable for rebuilds, diagnostics, evaluation, and audit.