The tools behind connected systems.

Evaluating and configuring locally hosted language models (Llama, Mistral and Qwen families) for application-specific inference, within the limits of the available hardware.

Experience: Ran Llama- and Mistral-family models locally through Ollama while prototyping an offline-capable assistant. Qwen-family models are being evaluated.

Works with: Python, Ollama API

Scripting, data handling and AI prototypes. Certified through HCL GUVI.

Works with: Flask, REST APIs

Integrating model-serving endpoints with applications, workflow engines and business software through authenticated APIs.

Experience: Called local model endpoints from app prototypes; authenticated endpoints for business software are at design stage.

Works with: Ollama API, REST APIs, Python

Comparing models on task quality, latency, context length, licence terms and running cost before choosing one for a workflow.

Experience: Comparing hosted and open-weight models on the same prompts during prototyping.

Retrieving relevant passages from approved documents with embeddings and a vector database, then grounding model answers in them.

Experience: Exploring retrieval pipelines for knowledge assistants; the design is documented in the Internal Knowledge Assistant blueprint.

Works with: Vector databases, Embeddings models

Running quantized model formats to fit larger models into limited memory, trading some quality for speed and cost.

Experience: Testing quantized models to compare answer quality against memory use and speed.

Containerised, reproducible model-serving environments on Linux hosts, with the model, runtime and API packaged together.

Experience: Containerising local model-serving setups so tests can be repeated on other machines.

Connecting controlled language-model inference with document retrieval, embeddings and permission-aware knowledge access.

Experience: Designed as the Internal Knowledge Assistant blueprint: answers drawn only from approved documents, with access following existing permissions.

Designing model-serving environments around GPU resources, inference endpoints, memory requirements and performance monitoring, on dedicated or cloud GPUs.

Experience: Planning GPU memory, latency, throughput and cost trade-offs; studying for NVIDIA AI Infrastructure and Operations. Not yet deployed for a client.

High-throughput serving with request batching and KV-cache memory management, exposed through an OpenAI-compatible API.

Experience: Studying serving architectures and their monitoring needs before a production deployment.