Running open-weights AI models locally can seem straightforward until you encounter unfamiliar technologies and acronyms such as GGUF, MLX, and CUDA. Why can a 4-bit quantised model fit comfortably into memory but run out of unified RAM as a conversation grows? And do techniques such as speculative decoding, multi-token prediction (MTP), and draft models deliver meaningful performance improvements?

In this session, you will explore what happens between the hardware running a local model and the system prompt that guides it. You will examine the architectural distinction between low-level inference engines, such as llama.cpp and oMLX, and developer harnesses, such as Hermes Agent and Pi. You will also explore how context windows, KV caches, and quantisation choices affect memory consumption, performance, and reasoning quality.

The session examines approaches to improving local inference performance, including speculative decoding and multi-token prediction, and how to assess their practical value. You will leave with a diagnostic toolkit and a practical guide to configuring, benchmarking, and tuning local models for speed, coherent output, and efficient hardware use.

What You Will Learn

• How local inference engines and developer harnesses differ and work together

• How model parameters, quantisation, context windows, and KV caches affect memory use and reasoning quality

• How speculative decoding, multi-token prediction, and draft models can influence inference performance

• How to diagnose, benchmark, configure, and tune local AI models for available hardware

Who Should Attend

AI Engineers, Software Developers, Software Architects, Platform Engineers, and technical practitioners interested in running, configuring, and optimising open-weights models on local hardware.