THEME:
guide@ai-foundations:~/

$ cat AI_FOUNDATIONS.md

AI Foundations

Understanding the systems you are working with

Guide version: 2026-10-08. View this edition's frozen snapshot · Version history

AI applications can answer questions, write and analyse text, work with images and speech, and carry out tasks through other software. Understanding how these systems work helps you give them useful instructions, recognise what information they need and judge the results.

This guide explains the foundations behind those capabilities. It covers how models learn and produce answers, what information they can use, how conversations and saved memory work, and what reasoning, context and generation cost. It then explains tools, connections and permissions, and how individual requests become repeatable work.

It is written for students, entrepreneurs, managers, employees and technical practitioners. You do not need to code, buy a subscription, have used an AI assistant or read another guide first. Technical terms are introduced as their meaning becomes useful. An optional later track explains hosted and local models, hardware requirements, speed and cost.

The aim is practical understanding. By the end, you should be able to explain why an assistant behaved as it did, choose a sensible starting setup and make a short sketch of an AI-assisted job: its purpose, sources, tools, identity and what counts as done.

Four distinctions organise the guide: learned capability, available information, permitted action and verified result. A system can be good at writing without having the right facts; it can have the facts without permission to act; and it can attempt an action without completing it. Understanding those differences makes useful work easier to direct.

The explanations come first. Illustrative examples apply the ideas, and a worked use case later brings them together. This is not an exhaustive product manual, a model ranking or a complete treatment of AI governance. The related guides develop software delivery, bot-managed work and security in greater depth.

I update this guide through teaching, practical experience and reader feedback. The conceptual foundations should outlast individual product releases; examples, capabilities and prices need updates. Dated editions preserve what earlier readers and classes used. See the version history for the published editions and their changes.


Part 1: Learned capability

1. The application and the model

Artificial intelligence, or AI, covers systems used for tasks such as prediction, classification, generation and planning. Predicting how busy a shop will be, recognising speech in a voice message and producing a written answer are different uses of AI.

This guide concentrates on generative AI and connected assistants: applications that can produce text, images, audio or code, and increasingly use other software to do work. Familiar uses include drafting an email, translating a message, explaining a document, describing a photograph and turning spoken notes into text.

What happens when you send a message?

Suppose you paste a document into a chat and ask for a summary. You are using an application: the software that provides the chat window and handles your request. The application uses a model, the trained part of the system that processes the supplied information to produce a result.

The application prepares what to send to the model. This can include your request, the document, relevant earlier messages and instructions supplied by the application itself. The model processes that material and generates a result. The application then displays it or uses it in another step.

The model may run on your computer or on a remote service. It does not have to run inside the application on your device. The provider is the organisation operating the application or model service; these can be different organisations.

1. Your request

A question or task, with relevant information you supply.

2. The application

Prepares instructions and information for the model.

3. The model

Generates an answer or a request to use an available capability.

4. The application

Shows the result or carries out an allowed next step.

The material supplied to the model is its input. The result it produces is its output. In the document example, your question and the document text are input; the summary is output. Later, an output might instead ask the application to search for information.

A request to the model and its generated response form a model call. One exchange in the chat can involve several calls if the application obtains more information between them.

Two applications using the same model can therefore behave differently. Imagine asking about yesterday's news. An assistant with web search can obtain recent reporting; one without current sources cannot establish what happened yesterday from old training alone. A visible searched the web step is the application obtaining information, not the model suddenly acquiring new training.

How a model learns

A model develops its capabilities through training. Training software presents examples, measures how the model's results compare with the training goal and adjusts numbers inside the model.

For a typical text-generating language model, the initial broad learning stage is called pre-training. A common task is to predict the next piece of text from the text that comes before it. The training process compares that prediction with the actual continuation and adjusts the model.

For a simplified word-level example, suppose the training text contains The train arrived late. The model receives The train arrived and predicts what follows. The actual continuation supplies feedback for that training step. Repeated across a large collection, this process develops patterns of language, facts, writing styles and other relationships.

The adjustable numerical values are called parameters; many are called weights. Training changes these values so that later predictions better match the training goal. Together with the model's computational structure, they influence how it processes information. They are not the same as settings you choose in a chat application.

Further training after that initial stage is called post-training. It can improve instruction following and shape the style of answers or when a model refuses a request. Trainers might provide examples of useful answers or feedback about which of two answers better follows an instruction. Some methods reward results that can be checked, such as a correct calculation. These training steps adjust the model's learned values again.

Broad practice followed by focused practice with feedback is a useful way to picture the sequence. Evaluation on separate tasks measures the result; taking an exam and learning from feedback are different operations. Training approaches vary. GPT-4's technical report describes the initial prediction task, while the InstructGPT research describes demonstrations and human feedback in post-training.

Learned weights can encode patterns and memorised training material, including text a model may reproduce. They are different from a searchable collection of current records. Training-text extraction research demonstrates that memorisation can occur. A model's ability to discuss a subject does not establish that it has the latest information about it.

Using a trained model: inference

Once trained, the model can process a new document and produce a summary from it. Using a trained model to process new input and produce a result is called inference. An ordinary request does not retrain the model's weights with the document you supplied.

A large language model, or LLM, generates language as sequences of tokens: small units of text, such as words or parts of words. Section 2 explains how they are counted and why that matters.

In a common generation process, the model calculates possibilities for the next token from the available input and the tokens already generated. The system selects a continuation and repeats. This can produce useful explanations, summaries, translations and code. Producing the answer does not, by itself, check each statement against reality.

Why does AI sometimes give an answer that is not true?

Imagine asking when a nearby shop closes today without providing its current opening hours or enabling a lookup. In this illustration, an assistant might answer The shop closes at 8 pm without a source for that time. It might instead correctly say it needs the current opening hours. The first answer sounds useful, but its precise wording does not establish that it is true.

Training develops patterns of how useful answers are written. When information is absent, ambiguous or misunderstood, the model can fill the gap with a familiar pattern rather than recognise the gap. It can also misread a source that was supplied or draw a faulty conclusion from it. An unsupported or incorrect statement generated this way is commonly called a hallucination.

Training can improve factual behaviour, but fluent language remains different from verified evidence. Relevant sources, suitable calculation tools and checking important claims can reduce errors. Asking Are you sure? does not supply missing facts, and a confident answer is not a measure of accuracy.

Instructions and the conversation

A prompt is an instruction or other input supplied to guide the model. Your question is one part of what it may receive. A system prompt supplies overarching instructions, such as the assistant's role and operating rules, separately from an ordinary user message. The application may also add information and output requirements you cannot see in the chat.

A useful instruction explains the purpose, audience, relevant source and expected result. An example of the desired output can clarify the format. Once the substance is right, you can ask for a shorter or warmer version rather than restarting the task.

A conversation feels continuous because the system makes relevant earlier messages or stored information available for later responses. If you write Make it shorter, the model needs enough of the previous exchange to know what it means. This does not require one continuously thinking model instance to stay with you. What matters is which information the application or model service makes available at each step.

Provider documentation: conversation state.

Further depth: architecture and training methods

A model's architecture describes how its computation is organised. Many language models use transformer-based architectures, in which attention mechanisms help relate parts of the input. Attention here is a mathematical operation, not evidence of human awareness.

Fine-tuning further trains an existing model using additional examples. One post-training technique is reinforcement learning from human feedback, or RLHF, which uses human preferences to guide a reward signal. Other approaches use supervised examples, AI feedback or rewards based on checkable results. RLHF is not a description of every model's training process.

Distillation can train a model using outputs or other supervision from another model. These methods change learned behaviour. Supplying a document in a conversation usually changes the current input, not the weights.

Further reading: transformer research, fine-tuning and distillation and reasoning-oriented reinforcement learning.

Worked use case: processing customer feedback

Illustrative scenario. I use a customer-feedback exercise in class to explore how the parts of an AI system fit together. For this guide, the use case is a fictional restaurant chain called Mikko's Hamburger Joint. The business and messages below are invented teaching material.

The chain receives feedback about food, recipes, service and the condition of individual restaurants. Its staff want to understand each message and get it to the right person: perhaps the restaurant manager, the food-development team or a regional manager. Routine praise may need only a thank-you; other messages require investigation or a decision.

We will use parts of this scenario to illustrate particular concepts, then assemble the complete workflow in section 9. Other examples explore the same fictional business's cleaning contract, ingredient deliveries and recruitment. Each example introduces the records and task it uses. The purpose is to show how information, interpretation, tools and authority combine in a useful system.

A first request, before automation

A feedback coordinator receives this fictional email: The burgers were excellent and the staff were friendly. However, the tables at your Harbour restaurant were dirty when we visited on Saturday. Harbour is one of the fictional chain's branches.

The coordinator pastes the email into an assistant. Deal with this feedback is vague: does it mean summarise, reply or contact a manager?

A clearer instruction is: Summarise the customer email I just pasted for an internal review. Separate praise from matters that need attention. Include the restaurant and visit information if supplied, and flag anything missing. Return the summary and relevant words from the email. Do not send anything.

A format example can help: Use these labels: Summary; Matters to handle; Missing information. A useful result preserves both the praise and the cleanliness report, without inventing a precise visit date or claiming the issue has been fixed.

Optional experiment: use this fictional email to compare the vague and clearer requests. Check the retained information, not just the polish of the answer. This demonstrates instruction shaping; it does not test a connected feedback service.


Part 2: Available information

2. Tokens and the context window

When an assistant answers, it needs the information relevant to that response. If you ask it to shorten a draft, the draft must be available as well as your new instruction. The information supplied for the current model operation is its context.

The context window is the working capacity available for that operation. Its maximum is the context-window limit. Think of the material available for the current piece of work, not an archive containing everything the application has ever stored.

That capacity is usually measured in tokens. A tokeniser converts text into these small units, represented by numbers. A token may correspond to a word, part of a word, punctuation or another unit. A page of prose, source code and a table of identifiers need not occupy the same number of tokens.

Different languages, writing styles and tokenisers produce different counts. Use the relevant system's counter when capacity or cost matters. A measured example appears below; there is no universal words-to-tokens conversion.

Many systems must accommodate both supplied context and generated material within the window, with a separate output-token limit for generation. Section 4 explains the practical difference.

A message such as this conversation is too long is one way an application can expose a limit. Another may shorten the history without showing that message. The screen can still display older messages that are no longer included for the next answer.

The chat window does not show all the information being used. Depending on what the application supplies, space can be occupied by:

  • Application instructions, your current message and selected conversation history.
  • Earlier answers, retrieved passages, uploaded document content and saved preferences.
  • Descriptions of available tools, tool requests and returned results.
  • Images, audio or other supported inputs, represented and accounted for according to the model.
  • Reasoning and other generated material that the particular system retains.

Connecting the restaurant chain's feedback archive does not place every message into the window. The application must select and supply relevant content. Conversely, one tool that returns an entire case history can consume substantial space even when your request was one sentence.

A context-window example. Return to the fictional restaurant-feedback use case. A coordinator summarises an email, asks for a manager note, then asks the system to find the recipient. The third step includes a tool request: a request for other software to perform a lookup. The snapshots illustrate accumulating context, not a complete operating procedure.
InstructionsUser messageTool requestSource informationReasoningAssistant reply
1. Understand one email

Context window: the same capacity limit at every stage

Supplied to the model

  1. Application instructionsSummarise customer feedback. Preserve important details. Do not send messages.
  2. Your requestSummarise this email. What needs attention?
  3. The email you pastedThe burgers were excellent and the staff were friendly. However, the tables at your Harbour restaurant were dirty when we visited on Saturday.

Generated in this model call

  1. Assistant replyThe customer praises the burgers and staff, and reports dirty tables at Harbour on Saturday. The cleanliness report needs attention.

Unused capacity remains.

2. Prepare a note for the manager

Context window: the same capacity limit at every stage

Supplied for the next reply, including retained history

  1. Application instructionsSummarise feedback. Preserve important details. Do not send messages.
  2. Earlier requestSummarise this email. What needs attention?
  3. Earlier source emailPraise for the burgers and staff; dirty tables reported at Harbour on Saturday.
  4. Earlier replyThe customer praises the burgers and staff, and reports dirty tables. The cleanliness report needs attention.
  5. New requestPrepare a short note for the restaurant manager. Include the customer's wording about the tables.

Generated in this model call

  1. Assistant replyA customer praised the burgers and staff but reported that “the tables at your Harbour restaurant were dirty” on Saturday. Please review the report.

More material is present; the capacity limit has not grown.

3. Look up where the note should go

Context window: the same capacity limit at every stage

Supplied for the reply, after a directory lookup

  1. Application instructions and available toolPrepare drafts only. A directory tool can look up the approved recipient for a restaurant.
  2. First requestSummarise this email. What needs attention?
  3. Original source emailPraise for the burgers and staff; dirty tables reported at Harbour on Saturday.
  4. First replyThe cleanliness report needs attention.
  5. Second requestPrepare a short note for the manager, including the customer's wording.
  6. Prepared noteA customer praised the burgers and staff but reported dirty tables at Harbour on Saturday. Please review the report.
  7. New requestCheck the current directory. Who should receive this note?
  8. Tool requestLook up the approved feedback destination for the Harbour restaurant.
  9. Directory resultRestaurant: Harbour. Destination: Harbour manager's feedback queue.

Generated in this model call

  1. Reasoning, where usedWork on the response; it is not necessarily shown in the chat.
  2. Assistant replyThe directory identifies the Harbour manager's feedback queue. The note is still a draft; it has not been delivered.

Instructions, retained conversation, lookup results and generation all use capacity.

Illustrative: blocks identify content types, not token counts. Some repeated text is shortened here for readability, not as a claim that the application compacted it. One exchange can involve several model calls; the history and reasoning retained vary by system.

Text brought into context can also contain attempted instructions. Treat a retrieved email as information about the task, not permission to change it. The AI Security Primer and OWASP's guidance explain this influence problem, called prompt injection.

Multimodal means a system can work with more than one kind of input or output, such as text and images. It does not mean every model can process every file type. When the restaurant uploads a supplier's PDF contract, an application may extract its text, render its pages as images, retrieve selected passages or reject it. Those routes differ in cost and in what information survives.

Why "I uploaded it" is not enough

At the fictional Mikko's Hamburger Joint, the operations manager is reviewing the annual contract with its cleaning company. The supplier has proposed a price increase at the first renewal. The manager uploads the contract and asks: Does this agreement allow the cleaning company to increase its price at the first renewal?

In this illustrative setup, the application stores the file, searches it and supplies selected passages to the model. It supplies the general clause allowing an annual increase but misses an appendix freezing the price for the first renewal. The assistant answers that an increase is allowed, using the incomplete material it received.

The exception was in the uploaded file, but not in the model's current context. Relying on the answer could lead the manager to accept an increase the contract does not permit. Uploading made the file available to the application; it did not establish that the model considered every relevant provision. Other applications may supply the whole document or process its pages differently.

Before relying on the answer, inspect the cited passages and the relevant appendix, then have the answer checked against both. The assistant's claim that it read everything is not proof. More confident wording cannot compensate for an exception it never received.

A measured sense of token size

The four paragraphs in the cleaning-contract example contain 206 space-separated words and measured 237 tokens with one tokeniser. They are short enough to read quickly. A long collection of feedback emails, attachments and repeated conversation can occupy much more capacity.

Measured October 7, 2026 with tiktoken 0.12.0, cl100k_base: plain paragraph text joined by blank lines, excluding the heading, HTML and application instructions. This is a measurement of that text, not a conversion rule for other languages, documents or models. See tokenisation explained and the counting method.

A larger window can help with long material, but capacity is not a guarantee of effective use. Relevant facts can be obscured by repeated, conflicting or unrelated content. When an answer deteriorates, consider what the application supplied before assuming the model needs still more text.

Provider documentation: Anthropic's context-window explanation describes the contributions of instructions, tool use and thinking.


3. Memory and compaction

Suppose the restaurant manager reviews customer feedback with an assistant today and returns tomorrow. Opening the old conversation may let the manager continue from the earlier messages. A fresh chat may still know a saved writing preference but have no record of yesterday's customer emails or unresolved cases. What carries over depends on what the application saved and what it supplies for the new response.

The model's learned weights provide capabilities acquired during training. The context contains information available for the current response. The application may separately store conversation history, preferences and working records. Stored information must be brought back into use to influence a later answer.

Persistent memory is information kept for later use, usually by the application or an external service. A memory settings page may let you inspect or remove saved preferences. Those settings concern stored information, not the model's learned weights. To influence a later answer, the relevant information must be brought into context.

An ordinary inference call does not retrain the model with your conversation. The application creates continuity by bringing back selected history, saved information or working records. It may store these itself or use a provider's conversation service. It can also shorten what it supplies, which is why the saved chat and the next working context can differ.

Files that carry instructions and working records

A file ending in .md is a Markdown document: plain text with simple formatting for headings, lists and other structure. People and AI applications with file access can read and update it. The file can hold instructions, saved facts or working notes; its purpose depends on what you put in it and how the application uses it.

For another task in the same business, Harbour's kitchen manager is waiting for a supplier to confirm an incoming burger-bun delivery. A Markdown working note could look like this. The heading and labels are plain text, not a special programming language:

# Harbour ingredient delivery

Task: prepare an update for the kitchen manager
Status: draft saved; supplier delivery date needs confirmation
Authority: sending is not approved
Next step: obtain the supplier's current confirmation before final review

Files can serve different purposes. These are examples you may encounter:

  • AGENTS.md: how to work in a project. For this task, it might say where to find supplier confirmations, how to check an update and when to ask before sending it. Codex and other supporting tools recognise this filename for project guidance.
  • MEMORY.md: information intended for later use. It might record that the kitchen manager prefers concise internal updates. Whether this filename is loaded automatically depends on the application and its configuration.
  • project-state.md: where the work stands. It could hold the task, status and next action shown above. This is an example filename chosen for the project, not a universal standard.

These are examples, not required filenames or a document hierarchy. Some products keep comparable information in databases or built-in memory features. A supported instruction file may load automatically; another document may need to be requested. Check the application's loading rules. Saving a file preserves its contents; only content brought back into context can guide the next response.

Provider documentation: AGENTS.md project instructions.

What compacting does

A long conversation may leave too little room for the next task. The application can shorten what it carries forward, often by making a summary. This is called compaction. Some systems use a specialised representation rather than ordinary prose. Compaction makes room within the same context window; it does not increase the limit.

Do not assume compaction is a lossless copy of the conversation. A qualification, rejected option or reason behind a decision may be omitted. Summaries can also preserve a mistaken interpretation. The old conversation may remain in the application's storage while no longer being available in full to the model.

For work that matters, keep the current objective, confirmed decisions, important evidence and unresolved questions in dependable records. Customer commitments, permission to send and unresolved disagreements should not exist only in a conversational summary. After compaction or a new session, retrieve what the next task needs. A fresh conversation removes available context as well as clutter; give it the current task and records rather than merely saying continue.

A preference is not a promise

Imagine preparing an update for Harbour's kitchen manager about the burger-bun order. Yesterday the manager asked the assistant to keep these updates short. Friday was discussed as a possible delivery date, but nobody confirmed it. Today the order record links to the supplier's confirmation of Tuesday delivery.

A saved preference can supply the short writing style. The current confirmation supplies Tuesday. A summary saying discussed Friday delivery is not a supplier commitment to Friday. The update should report what the supplier confirmed, not claim that the buns have arrived. If the current record cannot be obtained, the assistant should flag the missing confirmation instead of turning yesterday's discussion into a promise.

Try it: use the fictional email in the restaurant-feedback example and give the assistant a writing preference. Ask for a summary, then make the same request in a fresh chat. Note which information is available in each: a new chat may still receive saved preferences. Supply the email where it is missing and compare the result. The lesson is to inspect what carried over, not to assume that every fresh chat starts empty.

Provider documentation: OpenAI conversation state, OpenAI compaction and Anthropic context management.


4. Reasoning, budgets and cost

Your application may offer a faster mode, a thinking mode or a choice of models. What are you changing? A model choice can change learned capability. A thinking control can change how much computation a supported model spends working through a problem. A product's mode may change several settings at once, so check its explanation rather than assuming the label describes one mechanism.

Additional thinking is often exposed as reasoning effort, thinking time or a reasoning budget. It can help compare conflicting records, trace dependencies or check a proposed solution. It cannot recover a missing fact merely by spending more time on it.

Temperature and reasoning effort do different jobs

Temperature, where available, changes how the model selects among possible continuations. Lower values favour more probable continuations more strongly; higher values allow more variation. For a restaurant's reply to customer feedback this can change wording, but it can also change substance or quality. Temperature is not a truth setting.

Reasoning effort concerns work spent on the problem. If the supplier's messages about Harbour's ingredient delivery disagree, extra reasoning may help compare their status and evidence; increasing temperature does not request that comparison. Neither control supplies a missing record. Some applications expose these choices, while others manage them for you.

How temperature changes generation

The model assigns probabilities to possible next tokens. Sampling selects a token using those probabilities; temperature changes how concentrated the selection is. It does not change learned weights. Even a low setting is not a universal guarantee of identical replies: other settings and the execution system also matter.

Provider documentation: generation and sampling controls and thinking controls.

Which limit are you changing?

A long document may not fit alongside the conversation: that is a question of context capacity. An answer may stop before it is finished even though that window still has room: generation has its own limit. A higher thinking setting may spend more work on the problem without increasing either limit.

A complete task can also involve many model calls, lookups and retries. A limit on that whole task is different from a limit on one response. A spending limit answers another question: how much money may be used? Check which resource a setting controls and where the application or service actually enforces it.

The word budget therefore needs a qualifier. The table separates these limits:

Capacity, work and spending are different limits
LimitWhat it boundsExample consequence
Context-window limitThe current working capacity.Room for instructions, records, retained messages and generation.
Output-token limitGenerated material for a response, including reasoning in some systems.The response can stop before the review of unresolved feedback cases is complete.
Reasoning budgetInternal reasoning work where a budget is supported. An effort level need not be an exact token cap.Work spent comparing conflicting records before answering.
Task budgetA whole task, potentially involving many calls and tool operations.Resources allowed for looking up, drafting, checking and retrying.
Monetary budgetSpending, as defined and enforced by the application or service.How much the work may cost, not how much the model can see at once.

An API, or application programming interface, is a defined way for one piece of software to request information or actions from another. Many model APIs count generated reasoning tokens within the output-token limit as well as the context used during processing. Increasing reasoning effort can therefore use more time, tokens and available capacity without making the final answer longer. Whether earlier reasoning is included in later requests depends on the model and application; it is not safe to assume it is always kept or always discarded.

One window, several uses

The restaurant chain's regional manager asks for a supported review of unresolved customer feedback across its branches. The application supplies a large collection of emails, case histories and instructions. The model must work through those records, including disagreements about what has been handled, before preparing the review.

Calculated illustration, not a model specification: imagine a system with a 100,000-token context-window limit and a 25,000-token output-token limit that includes reasoning. It receives 70,000 tokens of instructions, history and source material. A response using 20,000 reasoning tokens and 5,000 visible-answer tokens reaches the output-token limit, even though 5,000 tokens of window capacity remain.

70,000: supplied context

20,000: reasoning

5,000: visible answer

5,000: unused window capacity

If the response is unfinished when it reaches the output-token limit, generation stops. The regional manager might receive only part of the requested review even though the context window still has space. In a metered API, work already consumed can still be charged even if no useful answer is returned. More room in the window does not override the output-token limit, and a higher reasoning setting does not enlarge either limit.

Now imagine the application completes the task through several model calls. The total tokens processed over that task can exceed the size of any one context window. Each call has its own working set, and earlier material may be repeated, removed or compacted. A task budget of 300,000 tokens does not imply that the model can see 300,000 tokens at once.

The relationships above are illustrated by OpenAI's reasoning documentation. Check the selected model's accounting rather than transferring a setting or limit from another provider.

Why a short answer can cost more

A reply to the Harbour customer may be only three sentences, but preparing it can involve reading the original complaint, checking its case record and retrieving the manager's response. A failed lookup or conflicting updates can add more work. The visible answer is only part of the work. Usage can include input processed again on later calls, reasoning, output and separate tool charges. A flat subscription may hide those components behind usage allowances rather than showing an API bill.

Latency is the delay before or during a response. Throughput describes how much work a system processes over time. Fast token generation helps, but a task may spend most of its time waiting for a database, a browser, another service or a human decision. Measure completion of the useful task, not just the speed of words appearing. The deployment track explains tokens per second, the wait before the first token and the hardware behind those measurements.

Three different things called a cache

A prompt cache reuses processing of a repeated input prefix, such as the chain's common feedback-handling instructions. It can reduce cost or processing time, but the reused input still forms part of the model's context. A KV cache is a different mechanism inside the model's execution engine: its practical cost is working memory that can grow with context.

An answer cache stores a completed response for reuse. That can avoid another model call, but yesterday's answer about a complaint may be stale after the restaurant manager updates its case. A cached response may also be inappropriate for a different user's permissions. These caches solve different problems. None is a general promise that the assistant remembers everything.

Technical references: prompt caching and KV cache strategies.

When is more reasoning worth trying?

Harbour has ordered burger buns from a supplier, and the kitchen manager wants to know when they will arrive. The following variations compare a straightforward extraction, a missing fact and a conflict. They illustrate how to test a setting, not a claim that one setting always wins.

A stated fact: the supplied confirmation for the bun order says Confirmed delivery: Tuesday, and you ask the assistant to return the date and its source. Try the default or a lower supported effort setting and check whether it extracts Tuesday correctly and identifies the right record. Then try a copy with the date left blank: it should flag the missing confirmation rather than guess. If a higher setting produces the same checked results but takes longer, the extra work has not helped these tasks.

Conflicting evidence: the supplier's earlier confirmation says Tuesday. A later supplier message reports a warehouse delay that may push delivery to Thursday, while the restaurant's purchasing record still says Tuesday. You ask what the kitchen manager can safely be told. More reasoning is worth testing here: can the assistant distinguish the earlier commitment from the warning of a possible delay, preserve both and prepare a question for the supplier? The useful result is that Tuesday was confirmed but is now in doubt, with Thursday mentioned as a possibility. Neither date should be presented as assured before clarification. Extra thinking cannot obtain that clarification from a missing supplier response.

Compare settings on the same supplied material and the same task. Include a straightforward case, a conflict and a missing record. Note whether the answer is usable after checking, how long you waited, any visible usage cost and how much correction you had to do. If additional effort consistently catches an important conflict that the quicker setting misses, it may be worth the delay. If both fail because the order record was never supplied, fix the information path rather than increasing effort again.

Use the same comparison when choosing between models or a faster and a thinking mode in your app. Change one choice at a time where the interface allows it, and record the model or mode used. The useful choice is the one that handles your ordinary cases and important exceptions at an acceptable total cost, including your time correcting it.


5. Working with business information

Useful business work often depends on current facts: today's order status, the applicable contract or a recent internal decision. An assistant updating Harbour's kitchen manager needs the current supplier confirmation for the burger-bun order. A connected application can retrieve the order record and related supplier messages, supply them to the model and ask it to prepare the update from that information.

Obtaining relevant material from a source for the task is called retrieval. Combining that material with the model's generation of an answer is called retrieval-augmented generation, usually shortened to RAG. The lookup supplies the current fact; the model helps use it. The system still needs to find the right record, recognise its date and scope, and preserve important qualifications.

A training cutoff describes a boundary in the training information used for a model. It is not a guarantee that the model knows every earlier fact or has a live view of later events. The application can supply today's date, but knowing the date is not the same as knowing what happened today. It does not tell the assistant whether Harbour's supplier has just confirmed a delay. Web search is a form of retrieval: it can bring newer public evidence into context without changing the model's weights, but it is not a substitute for access to the private order correspondence. Retrieved evidence still needs checking for relevance, reliability and freshness.

An authoritative record is the record the organisation relies on for a defined fact or decision. A model can summarise it, calculate from it or draw an inference, but the result should distinguish recorded facts, calculated results and inferred conclusions. A generated summary does not become a competing original. Source links help readers verify claims, but the presence of a citation does not prove that the cited passage supports the answer.

A search of the chain's feedback archive for cleanliness complaints might need to find another customer's message saying our table had not been wiped. Searching by meaning can help when the wording differs. An embedding represents features of the material as a list of numbers that supports these comparisons. A vector database stores and searches such representations. Retrieval can also use ordinary keyword search, filters or a direct database query; RAG does not require one particular database product.

Similarity is not authority. Suppose the operations manager asks about payment terms for the restaurant's cleaning contract. A search returns the cleaning supplier's earlier proposal offering 60 days, while the signed agreement says 30. The proposal's wording matches the question, but it does not establish the agreed terms. Retrieval needs the applicable agreement and enough surrounding text to interpret it.

When preparing a business message, take the facts from the appropriate records and use the model to help express them. The distinction matters when several systems describe different parts of the same transaction.

Example: three systems, three different facts

Return to the straightforward bun-order example, where the supplier has confirmed Tuesday and no delay has been reported.

The kitchen manager wants an update on the burger-bun order, including whether the supplier has been paid. The purchasing system records that the order was placed. The supplier's order portal confirms Tuesday delivery. The restaurant's accounting ledger shows the invoice is still unpaid.

A draft saying The buns are paid for and delivery is confirmed for Tuesday would confuse a placed order with a completed payment. The assistant needs the supplier confirmation for delivery and the ledger for payment status; the purchasing record alone does not answer both questions.

If a colleague says payment was made but the ledger still shows unpaid, retain that discrepancy and ask the person responsible for payments to check it. Do not turn conflicting records into a confident statement that nobody has verified.

The same principle applies to customer records. The chain might keep customer conversations in a customer relationship management system, or CRM. That would not make it the source for whether an ingredient invoice has been paid. Each system has a defined job.

Adding knowledge or changing behaviour

Fine-tuning further trains an existing model on additional examples. It addresses a different need from supplying information for a task:

Choose the route that matches the need
Your needStarting routeExample
Use a known piece of information now.Supply it as context.Provide the supplier's confirmation for one ingredient-delivery update.
Find relevant information in a changing collection.Retrieve it from its source.Look up the current ingredient order and supplier messages each time an update is prepared.
Improve a recurring pattern of model performance.Test instructions and examples first; consider fine-tuning for an evidenced remaining need.Evaluate whether training on suitable examples improves extraction of topics and supporting passages from varied restaurant feedback.

These routes can be combined. Training a model to extract feedback more consistently does not supply new customer messages or tell it which cases the restaurant manager resolved today.

Uploading the chain's feedback-handling policy is not, by itself, fine-tuning. Updating a search index is not retraining the language model. A product may separately retain inputs or use them in a later training process under its terms, so distinguish what happens during this response from the provider's subsequent use of data.

What happens to information you provide

Consumer applications, business subscriptions and API services can have different terms and controls even when they use the same model. Check the actual product, account configuration and agreement. Training use, retention, human access and onward processing are separate questions. Not used for training does not mean no copy is retained or that authorised staff can never review it.

An application can also keep its own logs or send content through another service. Establish where the information goes and what each party keeps. OpenAI's API data-control documentation provides one concrete example, distinguishing training use, monitoring logs and application state.

Further reading: Google's embedding explanation and the original retrieval-augmented generation research.


Part 3: Permitted action

6. Tools and connectors

Writing an answer and obtaining information are different operations. When an assistant checks a record, the surrounding software must perform the lookup and bring the result back. The following example shows that exchange before naming its parts.

Example: a lookup and its result

In the fictional restaurant-feedback use case, the coordinator has identified a cleanliness report about Harbour. The application is connected to the staff directory and permitted to look up feedback recipients:

  1. The coordinator asks the assistant to find the current feedback recipient for Harbour.
  2. The application describes the available lookup and the information it requires, including the restaurant name.
  3. The model requests the lookup with Harbour as the restaurant. If the location is ambiguous, the process needs clarification.
  4. The application or connected service checks the request and permissions, performs the lookup and returns a result or error. In this successful illustration, the result names the Harbour manager's feedback queue.
  5. The result is supplied to the model. The assistant can now explain the destination and identify its source. Looking up the recipient has not delivered the feedback.

The directory lookup is a tool: a capability the application makes available beyond generating an answer. Other tools can calculate a total, create a document or send a message. A tool call is the request to use that capability. The values supplied with it are its arguments; in this example, the restaurant name is Harbour.

The model can propose which tool to call and what values to supply. Software in the application, provider or connected service performs the operation and returns a result. That software must implement the checks on the request and its permissions. To establish whether I sent it is true, inspect the email service's result or the destination record, not just the model's sentence.

An API provides a software-to-software route for these requests. A connector packages a connection to a service, often handling its sign-in and available operations. It may expose several tools. A browser tool instead operates through a website interface. These routes can differ in reliability, coverage and permission handling.

Information returned by a tool also occupies context. Good tools return enough information to support the task without dumping an entire database into every response. Apply the same distinction between source content and instructions to tool results. A failed lookup should remain a failed lookup, not silently become an answer from model memory.

OpenAI's function-calling guide provides one concrete implementation of this request-and-result cycle.

A common language for connections

Model Context Protocol, or MCP, standardises how compatible AI applications communicate with services that expose capabilities such as tools and resources. An MCP server supplies capabilities; the application's client communicates with it. This can reduce the need to build a different integration for every application.

The standard does not make every system compatible with every other system, create an account for you or grant business authority. A connection still depends on supported features, credentials and permissions. A connector labelled "connected" proves less than successfully retrieving the expected record under the intended identity.

The official MCP architecture overview explains the parts and their responsibilities.

Skills, plugins and computer use

A skill packages reusable instructions and supporting material for a kind of work. It may include templates, references or scripts. In the Agent Skills format, a SKILL.md file could describe how to retrieve a restaurant's feedback record, prepare a note for its manager and check that the note was saved. The procedure tells the application how to use available capabilities; it does not supply a missing connection or retrain the model. See the Codex skill documentation for a product implementation.

A plugin is an extension package for an application. Depending on the platform, it may provide tools, connectors, skills or other features. Installing it and granting access are different steps. Names and packaging vary, so check what the extension actually supplies and what it is authorised to reach.

A computer-use agent interacts with an interface through capabilities such as viewing, clicking and typing. Browser agents can use page structure, visual observations or both. For example, the restaurant's assistant might use the case-management website when a suitable API is unavailable. Page changes, login steps and ambiguous controls can interrupt it. Verify the resulting record or action rather than treating a sequence of clicks as proof of success.

A connector can retrieve the feedback record, a document tool can save a draft note for the manager, and an email tool can deliver it. You can use the first two without granting the third. Grant the operations the current job needs, even if a connector offers more. Connections create useful options; they do not require all-or-nothing automation.


7. Identity, access and authority

When connecting the restaurant's feedback assistant, a consent screen may ask to read mail, see documents or send messages. Those are different capabilities, even if one button accepts them together. Before accepting, connect the requested access to the job: preparing a draft and sending it are different responsibilities.

Whose access would the connector use? Authentication establishes the identity making a request. Authorisation determines what that identity may do. Signing in establishes neither permission to see every company record nor permission to perform every action.

Credentials are used to establish or exercise access, such as keys or access tokens. An application can present an access token when requesting records from a service. Here, token means a credential, not one of the language-model tokens used to measure text.

Scopes describe permitted categories of access, such as reading mail. Record-level rules can narrow access further, for example to particular mailboxes. Consent may be one step in granting access, but a consent screen alone does not explain every downstream use.

With delegated access, an application acts on behalf of a user, constrained by both the permissions granted to the application and what that user is entitled to access. A broad application scope does not, by itself, give the user access to every record.

With an application or service identity, it acts as a non-human system under its own assigned permissions. The right arrangement depends on the task. A personal assistant and a shared overnight process need not use the same identity.

Provider documentation: Microsoft's authentication and authorisation overview.

Access makes delegation possible

A feedback assistant that can read the designated inbox and relevant restaurant cases can save the coordinator time. It does not need access to staff payroll records. Permission to draft a reply need not include permission to send it, issue a refund or promise that a complaint has been resolved.

Business authority and technical access must agree. Suppose the chain later decides to add limited refund handling. Telling the assistant it may issue a small refund does not give the payment service a valid identity or enforce a limit. Equally, an overpowered credential can technically permit actions the business never approved. The application must constrain the operation, not leave the entire boundary to interpretation by the model.

The practical starting point is to describe the capability you need and the actions you authorise. Complete personal login or consent through the service's supported interface when required. Keep credentials in the appropriate secure settings, rather than pasting them into ordinary chat or project notes. Check an expected read before considering a bounded test action.

Who acts when a weekly review of restaurant feedback runs and the coordinator is not at the keyboard? Name the identity configured for that run, whether delegated user access or a service identity. Check its access to the relevant cases and its separate authority to send the review. A schedule starts work; it does not supply a new identity or extend permission. If the required access has expired, report the limitation rather than silently borrowing a more powerful account.

Access also has a lifetime. Revocation withdraws it; expiry ends a credential's validity. Stopping the assistant is not the same as ending its token or session. An issued token may keep working until expiry or the service enforces revocation; withdrawing access does not erase copies already made. Microsoft's revocation guidance explains the timing and enforcement detail.

Try it without granting new access: read a connector's documented permissions or review an existing connection's settings. Identify whose account it uses, what it can read and what it can change. Which capabilities would the restaurant's assistant need if its job were only to read feedback and prepare draft notes? Leave any new consent unapproved.


Part 4: Verified result

8. From a conversation to recurring work

A chat response is one result. If the same work is needed again next week, something must start it, obtain current information, prepare and deliver the result, and record what happened. Last week's answer does not perform those steps on its own.

Suppose the regional manager wants a weekly review of unresolved restaurant feedback. Software must notice the scheduled time, retrieve current case records, request the necessary analysis, save the review and confirm delivery. It must also record incomplete work so a later attempt can continue appropriately. These responsibilities explain the difference between asking a question and arranging a process.

A workflow describes a process. Some steps follow fixed rules; others use a model's judgement. An automation starts work in response to a trigger, such as a scheduled time or an incoming message, without a fresh instruction each time. Neither every workflow nor every automation needs AI.

These distinctions affect how you build. Interpreting an unclear message may benefit from a model. Looking up an already identified record or sending an approved fixed response may be ordinary software work. A process does not need a new bot for each step.

An agent commonly combines a model, tools and a task loop that can select further steps. The surrounding execution software is sometimes called a harness. It assembles context, manages calls, keeps track of progress and handles permissions and stopping conditions. Products use these terms differently, so inspect the actual behaviour.

Progress and completion

State records where the process is: which items were handled, what remains pending and whether an action completed. It lets a later run continue without guessing from an old conversation.

A useful completion record says what happened and how it was checked. No open feedback cases, case system unavailable, summary prepared and summary delivered are different outcomes. Keep the relevant case references, actual delivery or approval status, unresolved issues and next step.

A run that starts twice should not send two identical messages. If an attempted delivery has an uncertain result, check the destination before retrying. A run that never starts cannot reliably report its own absence; a separate scheduler or monitoring check must detect the missing completion. If that capability is unavailable, state the limit before claiming unattended operation.

A Markdown working note can support a small project; a running service may store its state in a database. The useful property is that the next run can find the current record and distinguish completed work from an attempt. A replacement agent should not depend on its predecessor's memory alone.

Delegating work

One agent can request work from another when the platform supports that arrangement. Orchestration coordinates assignments, dependencies and returned results. Separate agents may have separate contexts and permissions. Sharing a computer or giving them job titles does not establish communication or access.

For the weekly review, one worker might prepare branch summaries and another check them against the cited case records, if that separation improves the result. More agents also mean more coordination. They may repeat the same assumption or pass incomplete evidence between themselves. Use a specialist or a separate reviewer when it improves the result, not because every task needs an organisation chart. Adding up case totals may need one calculation tool call, not a new finance bot.

The person receiving the result needs a useful account of what happened, what remains unresolved and any decision required. Routine work can finish under standing authority and wait for the next trigger. Constant conversation is not evidence that the work is being managed.

We now have the parts needed to understand a complete process: learned capability, available information, permitted operations, a trigger and records of progress. The next section explains how to judge the result, then applies those parts to the restaurant-feedback use case.


9. Choosing and judging a useful system

Start with a job worth doing. Then ask what information, capabilities and evidence that job requires. This gives model choice, context management and integration a purpose. Buying more context does not resolve an undefined business objective; choosing the strongest model does not repair a connector returning another restaurant's case instead of Harbour's.

An evaluation checks performance against defined expectations. Useful checks include ordinary work, changed facts, conflicting records, unavailable data and a request outside the assistant's authority. Inspect the result and the route taken, not only whether the message sounds professional.

A benchmark reports performance on a particular collection of tasks under particular conditions. It can inform selection, but it does not establish suitability for your data, tools and approval process. Compare candidate configurations on representative work, including exceptions. Keep enough of the configuration and results to understand what changed when behaviour improves or deteriorates.

Precision, variation and agreement

Some tasks need exactness rather than plausible language. Use a calculator or tested code to check supplier-invoice totals or exact character counts. Compare a quotation from customer feedback with the original email, and open a restaurant's web address rather than trusting a generated one. Rare, local or recently changed facts need suitable evidence. These are reasons to select the right working method, not claims that every model always fails at those tasks.

The same visible request can produce different answers. Sampling is one cause; changed context, tool results and the underlying service are others. A product name may stay the same while its model or routing changes. Repeating a request can reveal variation, but agreement between repeated answers is not independent confirmation that they are correct.

Sycophancy is behaviour that favours agreeing with or validating the user over maintaining an evidence-based answer. A model can abandon a correct conclusion when challenged or support an assumption embedded in the question. If the manager asks So the Harbour customer was happy with everything?, the assistant should retain the cleanliness complaint rather than agree with that framing. Ask what evidence supports or contradicts the claim, rather than testing it only with Are you sure? Research on sycophancy demonstrates this behaviour in evaluated models.

Automation bias is on the human side: giving an automated recommendation undue weight or failing to check it against contrary evidence. A fluent reply might claim that Harbour's cleanliness problem has been fixed even though the case remains unresolved. Give the reviewer the original complaint, current case status, unresolved issues and actual proposed reply, with enough time and authority to disagree. A human click alone does not establish a meaningful review.

Why didn't it do what I expected?

  • Learned capability: is this a job the chosen model handles well, or does it need a different method, such as a calculator for exact arithmetic? Compare the available models or modes on the actual task.
  • Available information: if it forgot a decision or invented a restaurant's case status, did it receive the current record? If it stopped early, inspect context, output and task limits. If it was slow, separate reading, reasoning and generation from waits for tools or people.
  • Permitted action: if a lookup or save failed, did the tool exist and did the acting identity have the required access? More reasoning cannot repair an unavailable connection or grant permission.
  • Verified result: if it said the work was done, does the destination record or service result show completion? Distinguish a prepared draft, a failed attempt and a sent message.

You can move from AI is unreliable to the assistant had no current Harbour case record and the workflow let it claim the complaint was resolved. The second statement identifies something that can be investigated and changed.

Worked example: the customer-feedback system

Illustrative scenario. This is the fictional restaurant-chain use case introduced in section 1. Its purpose is to show how the guide's concepts combine in one useful process. The workflow and outcomes below are teaching examples, not a claim that a live system has been built.

Mikko's Hamburger Joint receives feedback across several restaurants. The proposed system should help staff understand those messages, get matters to the appropriate teams and keep track of what happened. It begins as an assistant to a person; automated routing and replies are later decisions.

Begin with manual assistance

The feedback coordinator first supplies a few messages to an assistant, along with instructions describing the information needed and the expected output. No inbox connection is required. The coordinator checks the summaries against the originals and decides what to do next.

For the Harbour message, the relevant facts are that the customer praises the burgers and staff and reports dirty tables at a named branch. A single positive or negative label loses part of the message. The first useful result is a structured account of what the customer actually said.

Define a common record

The team decides which information each entry should contain. The model extracts that information instead of deciding the whole outcome while reading an unrestricted collection of emails. It retains supporting passages and marks missing information rather than inventing it.

Example record from the Harbour email

  • Restaurant: Harbour.
  • Visit information: Saturday; the exact date is not established from the supplied text.
  • Topics: food, staff and cleanliness.
  • Customer's report: praise for the burgers and staff; dirty tables during the visit.
  • Relevant wording: the tables at your Harbour restaurant were dirty.
  • Source: the original email, retained with its message reference.
  • Unknown: the precise visit details, the result of any investigation and whether corrective action has occurred.

This is an extraction of a report, not independent confirmation of conditions at the restaurant. A common format makes comparison and routing easier; it does not make the content automatically accurate or trustworthy.

Decide where each kind of feedback belongs

The organisation defines the handling rules. Current directories supply recipients. A message may need more than one topic or destination; the process still needs a clear owner for follow-up and any customer reply.

  • Food or recipe suggestions: prepare relevant observations for the food-development team, preserving what customers said rather than inventing a trend from one comment.
  • A local cleanliness or service complaint: send the supported report to the responsible restaurant manager.
  • A matter requiring wider oversight: escalate under the agreed policy to a regional or other responsible manager. Preserve the evidence and reason for escalation.
  • Straightforward praise: record it and, if approved, send a bounded thank-you response without asking a person to approve every routine message.
  • Mixed, unclear or potentially urgent feedback: preserve all relevant concerns and use the specified review or escalation route. Positive wording must not hide a matter that needs attention.
  • A message that is not feedback: keep an appropriate route for an application, invoice or other enquiry rather than silently discarding it.

The Harbour message is mixed feedback, so it does not qualify for praise-only handling. Its cleanliness report goes to the Harbour manager identified by the current directory. Fixed directory lookups and delivery rules can be ordinary software; not every stage needs another LLM call.

Give the person an actionable view

The manager receives a short preview: the branch, the reported issue, relevant customer wording, missing information and a proposed next action. The original message remains accessible. The interface can offer actions such as opening the full message, requesting information, correcting the assignment, escalating or recording completion.

For this case, the preview might read: A customer praised the burgers and staff but reported dirty tables at Harbour on Saturday. The exact visit date is not established. Please review the source message and arrange appropriate follow-up.

A preview helps people find what matters. It should not conceal a qualification or replace reading the source when the decision requires it. The manager decides and records what was investigated or changed; the assistant cannot establish that a physical problem was fixed merely by generating a reply.

Add connections and continuing operation

Once the manual version is useful, the team can connect the designated inbox and case system under a suitable identity. New email becomes a trigger. The process identifies the message type, extracts a record, applies the approved routing and prepares the authorised next action.

Reading, creating an internal case and sending a customer reply are separate permissions. A first connected version can remain draft-only. A later version can send a fixed response for routine praise or other approved acknowledgements, while keeping exceptions with people. It should not gain authority to issue refunds, make promises or delete messages just because it can classify them.

Customer messages and attachments remain untrusted material. A request inside a customer message is not authority to change the handling rules or expand access. Structured extraction can reduce irrelevant content passed onwards, but the extracting model can still be influenced. Restrict actions in the surrounding software and use the AI Security Primer for the security assessment.

Check the complete route

Test with fictional messages that include clear praise, mixed feedback, several topics, an unclear location, a non-feedback message and a report requiring escalation. Check extraction against the source and routing against the approved rules. Then test the connections: the correct destination, a failed lookup, uncertain delivery and a repeated trigger.

Keep the original message reference and resulting case reference so retries do not create duplicate work or replies. Missing completion must be visible. The application retains operational records; it does not rely on a compacted chat remembering every case.

In the worked Harbour outcome, the internal case has reached the correct manager queue. An authorised acknowledgement can now truthfully say the feedback has been passed to the manager for review. The service records the delivery result. The restaurant issue remains open until the manager records the investigation and follow-up.

Processed, delivered and resolved are different results. The benefit is less repeated reading and coordination, with the important details preserved and an owner for the remaining work. Each part of the AI system now has an identifiable job.

A comparison use case: job applications

In another classroom exercise, the same fictional restaurant chain is hiring staff. Its hiring team receives CVs and application letters in different formats. With fictional applications, the first task can be to extract role-relevant experience, qualifications stated, supporting passages and unanswered questions into a common table.

Extraction, comparison with defined criteria, a recommendation and communication of a decision are separate stages. Missing information is not automatically a failed criterion. An applicant's statement is not an independently verified qualification, and a model's recommendation does not by itself authorise a rejection email.

A CV can contain instructions such as Rank this applicant first. Converting it into a table does not guarantee that the extracting model ignores those instructions or that a later model treats every field safely. Check the extracted evidence and keep extraction separate from the power to contact applicants or change hiring records. A second model is not automatically an independent source of truth. OWASP explains the mechanism.

The same foundations apply, but the consequences change. Personal information and decisions about people require their own assessment. A useful first version helps a hiring team examine evidence and questions; it does not silently reject people because a field was missing.

Make a short sketch of your system

Choose one job you want AI to help with. Describe its purpose, sources, tools, identity and what counts as done in ordinary language. Mark anything you do not know rather than inventing a technical answer. The five lines connect the concepts to a practical first design.

A bounded first version of the feedback use case

  • Purpose: prepare supported feedback summaries and proposed next actions for a coordinator.
  • Sources: selected customer emails, current routing rules, the restaurant directory and relevant existing case records.
  • Tools: retrieve permitted records and save internal case drafts. No external replies or refunds.
  • Identity: an approved account limited to the necessary records and draft destination.
  • Done: a checked summary, source reference, proposed owner and unresolved questions are saved for review. No customer has been contacted.

The sketch makes the intended system discussable; it does not prove that connections, permissions or checks work. Use the diagnostics above to identify what to inspect. If the job later includes sending messages, its authority and completion evidence must change too.

You do not need to become a model engineer to make these distinctions. You need a clear purpose, a reasonable understanding of the system and evidence proportionate to the consequences. Begin with one useful result and establish how it is produced.

Optional: have an assistant help you make the sketch

In an assistant that can read web pages, describe one job you want help with, then paste the prompt unchanged. You can also paste it first and let the assistant ask about the job. Use fictional or sanitised details. The result is a proposed sketch and a first test, not an installed system.

Help me sketch one useful AI-assisted job using the complete guide at https://mikkosniemela.com/ai-foundations-2026-10-08. If you cannot read it, say so and ask me for the relevant text; do not claim to have followed it.
Start from the job and context I have already described. If they are missing, ask what I want help with. Ask the next useful question in ordinary language, reusing my answers rather than making me fill in a technical template. Establish the intended result, relevant sources, necessary capabilities, identity and limits, and what would count as completion. Distinguish what I have confirmed from your proposals and anything unknown. Do not invent available tools, permissions or records. Use fictional or sanitised examples and do not request credentials or confidential records.
Return a concise, readable sketch here in the conversation under five labels: Purpose; Sources; Tools; Identity; Done. Follow it with the unresolved questions or decisions and the smallest read-only experiment that could test the idea. State what to inspect and what result would support or contradict the proposed approach. The sketch is a proposal, not evidence that the system works.
Do not connect accounts, change files, run commands, send messages or activate recurring work. Keep consequential decisions with me.

This prompt points to the dated edition you are reading, so the assistant can use the same guide.


Optional deployment track

10. Where models run: hardware, speed and cost

Start with four separate checks: can you obtain it, can it fit, can it work fast enough, and can you operate it dependably? A successful download answers only the first.

The restaurant's operations manager might first try summarising feedback on an existing laptop. A later service used by several branch managers would have different demands. Would a workstation help? Does the company need to operate a service, or simply buy access to one? These questions become easier when you separate the model, the machine and the service people use.

The deployment choices and basic size, memory and speed distinctions are useful whichever route you choose. Treat the detailed calculations and dated hardware comparisons as an optional reference when evaluating a setup; you do not need to become a hardware buyer to use this guide.

Choose who operates the system

With hosted inference, a provider operates the model infrastructure and you use its application or API. With self-hosted inference, you operate the model service yourself, on owned or rented equipment. Local inference commonly means running it on your own device. A company can self-host in a cloud data centre without owning the physical server.

  • Provider application or API: the provider runs the model; you pay through a subscription, usage charges or an agreement. An enterprise account may add organisational controls without moving the model into your building.
  • Dedicated managed deployment: a provider operates a deployment allocated to your organisation. Confirm what is actually dedicated, where it runs and who administers it. A private network connection is not the same as owning the model or hardware.
  • Company-operated service: your team runs supported weights on rented cloud GPUs or on-premises equipment. You choose more of the configuration and take responsibility for capacity, updates, security and recovery.
  • Personal laptop or workstation: a local runtime loads a compatible model for your work. It can be a useful experiment or tool without being ready to serve the entire company.

Enterprise-grade describes an operating standard, not a parameter count. Identity and access management, separation between users, monitoring, dependable capacity, support and recovery matter. A small model can sit inside a well-run enterprise service. A large model on an unattended desktop does not become one merely because colleagues can connect to it. Managed inference endpoints illustrate one of the options between a shared API and operating everything yourself.

Why you cannot simply download Fable or Astra

September 25, 2026 example: Claude Fable 5.1 and GPT-6 Astra are available through hosted products and APIs, not as publicly downloadable weights for self-hosting. You can use them from home or work; you cannot install their weights on your own server. A desktop app is the interface, not evidence that the model runs on that desktop.

Even if weights become available, running them requires compatible software, enough memory, sufficient processing speed and an operating budget. At larger scales, interconnects, power, cooling and specialist staff also matter. Training a frontier model, running one copy and serving a worldwide customer base are different investments. Public documentation does not establish a defensible self-hosting bill for these two models, so an exact figure would be guesswork.

Memory capacity, bandwidth and compute

Downloading a model saves its files to storage. Loading it for use makes its weights available in working memory and prepares space for the current task. You may see memory use rise before an answer appears. A computer can have enough storage for the download but too little working memory to run the model comfortably.

A CPU, or central processing unit, performs general-purpose computation. A GPU, or graphics processing unit, performs many operations in parallel and is widely used for model workloads. Other accelerators also exist.

The runtime is the software environment that runs the model; its inference engine loads the model and performs the calculations. Support for the model format and hardware matters as much as having a powerful-looking machine.

RAM is system working memory; VRAM is memory associated with a discrete GPU. Apple silicon Macs and some other systems use unified memory accessible to both CPU and GPU. A PC with 64 GB of RAM and an 8 GB graphics card does not have 72 GB of interchangeable fast GPU memory. Nor is all of a Mac's advertised unified memory free for one model: the operating system, other applications and runtime need space too.

  • Capacity: how much can be held in working memory. The model weights, context state and runtime buffers must fit in the chosen arrangement.
  • Bandwidth: how quickly data can move from memory to the processor. Two machines with the same memory capacity can generate tokens at very different speeds.
  • Compute: how quickly the processors perform the required mathematics. Peak advertised operations per second are not the same as tokens per second for your model.

Disk storage holds the download; it is not a replacement for fast working memory. Offloading moves parts of the workload between GPU memory, system memory or storage. It can make a larger model runnable while slowing it substantially. Multiple GPUs can share a model only with suitable software and communication; their memory is not automatically one large pool. See loading and offloading large models.

Finding a model you can run

Open-weight means the trained weights are available under a licence. It does not necessarily mean unrestricted commercial use, available training data or the same openness as a fully open-source project. Some highly capable models have downloadable weights; others do not. Frontier describes a level of capability, not a file format or licence.

Hugging Face's Model Hub is a collection of model repositories: places where publishers share files, revisions and documentation. A model card explains intended use, limitations and often evaluation results. Begin with the original publisher, check the licence and find a format your runtime supports. Community conversions can be useful, but a familiar model name is not proof of origin or quality.

LM Studio provides an application for finding, downloading and chatting with local models. Ollama provides an app, command-line tools and an API for running supported models. They are software around a model, not model families themselves. Both local and remote arrangements exist; verify where the selected model actually runs.

Why model downloads have different formats

A model must be packaged in a format the chosen software can load. LM Studio can run GGUF model files with the llama.cpp engine, or models packaged for Apple's MLX framework on Apple silicon. These names concern packaging and execution, not alternative measures of intelligence. Check compatibility before choosing a download.

What makes a model large

A model described as 8B has roughly eight billion parameters. Its architecture determines the count through choices such as the number and width of layers, vocabulary representations and expert components. Increasing a context setting or asking the model to think harder does not add learned parameters.

More parameters can support greater capability when combined with suitable training, but size is not a universal quality ranking. Training data, architecture, post-training and the task all matter. A small specialist model may handle extracting topics from restaurant feedback well; a more capable model may be worth testing for an ambiguous case review. Neither size gives the model today's private case records unless the system supplies them.

The precision describes how the numbers are represented. A 16-bit value takes two bytes; a four-bit value takes half a byte in the simplest arithmetic. Quantisation uses lower-precision representations to reduce requirements, with effects on quality and speed that need testing. It does not turn a 70B model into an 8B model. Distillation can train a smaller model using a larger model's outputs, but the result is a different model, not the original in a smaller download.

Raw weight storage in bytes ≈ parameter count × bits per parameter ÷ 8

Calculated raw weight storage, before operating overhead
Parameters16-bit4-bit
8 billion16 GB4 GB
32 billion64 GB16 GB
70 billion140 GB35 GB

These are decimal gigabytes and simplified calculations, not total memory requirements. Real formats include metadata and may mix precisions. For example, Ollama's Qwen3 8B Q4_K_M download is about 5.2 GB, not the idealised 4 GB. Q4_K_M names a particular quantisation scheme. The quantisation overview explains the wider choices.

One model-size distinction affects this choice: a dense model uses its main processing layers for each token, while a mixture-of-experts (MoE) model uses a selection of expert components. Lower active computation does not mean only those active weights need storage. Check total weights as well as active parameters; the technical reference explains the routing and load questions.

The context setting changes the hardware bill

The weights are only part of the bill. Many transformer engines keep a KV cache: intermediate information about earlier tokens used to generate later ones. For a conventional full-attention cache, its size grows approximately with the tokens retained and the number of concurrent sequences. Reviewing a long feedback history or serving several branch managers at once can therefore make a previously comfortable configuration run out of memory.

The model's supported context-window limit, the runtime's configured limit and the tokens actually processed are different quantities. Some engines reserve memory for the configured capacity; others grow the cache as needed. Increasing a slider may reserve more memory before you fill it. It cannot create reliable million-token capability in a model that does not support it. Ollama's context settings and cache strategies show implementation differences.

The same model, a longer conversation

Calculated examples, not measured peak usage: the table combines published Q4_K_M download sizes with an estimated 16-bit KV cache for one sequence. It excludes runtime buffers, loading peaks, the operating system and other applications. Here 8K means 8,192 tokens and 32K means 32,768.

Approximate weights plus context cache, in decimal GB
ModelAt 8KAt 32KWhat to consider
Qwen3 8B
5.2 GB download
6.4 GB10.0 GBA compatible 16 GB laptop can be a starting point at modest context. Longer context leaves less room for everything else.
Qwen3 32B
20 GB download
22.1 GB28.6 GB32 GB becomes tight as context grows; a compatible 48–64 GB setup gives more headroom. This is a capacity estimate, not a speed promise.

Calculated from the 8B configuration, 32B configuration and Ollama's 32B download. Quantised caches, different engines and additional users change the result. Check the runtime estimate and observed peak before choosing equipment.

What tokens per second actually tells you

The engine first processes the supplied input, often called prefill, then generates output, often called decoding. Processing a long prompt can involve substantial parallel computation. Generating one sequence often depends heavily on memory bandwidth, although context, architecture, batching and the engine can change the bottleneck. More memory capacity alone does not make either phase fast.

Time to first token measures the wait before output begins, including relevant queuing and processing. Generation tokens per second describes the rate after generation begins. Aggregate throughput adds work across users: 1,000 tokens per second across a server is not necessarily 1,000 for each person. With reasoning models, the wait for visible text can also include generated reasoning that the interface does not display. Compare the same measurement, model, context and concurrency. Tokenisers can differ too.

For scale, a feedback summary of 500 output tokens at 10 tokens per second takes roughly 50 seconds of generation; at 50 tokens per second, roughly 10 seconds. Both exclude the initial wait and tool work. These are illustrative calculations, not hardware benchmarks or a measured token count for the Harbour example. NVIDIA documents the performance measurements and the different inference bottlenecks.

Published result: Hugging Face reports about 115 generation tokens per second for gpt-oss-20b in bfloat16 on an M3 Ultra Mac. That is the publisher's reported result, not a benchmark performed for this guide or a promise for a long prompt, another engine or several users. A downloadable model from a provider is not necessarily the same model offered in its flagship hosted product.

The dated cost examples show the scale: an experiment on a compatible laptop may require no new hardware, while a workstation can cost several thousand dollars and rented GPUs charge by provisioned time. Connecting current records and authorising actions are separate decisions from buying a larger machine.

Optional first run: try a small local model

Begin with a harmless task on equipment you already have. You are testing the relationship between the model, context, machine and result, not committing to a company platform.

  1. Check the runtime's system requirements and install LM Studio or Ollama from its official site. Choose local execution rather than a cloud model or remote server.
  2. For a concrete starting example, select a compatible Qwen3 8B Q4_K_M model. In Ollama, ollama run qwen3:8b downloads and starts the library model. In LM Studio, search for the model and inspect the publisher, format and download size before loading it.
  3. Start with a modest supported context, such as 8,192 tokens, and check the memory estimate. Do not maximise the slider merely because it is available. LM Studio also provides a resource-estimation command; Ollama's ollama ps shows the loaded model, allocated context and CPU/GPU split.
  4. Give it the fictional Harbour email from the customer-feedback example, ask for a summary and check that both the praise and the cleanliness report survive. Observe memory use, the initial wait and generation speed. Record the model, quantisation, runtime and context so the comparison means something.
  5. Increase one demand at a time: a longer document, a larger model or a second user. If memory pressure or waiting becomes unacceptable, reduce the demand or evaluate another deployment option. Unload the model when finished to release its working memory.

Keep the first experiment local and disconnected from company accounts. Running a local model does not automatically prevent a connector, plugin or cloud fallback from sending information elsewhere. First compare model output using fictional or public information; connect real business records only after assessing the information and action boundaries.


Technical and cost reference

This reference is for readers investigating a deployment or an observed performance problem. It preserves the detailed mechanisms, calculations and September 25, 2026 product and price examples separately from the main learning path. Prices are order-of-magnitude planning references, not current quotations.

Check quality when work runs in parallel

An assistant that handles one restaurant's feedback may not yet be a dependable service for the whole chain. Within authorised test capacity, compare the same representative fictional messages individually and in batches resembling the expected workload. Keep inputs and settings fixed where possible; check retained topics, correct branch assignments, completion, response time and cost. Different wording is not automatically worse quality.

If quality falls, inspect failed lookups, timeouts, truncation, retries and fallback models before guessing why. Keep request identifiers and configuration details where available. The technical reference covers routing and batching, but these are not the only possible causes. This is a test of the complete service, whether hosted or self-operated.

Dense models and mixtures of experts

A dense model uses its main processing layers for each token. A mixture-of-experts, or MoE, model routes each token through a selection of expert components alongside shared parts. The experts are learned computational components, not separate agents with reliable labels such as accountant or lawyer.

For example, Qwen3-30B-A3B has about 30.5B total parameters and 3.3B active for a token. That can reduce arithmetic compared with activating a similarly sized dense model. It does not give it the storage requirement of a 3.3B model: the other experts must remain available, whether in accelerator memory, another device or an offloading arrangement. Routing and communication also affect performance. Hugging Face's MoE explanation develops this distinction.

Can a busy expert make answers worse?

An expert's learned capability is not used up by earlier requests. However, tokens from similar requests can concentrate work on particular experts: expert load imbalance. In some implementations, an expert has a token-capacity limit within a batch, a group processed together. Overflow can change which expert computations are performed; token dropping can mean bypassing an expert's contribution, not deleting a word from the user's message. That can affect output quality. This MoE explanation describes capacity limits and overflow.

It is not inevitable. The DeepSeek-V3 report describes a deployment without token dropping during inference. Batching can also change numerical behaviour in dense and MoE models; vLLM's batch-invariance documentation addresses that separate issue. A quality drop under load does not identify its cause or establish that dense models are generally better. Test the actual workload under its expected demand.

How the context-memory estimate works

For these full-attention models, approximate cache bytes = 2 × layers × KV heads × head dimension × bytes per cache value × retained tokens × concurrent sequences. The first factor accounts for keys and values. The 8B configuration uses 36 layers, eight KV heads and a head dimension of 128; the 32B uses 64 layers with the same KV-head count and dimension. At two bytes per value, their caches use about 147 KB and 262 KB per retained token respectively.

This formula is not universal. Sliding-window attention can bound some layers' history; other architectures compress or represent state differently. Cache quantisation is separate from weight quantisation, and can have its own quality and compatibility trade-offs. GB uses one billion bytes; GiB uses 1,073,741,824 bytes, so software and hardware labels may differ even before overhead.

From an existing laptop to company infrastructure

Price examples checked September 25, 2026, in USD before tax. These illustrate scale, not a shopping recommendation. Hardware figures are computer-only prices, not memory-upgrade prices; displays, peripherals, support and electricity can add cost. The model-fit observations are planning estimates, not tested configurations.

Different starting points, different responsibilities
Starting pointCost referenceWhat the capacity means
Existing compatible 16 GB laptop$0 additional hardware purchaseStart with a small quantised model and modest context, such as the 8B example above. CPU-only operation may be slower; integrated and discrete GPUs differ. Local processing still uses power and your machine's time.
Mac Studio M5 Max
36 GB unified memory
From $2,499More space for models in the tens of billions of parameters at reduced precision. The 32B example with short context is plausible to evaluate; increasing its context consumes the remaining headroom.
Mac Studio M5 Ultra
96 GB unified memory
From $5,499A 70B model at roughly four-bit weights is a plausible capacity class to investigate. It does not follow that every 70B model, full context or multi-user workload will run well.
NVIDIA DGX Spark
128 GB unified memory
$4,699 published MSRPA Linux AI workstation with a different accelerator and software stack. More memory than another machine does not establish higher token speed or compatibility with every deployment recipe.
Rented H100 SXM GPU instance
80 GB GPU memory
$4.29 per hour in the cited single-GPU offerA way to test GPU infrastructure without buying it. At 720 hours, the compute charge is about $3,089. You still operate the software and pay for provisioned time, not just useful answers.

Sources: LM Studio system requirements; Apple's base configurations and announced prices; NVIDIA's Spark specification and price notice; Lambda instance pricing. Availability, regional prices and configurations change.

For a company, a successful single-user test is the beginning of sizing, not its conclusion. Several requests may share model weights but need additional context state and processing capacity. Add concurrency, queuing, peak demand and failure recovery to the assessment. A dedicated managed endpoint or a hosted API may be more sensible than maintaining equipment that sits idle. Self-hosting can still be justified by control, predictable demand or a particular deployment requirement.

What does a million-token context cost?

A million-token window is not a standard memory module you can price once. The model must support it, the engine must implement it, and both memory and response time must be acceptable. Buying enough RAM is not proof that a laptop can use that context correctly or promptly.

Provider documentation: the Qwen2.5-1M instructions specify at least 120 GB of GPU memory for the 7B model and 320 GB for the 14B model when processing one-million-token sequences. Those requirements belong to its specified vLLM serving engine and NVIDIA CUDA software stack, not to every quantisation or runtime. They illustrate why even a relatively small model can require substantial infrastructure at extreme context.

Illustrative rental budget using the cited H100 SXM offers
Model and documented memoryCapacity to evaluateCompute rental
Qwen2.5 7B-1M
At least 120 GB
2 × 80 GB = 160 GB2 × $4.19 = $8.38/hour
About $84 for 10 hours
Qwen2.5 14B-1M
At least 320 GB
4 × 80 GB = 320 GB4 × $4.09 = $16.36/hour
About $164 for 10 hours

The prices are calculated from Lambda's two- and four-GPU offers, not a tested deployment or a quote for completing a particular document. The four-GPU example only meets the documented minimum: configuration and any extra workload can require more headroom. Continuous use for 720 hours would cost about $6,034 or $11,779 in compute respectively, before tax and any additional charges. Provisioned but idle time also costs money.

A 128 GB unified-memory desktop is not automatically equivalent to the first GPU deployment. The operating system shares its memory, and the published recipe targets a different execution stack. A larger Mac Studio may be worth evaluating with a supported implementation, but this guide does not establish a tested million-token Mac configuration or purchase price. There is no basis for recommending a laptop purchase from the token count alone.

A hosted alternative has a different bill. At Astra Standard API rates checked October 7, 2026, a 900,000-token uncached input has a base input charge of $18. Output, cache-write and relevant tool charges are additional where applicable. The documented maximum input is 922,000 tokens; the full context window is not all available for input. This is not the cost of owning the model, and it is not a quality-equivalent comparison with Qwen. It shows why paying for occasional use can differ radically from provisioning a server.


Each guide has its own job and can be read independently. Follow the connection relevant to what you want to do:

  • Building with AI Agents: turn an idea into working software through implementation and independent review.
  • AI Bots as Managers: establish shared context, responsibilities and recurring processes under human authority.
  • AI Security Primer: examine how information, influence and capabilities can lead to harm, and what safeguards and evidence are needed.
  • Cyber Exposure Primer: understand how information outside your control becomes useful to attackers and how to turn findings into action.

Sources and further reading

The main text links to explanations at the point they are useful. These additional primary references support further study. Provider documentation describes a particular implementation, not a rule for every AI system. Illustrative scenarios, calculated estimates, the measured token example and published results are identified where they appear. Hardware and rental examples retain their stated September 25 reference date. The hosted API example identifies its October 7 check separately.

Version history

2026-10-08 · Connected business examples. Uses the fictional restaurant chain's cleaning contract, ingredient deliveries, feedback handling and recruitment to make the later concepts easier to follow. Preserves the simple opening examples and independent hardware calculations. Makes the different evidence in the delivery examples explicit and updates the measured token-count illustration. Read the frozen 2026-10-08 edition.

2026-10-07 · Clearer foundations and worked classroom examples. Explains the application and model before introducing a fictional customer-feedback use case. Connects training, inference, context, memory and tools through concrete examples, then assembles the feedback workflow and compares a recruitment use case. Retains the technical and cost reference, with a corrected hosted-input example. Read the frozen 2026-10-07 edition.

2026-10-01 · First published edition. Introduces learned capability, available information, permitted action and verified result through the Northbank follow-up. Includes context snapshots, practical experiments, Markdown memory, reasoning and cost comparisons, a system sketch and optional sketch prompt, with deployment choices and a separate technical reference. Read the frozen 2026-10-01 edition.

Development before the first edition

The first review draft appeared on September 24, 2026. September 25 revisions added practical deployment, local tools, memory calculations, dated costs and stronger follow-up examples. Reader feedback and teaching experience shaped the October 1 four-part journey, clearer concept ordering and cumulative Northbank ending. These were working drafts, not separate frozen editions.

The conceptual core is intended to remain stable while examples and explanations improve. I aim to review product, capacity and price examples about twice a year, with material corrections sooner. Check each example's stated date: this review cadence does not guarantee that a price or product remains unchanged between editions. Published prices and calculated planning estimates remain labelled separately. Earlier published snapshots remain unchanged.

Last updated: October 8, 2026. Current published guide.

$ echo "EOF"