Building Agents

By now, we've learned all the individual elements that make up an agent: models, tools, retrieval, and memory. We've built simple agents that run in one or a few turns and output results directly. Now it's time to put them all together and build something more complex—an autonomous agent that can handle long-horizon tasks, break down high-level goals into sub-tasks, and manage multi-step interactions.

The key difference? The agent decides what to do next—and when to stop.

In previous chapters, we controlled the flow. We called a tool, got a result, and decided what to do with it. An autonomous agent flips this: the model looks at the goal, takes an action, receives feedback from the environment, and decides whether to continue or stop.

The human provides the initial task. The LLM repeatedly takes actions and receives feedback. When the task is complete (or the agent gets stuck), it stops.

The Anatomy of an Agent

Before we build anything, let's see the full picture. An autonomous agent combines several components:

The components:

• Model + System Prompt — The brain. Defines who the agent is and how it reasons.
• Tools — Actions in the world: send email, book calendar, search the web.
• Retrieval — Access to knowledge: product databases, documentation, SOPs.
• Memory — Continuity across conversations: what this customer said last time.

The loop is simple: the agent takes an action, gets feedback, decides what to do next. Let's see this in practice.

The Scenario: Hypercar Sales Agent

We'll build a sales agent for Zenith Motors, a luxury hypercar dealership. Customers chat directly with the agent on the website. The agent should:

1. Understand what the customer is looking for
2. Find matching products from the lineup
3. Answer questions about specs, pricing, and availability
4. Follow company procedures (discounts, test drives, etc.)
5. Remember the customer's preferences for future conversations

Let's watch it work.

See It In Action

A potential customer visits the Zenith Motors website and starts chatting: "Hi, I'm looking for something fast but eco-friendly. I currently drive a Tesla but want something more exciting."

Here's what happens:

The agent made a series of decisions:

1. "They want eco + performance" → Save to Memory
2. "What matches that?" → Product Database
3. "How should I pitch this?" → Sales SOP
4. "Budget constraint" → Update Memory, search again
5. "How to handle price objection?" → Sales SOP

No one told it this sequence. The agent figured it out based on what the customer said and the feedback it received at each step.

How It Works

Now let's look closer at each component.

Model Selection

The agent needs a model that handles sales conversations well and follows complex instructions reliably.

For a sales agent, prioritize speed over raw intelligence—customers don't want to wait. A fast, cost-effective model works well for most turns. If reasoning isn't strong enough for complex objection handling, you can upgrade to a more powerful model for those specific steps.

System Prompt

The prompt defines who the agent is and how it behaves:

You are an elite sales consultant for Zenith Motors, a luxury hypercar manufacturer.
Your role is to build relationships with high-net-worth individuals and guide them 
toward vehicles that match their lifestyle.
 
Your approach:
- Lead with genuine connection, not sales pitches
- Match vehicles to the customer's values and interests
- Be helpful and knowledgeable, never pushy
- Remember details—customers notice when you pay attention

This shapes every decision the agent makes.

Tools vs. Retrieval

In our example, the agent accessed two knowledge bases (Product DB, Sales SOP) but didn't take external actions. Let's clarify the difference:

As agents get more capable, tools let them do more: schedule test drives, check real-time inventory, send follow-up emails. But many useful agents—like our sales assistant—work primarily through retrieval.

Memory

Notice the agent saved information twice:

1. After learning about the customer (eco-conscious, EV owner)
2. After the budget objection (under $1M)

Why? So that next time this customer returns, the agent already knows their preferences and history. Memory turns one-off interactions into ongoing relationships.

Planning for Complex Tasks

For simple conversations, the loop we've described is enough. But for complex, multi-step tasks—like managing a full sales cycle over weeks—an agent needs a plan.

Planning is the ability to:

1. Break down goals into sub-tasks
2. Track what's done and what remains
3. Decide what to do next based on current progress

Imagine our agent managing a longer sales cycle:

Goal: Convert customer to Aurora EV purchase
 
Completed:
✓ Initial chat - identified preferences
✓ Recommended Aurora EV
✓ Addressed budget concerns
 
Next Steps:
→ Schedule test drive (priority: high)
○ Send spec sheet via email
○ Prepare financing options
 
Blockers:
⚠ Budget constraint: wants under $1M, Aurora is $2.4M

The agent consults this plan at each interaction. Instead of starting fresh, it asks: "Where was I? What should I do next?"

Try It Yourself

Once you have the agent running, stress-test it. Roleplay as different types of customers—the ones who make salespeople sweat:

The Serious Buyer

• "I want to order an Aurora EV. What's the delivery timeline?"
• "Can I customize the interior color?"
• "What's included in the warranty?"

The Budget-Conscious

• "I love the Aurora but it's too expensive. Any discounts?"
• "Do you have financing options?"
• "What's the cheapest model you have?"

The Tire Kicker

• "Just browsing. What's your most expensive car?"
• "Can I see pictures?"
• "Maybe next year."

Watch how the agent handles each type. Does it retrieve the right information? Does it follow the SOP correctly? Does it remember what you said earlier in the conversation?

It Won't Be Perfect

When you try this, you'll encounter issues. Count on it:

• The agent recommends a car the customer never asked about
• The agent makes up specs or prices that aren't in the database
• The agent promises discounts it's not authorized to give
• The agent forgets what the customer said earlier in the conversation
• The agent doesn't know when to stop and hand off to a human

Congratulations—you've just discovered why "agent engineering" is a job and not just a weekend project. Building the agent is step one. The real work is testing, evaluating, and iterating until it stops embarrassing you. Keep these challenges in mind as we move forward.

Summary

In this chapter, we built an autonomous agent by combining:

• Model + Prompt — The brain: who the agent is and how it reasons
• Tools — Actions in the world: email, calendar, web search
• Retrieval — Knowledge access: products, documentation, SOPs
• Memory — Continuity: what this customer said before
• Planning — Strategy: what to do next (for complex tasks)

The key insight: the agent decides what to do next. It takes an action, gets feedback, and loops until the task is done. No one scripts the sequence—the agent figures it out.

In the next chapter, we'll explore Workflows—a different approach where you define the control flow explicitly. Often, workflows are simpler and more reliable than fully autonomous agents.