Overall approach Started reactive rather than planned — the first few sessions were diving straight into building (extraction, matching, schema) and only shifted to a more deliberate process when the eval scores came back low. In hindsight, planning the eval rubric earlier would have saved a full iteration cycle. The agent drove most of the implementation; control was tightest around the judgment calls (what counts as hallucination, when to override confidence in code vs. leave it to the model).
What you reviewed by hand vs. trusted Read carefully: the judge rubric changes, because a bad rubric poisons every score downstream. Also the extraction prompt fix for Lisbon — small prompt changes have outsized effects and are easy to get subtly wrong. Skimmed and trusted: the budget math logic once the tool-split pattern was established, and the typecheck output (green = ship). Shipped without fully verifying: the catalog reason strings — the Punta Cana "February" text had been wrong for a while before the eval caught it, meaning it wasn't part of any manual review pass.
What you'd do differently next time The rubric confusion (penalizing mock-data specifics as hallucination) cost a full debugging session that a clearer rubric definition upfront would have avoided. Also version the eval results from the start — having results.json in .gitignore meant there was no baseline to diff against when scores changed.
If this had to run at scale (10k req/hour) The biggest bottleneck would be the sequential tool calls — extraction, then matching, then synthesis all happening in series per request. At volume, those need to be parallelized where dependencies allow: destination matching and flight lookup can run concurrently once extraction is done. The LLM synthesis step should be the last thing called and given the smallest possible context — strip out any catalog data that didn't make it into the final candidates before it hits the model, since you're paying per token. For model choice, the synthesis step doesn't need a large model; most of the reasoning is already done deterministically in code, so a smaller faster model (20b range) handles the narration fine and cuts inference cost significantly. Caching is an easy win for the matching layer — destination scores for common tag combinations are fully deterministic and can be memoized. For the eval at that scale, you can't run a judge LLM on every response; instead promote the deterministic constraint checks (schema valid, withinBudget, confidence level, caveat count) to the primary quality signal and sample maybe 1-2% through the full judge for drift detection. Also the mock examples needs to be connected to real APIs with potentially changes on the LLM evalution.
I am trying to implement a project related to a trip agent in the simplest form yet with good testing. Think of an agent that would connect to the claude, openrouter.ai, or gemini APIs to use for LLM as planner, judge, etc. I will enter the API keys in a notes section to be used by the program and can make decision on what model and what API to use. Make the provider/model configurable, keep secrets out of the repo, and document setup in your README so we can see which provider/model you used, what env vars are required, and what happens when credentials are missing or invalid.
Make simple decisions when desiging as this is a prototype of a working solution for trip agent / a simple yet working solution. Here is the description. for the stack use Next.js + TypeScript.
---
The Build Task
Build a small
"Trip Idea" agent; a single-turn function that takes a natural-language travel request and returns a structured suggestion.
The agent should be able to reason about
destinations, budget, flights, and accommodation
. How you split this into tools is up to you; at minimum 3, more if it makes sense. Mocks are fine (hardcoded responses for a couple of inputs is plenty).
My thinking here: Let's first think about what can be legitimate tools that we can build 3-4 of them as of high priority. Flight and Hotel are must, rest we can discuss.
Must have
- A typed, structured response (destinations, reasoning, rough budget, caveats)
- Graceful handling when critical info is missing: ask, flag, or degrade rather than hallucinate
- A small eval harness: 5+ cases, scored on at least 2-3 dimensions (maybe with an llm-as-judge), with a summary report
- A README.md that gets us running in under 5 minutes
Explicitly out of scope
- Real travel data or any third-party travel APIs; keep the mocks simple
- A UI. A CLI or a single function entry point is plenty.
- Persistence, sessions, or multi-turn conversation. One request → one response.
- Authentication, accounts, or any notion of users.
- Production concerns: deployment, caching, observability infrastructure, retries.
- A "perfect" agent. cutting scope intelligently is important.
Let's discuss these items before you build, i want to see what are reasonable options first, let's plan these items and discuss before we actually go and write code:
1. How to slice the tools across destinations, budget, flights, and accommodation. One tool per concern, combined, or some other split — defend it.
2. How the agent orchestrates its tool calls. A single LLM-as-planner pass, a ReAct loop, a structured workflow — your call, but be ready to explain.
3. How you handle missing or ambiguous information. Ask the user, infer with caveats, refuse to answer — different choices serve different products.
4. What dimensions your eval scores on, and how. Deterministic checks, schema validation, LLM-as-judge — Pick things that would give us real signals instead of overthinking about the real cases. Simplify!
5. What other dimensions you think are important? list them for me.
A few examples to go over (as you see they have different levels of information, it is important for us to think on different levels of user provided information and scenarios to handle them)
Example inputs:
"Relaxing beach week in February under $2000 for two, leaving from JFK. Want anice hotel."
"Long weekend in Europe, flying from SFO. Walkable city, great food, boutiquehotel under $300/night."
"Family of 4 (kids 6 and 9), warm weather over spring break, ~$5k all-in. Need a place with a pool."
"Anniversary trip in May, somewhere romantic and not too touristy. We hated Santorini. Budget flexible but flag if it's blowing past $4k."
"I'll be in Lisbon Oct 10–13 for a wedding. Want to tack on 5 days somewherenearby. Already have the flight to Lisbon."
"Tokyo for 8 days in November, two adults, $6k budget excluding flights. We'vebeen once before."
"Cheap."
What you were trying to do:
This is my first prompt, where i have extracted all needed/asked information from the homework and created a clean prompt for execution and planning. I have asked the main parts to be discussion on what can be the real decisions. Most of suggested items where agreeable for me, on tools slicing, orchestrates, and evaluation criteria. Based on this i have dived deeper into the missing or ambiguous information part, based on the evaluation results. What the tool gave you back (briefly): Claude Code has analyzed and gave me overall design suggestions along with the some questionaire to pick model decisions, and some other decisions to make on tools and orchestration etc. What you kept, changed, or threw away — and why: Most of the suggested items were legit, so went with the design and plan to execute so that i can run and review the code myself.
should we sepereate the llm-as-judge vs. the inference model for customers? in terms of cost vs. efficiency we might be able to use cheaper models for less critical tasks, though for a judge we can use a better model for more reliablity
What you were trying to do:
I wanted to make this a choice for the user to pick the right model for their usecase. In addition, it helped with my iterations over code and evaluations to be faster as I can use faster models for the inference, and use more reliable models for evaluations What the tool gave you back (briefly): It agreed and executed the task for me What you kept, changed, or threw away — and why: The suggested changes were perfect
still the quality of responses are quite low, can you go through these examples and suggest what we can improve: [
{
"name": "beach-under-budget",
"input": "Relaxing beach week in February under $2000 for two, leaving from JFK. Want a nice hotel.",
"schemaValid": true,
"constraintsPass": false,
"constraintDetail": "flight present=true, withinBudget=false",
"groundedScore": 2,
"groundedRationale": "The agent fabricates specific details (exact flight cost $350, hotel name \"Cancun Beachfront Resort\", price $220/night) without any source or indication that these are estimates. While it includes some caveats about the budget overrun, it fails to flag that the flight and hotel prices are assumed or approximate, effectively presenting invented facts as facts. This reduces grounding, though the overall structure (suggesting destinations, noting direct flights, and providing a budget breakdown) is reasonable.",
"output": {
"destinations": [
{
"name": "Cancun",
"country": "Mexico",
"reason": "Direct flights from most US hubs, warm in winter, all-inclusive resorts with pools."
},
{
"name": "Punta Cana",
"country": "Dominican Republic",
"reason": "Reliable beach weather in February and a wide range of nice hotels at varying price points."
}
],
"flights": {
"origin": "JFK",
"destination": "Cancun",
"estimatedPriceUSD": 350,
"notes": "Round-trip economy, direct flights common."
},
"accommodation": {
"name": "Cancun Beachfront Resort",
"type": "all-inclusive resort",
"pricePerNightUSD": 220,
"notes": "Pool, beach access, family-friendly."
},
"budget": {
"estimatedTotalUSD": 2020,
"withinBudget": false,
"breakdown": "Flights: $350 x 2 traveler(s) = $700. Hotel: $220/night x 6 night(s) = $1320. Total: $2020."
},
"reasoning": "Cancun is the better fit because its total estimated cost of $2020 is only $20 over the $2000 budget, while Punta Cana exceeds the budget by $200. Both destinations offer beach relaxing vibes in February with direct flights from JFK and all-inclusive resorts with pools. Cancun's hotel is slightly cheaper and still meets the 'nice hotel' preference, making it the more budget-conscious choice without sacrificing key requirements.",
"caveats": [
"Cancun's total budget is $20 over the limit, which may be unacceptable since budgetIsFlexible is false.",
"The hotel price is based on 6 nights, but the trip is 7 days — assuming check-out on day 7, so 6 nights is correct.",
"No mention of taxes or fees in the breakdown, which could push costs higher."
],
"confidence": "medium"
}
},
{
"name": "europe-boutique-weekend",
"input": "Long weekend in Europe, flying from SFO. Walkable city, great food, boutique hotel under $300/night.",
"schemaValid": true,
"constraintsPass": true,
"constraintDetail": "nightly=180, flight present=true",
"groundedScore": 3,
"groundedRationale": "The agent introduced specific facts not provided by the user—exact flight price ($650), a specific boutique hotel name and nightly rate ($180), assumed 2 travelers and 2 nights, and a detailed budget total. While some caveats are included (e.g., traveler count unknown, flight price variability), the response presents these invented details as firm estimates without clearly flagging them as tentative or example values. This mix of invented specifics and partial caveating lowers the groundedness.",
"output": {
"destinations": [
{
"name": "Lisbon",
"country": "Portugal",
"reason": "Compact, walkable center, strong food scene, and direct flights from several US cities."
},
{
"name": "Porto",
"country": "Portugal",
"reason": "An easy train ride from Lisbon, walkable old town, well known for food and wine."
},
{
"name": "Sicily",
"country": "Italy",
"reason": "Romantic coastal towns with a more local feel than the typical Greek-island circuit."
}
],
"flights": {
"origin": "SFO",
"destination": "Lisbon",
"estimatedPriceUSD": 650,
"notes": "Round-trip economy, one stop typical from SFO."
},
"accommodation": {
"name": "Bairro Boutique Hotel",
"type": "boutique hotel",
"pricePerNightUSD": 180,
"notes": "Walkable old-town location, well reviewed."
},
"budget": {
"estimatedTotalUSD": 1660,
"withinBudget": null,
"breakdown": "Flights: $650 x 2 traveler(s) = $1300. Hotel: $180/night x 2 night(s) = $360. Total: $1660."
},
"reasoning": "Lisbon best matches the request for a walkable city with great food and a boutique hotel stay. It has direct flight options from SFO (though one stop), a highly walkable historic center, and a renowned food scene — all aligned with the trip vibe. The Bairro Boutique Hotel is explicitly labeled as boutique and located in a walkable area. Its total estimated cost ($1660) is lower than Porto and Sicily, making it the most efficient fit for a 3-day trip with no specified budget but inflexible spending preferences. Sicily lacks clear walkability and direct flight data, while Porto, though similar, has a higher estimated total cost and is less directly connected from SFO.",
"caveats": [
"Traveler count not specified — assumed 2 travelers.",
"Traveler count is unknown; budget assumes 2 travelers, which may not be accurate.",
"Flight prices are estimated and may vary with booking time or season.",
"Budget is inflexible but not specified — if the actual budget is below $1660, this may be over.",
"Sicily’s tags don’t mention walkability or food as strongly as Lisbon’s, and its flight route is unspecified."
],
"confidence": "high"
}
},
{
"name": "family-pool-spring-break",
"input": "Family of 4 (kids 6 and 9), warm weather over spring break, ~$5k all-in. Need a place with a pool.",
"schemaValid": true,
"constraintsPass": true,
"constraintDetail": "accommodation notes: Large pool, suites that sleep 4.",
"groundedScore": 2,
"groundedRationale": "The response invents a specific accommodation (\"Orlando Family Suites\") and a precise price per night ($180) without any source or user-provided data, which is a hallucination. While it correctly flags missing departure city and trip length, the inclusion of unverified concrete details reduces the overall groundedness. The rest of the answer is cautious, but the fabricated hotel information prevents a higher score.",
"output": {
"destinations": [
{
"name": "Orlando",
"country": "USA",
"reason": "Warm in spring, huge range of family resorts with pools, no international flight needed."
},
{
"name": "Cancun",
"country": "Mexico",
"reason": "Direct flights from most US hubs, warm in winter, all-inclusive resorts with pools."
},
{
"name": "Punta Cana",
"country": "Dominican Republic",
"reason": "Reliable beach weather in February and a wide range of nice hotels at varying price points."
}
],
"flights": null,
"accommodation": {
"name": "Orlando Family Suites",
"type": "resort",
"pricePerNightUSD": 180,
"notes": "Large pool, suites that sleep 4."
},
"budget": {
"estimatedTotalUSD": null,
"withinBudget": null,
"breakdown": "Could not compute a total — missing flight price, hotel price, or trip length."
},
"reasoning": "Orlando is the best fit because it explicitly matches the trip vibe of warm weather during spring break, is family-oriented with kids in mind, and offers large pool-equipped resorts suitable for a group of four. Unlike Cancun and Punta Cana, which are more beach/relaxation focused and have tags indicating winter (not spring break), Orlando is directly tagged for spring break and family travel. Additionally, Orlando is in the USA, eliminating international travel complications for a family with children, and the hotel is specifically designed to sleep four in suites — ideal for the traveler count. While flight and duration data are missing, the hotel cost is lower than the alternatives, making it more likely to fit within the $5,000 budget if flight costs are reasonable.",
"caveats": [
"No departure city given — flight price is not estimated. Provide an origin for a flight quote.",
"Trip length not specified — assumed 5 days for budgeting purposes.",
"Flight prices are missing — if flights are expensive (e.g., from a remote origin), the budget could be exceeded.",
"Trip duration is unknown — if the trip is longer than 7–8 nights, the hotel cost alone may exceed the budget.",
"Budget is not flexible, so any unexpected costs (e.g., meals, park tickets) could strain finances.",
"Orlando’s spring break weather is warm but not guaranteed — occasional cold snaps can occur."
],
"confidence": "medium"
}
},
{
"name": "anniversary-no-santorini",
"input": "Anniversary trip in May, somewhere romantic and not too touristy. We hated Santorini. Budget flexible but flag if it's blowing past $4k.",
"schemaValid": true,
"constraintsPass": true,
"constraintDetail": "destinations: Azores, Sicily, Punta Cana",
"groundedScore": 2,
"groundedRationale": "The response invents specific details not provided by the user, such as a named boutique hotel ('Azores Ocean View Lodge') and an exact nightly price of $190, which are not grounded in any supplied data. It also includes an inaccurate claim about Punta Cana's weather in February, unrelated to the user's May travel window. While the agent does flag missing information (origin, traveler count, trip length) and notes the inability to compute a total budget, the presence of fabricated specifics and factual errors reduces the overall groundedness.",
"output": {
"destinations": [
{
"name": "Azores",
"country": "Portugal",
"reason": "Dramatic scenery, far less touristy than the Greek islands, good for a quiet romantic trip."
},
{
"name": "Sicily",
"country": "Italy",
"reason": "Romantic coastal towns with a more local feel than the typical Greek-island circuit."
},
{
"name": "Punta Cana",
"country": "Dominican Republic",
"reason": "Reliable beach weather in February and a wide range of nice hotels at varying price points."
}
],
"flights": null,
"accommodation": {
"name": "Azores Ocean View Lodge",
"type": "boutique hotel",
"pricePerNightUSD": 190,
"notes": "Small, quiet, scenic — not a tourist-heavy area."
},
"budget": {
"estimatedTotalUSD": null,
"withinBudget": null,
"breakdown": "Could not compute a total — missing flight price, hotel price, or trip length."
},
"reasoning": "The Azores best match the request for a romantic, not-too-touristy destination in May. It is explicitly tagged as 'romantic' and 'not-touristy', and its description highlights dramatic scenery and quietness — aligning perfectly with the trip vibe. Unlike Punta Cana, which is more resort-focused and better suited for winter, and Sicily, which while romantic is slightly more well-known and potentially busier, the Azores offer a secluded, nature-rich experience ideal for couples seeking privacy. The hotel option is a boutique, quiet lodge, reinforcing the romantic, low-key vibe. Budget flexibility allows for potential flight costs to be absorbed, as the hotel is reasonably priced at $190/night.",
"caveats": [
"No departure city given — flight price is not estimated. Provide an origin for a flight quote.",
"Traveler count not specified — assumed 2 travelers.",
"Trip length not specified — assumed 5 days for budgeting purposes.",
"Flight prices are missing, so total budget cannot be confirmed — flights to the Azores may be expensive depending on origin.",
"Trip duration is unknown, so total hotel cost cannot be calculated.",
"May weather in the Azores can be cool or rainy, which may affect outdoor romantic activities despite the scenic appeal."
],
"confidence": "medium"
}
},
{
"name": "lisbon-existing-flight",
"input": "I'll be in Lisbon Oct 10-13 for a wedding. Want to tack on 5 days somewhere nearby. Already have the flight to Lisbon.",
"schemaValid": true,
"constraintsPass": false,
"constraintDetail": "flights=null, destinations=Nearby",
"groundedScore": 2,
"groundedRationale": "The response fabricates specific details (e.g., a hotel named \"Standard Mid-Range Hotel\" at $150/night, a total budget estimate) without any supporting mock catalog data or user-provided information. While it includes many caveats and acknowledges the lack of concrete data, it still presents these invented figures as if they were realistic options, which is a hallucination. The agent should have asked clarifying questions or limited its answer to the known facts rather than providing ungrounded estimates.",
"output": {
"destinations": [
{
"name": "Nearby",
"country": "Unknown",
"reason": "Echoed back from your request — no mock catalog data exists for this destination."
}
],
"flights": null,
"accommodation": {
"name": "Standard Mid-Range Hotel",
"type": "hotel",
"pricePerNightUSD": 150,
"notes": "No specific match in our mock data — generic estimate."
},
"budget": {
"estimatedTotalUSD": 600,
"withinBudget": null,
"breakdown": "Flights: $0 x 2 traveler(s) = $0. Hotel: $150/night x 4 night(s) = $600. Total: $600."
},
"reasoning": "The request specifies 'nearby' as a destination hint with no other specific destinations provided, and no mock data exists for any other candidate. The only available option is labeled 'Nearby' with a generic hotel estimate of $150/night for 4 nights (5-day trip, assuming one night is travel), totaling $600. Since the user has an existing flight and no budget is specified but is not flexible, this is the only feasible option that matches the echo of the request.",
"caveats": [
"No mock catalog entry for \"nearby\" — destination details below are a placeholder, not verified data.",
"Traveler count not specified — assumed 2 travelers.",
"No actual destination or country is defined for 'Nearby' — it is a placeholder.",
"Flight cost is assumed $0 due to existing flight, but no details are provided to verify this.",
"Hotel data is a generic estimate with no real property match in the mock catalog.",
"Budget total is estimated at $600, but user's actual budget is unknown — risk of mismatch if budget is lower.",
"User already has a flight booked — flight cost excluded from budget."
],
"confidence": "low"
}
},
{
"name": "tokyo-repeat-visitor",
"input": "Tokyo for 8 days in November, two adults, $6k budget excluding flights. We've been once before.",
"schemaValid": true,
"constraintsPass": true,
"constraintDetail": "destinations=Tokyo, Kyoto, Hakone",
"groundedScore": 2,
"groundedRationale": "The response correctly flags missing flight information and notes that budget cannot be computed, which shows appropriate caveating. However, it invents a specific hotel name ('Shinjuku Central Hotel') and a precise price per night ($180) that were not provided by the user or any supplied data, constituting a hallucination. This undermines the groundedness despite the correct handling of other uncertainties.",
"output": {
"destinations": [
{
"name": "Tokyo",
"country": "Japan",
"reason": "Matches the requested destination directly — major food scene, good November weather."
},
{
"name": "Kyoto",
"country": "Japan",
"reason": "A short bullet-train ride from Tokyo — a good pairing for travelers who have already done Tokyo."
},
{
"name": "Hakone",
"country": "Japan",
"reason": "Hot springs and mountain views within day-trip distance of Tokyo, good for a return visitor."
}
],
"flights": null,
"accommodation": {
"name": "Shinjuku Central Hotel",
"type": "hotel",
"pricePerNightUSD": 180,
"notes": "Central, well connected by train."
},
"budget": {
"estimatedTotalUSD": null,
"withinBudget": null,
"breakdown": "Could not compute a total — missing flight price, hotel price, or trip length."
},
"reasoning": "The request explicitly hints at Tokyo as the destination, and the candidate data for Tokyo directly matches this preference. Tokyo is listed with relevant tags including 'tokyo', 'japan', 'food', and 'november', aligning with the travel season and likely interests. While all three options are in Japan and suitable for November, Tokyo is the only one that fulfills the explicit destination hint. The hotel cost of $180/night for 8 nights is $1,440, and while flight costs are missing, Tokyo is the most logical primary destination given the request.",
"caveats": [
"No departure city given — flight price is not estimated. Provide an origin for a flight quote.",
"Flight prices are missing for all candidates, so total budget cannot be verified.",
"Assumed no additional transportation or activity costs beyond hotel and flight.",
"Budget flexibility is false, so any unexpected cost could exceed the $6,000 limit.",
"No flight data means we cannot confirm if a direct flight to Tokyo is available or affordable."
],
"confidence": "medium"
}
},
{
"name": "minimal-info",
"input": "Cheap.",
"schemaValid": true,
"constraintsPass": false,
"constraintDetail": "confidence=medium, caveats=8",
"groundedScore": 1,
"groundedRationale": "The response invents a specific hotel name and a precise price per night ($220) despite the user providing no details beyond a vague request for something \"Cheap.\" This constitutes confident hallucination rather than grounded inference. While the answer includes caveats about missing information, it still presents unsupported specifics, violating the requirement to avoid fabricating facts and to clearly flag unknowns.",
"output": {
"destinations": [
{
"name": "Cancun",
"country": "Mexico",
"reason": "Direct flights from most US hubs, warm in winter, all-inclusive resorts with pools."
}
],
"flights": null,
"accommodation": {
"name": "Cancun Beachfront Resort",
"type": "all-inclusive resort",
"pricePerNightUSD": 220,
"notes": "Pool, beach access, family-friendly."
},
"budget": {
"estimatedTotalUSD": null,
"withinBudget": null,
"breakdown": "Could not compute a total — missing flight price, hotel price, or trip length."
},
"reasoning": "Cancun is the only candidate provided and matches several implicit preferences: it is a warm, beach-oriented destination with a pool and family-friendly all-inclusive resort, which aligns with common travel preferences when no specific constraints are given. The hotel is well-described and fits a typical leisure traveler’s expectations.",
"caveats": [
"Very little information was provided — this suggestion is a generic placeholder, not a tailored match.",
"No departure city given — flight price is not estimated. Provide an origin for a flight quote.",
"Traveler count not specified — assumed 2 travelers.",
"Trip length not specified — assumed 5 days for budgeting purposes.",
"Flight price is missing, so total budget cannot be verified.",
"Trip duration is unknown, making total cost estimation impossible.",
"BudgetTotalUSD is null, so it's unclear if the stay is affordable.",
"No other destinations are provided to compare against."
],
"confidence": "medium"
}
}
]
What you were trying to do:
I wanted to separate what the LLM should decide (destinations, reasoning) from what code should compute (prices, budget math). The eval showed the split was actually working correctly, but the judge couldn't distinguish mock-data numbers from hallucinated ones. We kept the architecture and fixed the judge rubric instead, since the design itself was sound.
What the tool gave you back (briefly):
The Lisbon case was broken because the extractor latched onto the word "nearby" instead of the city name "Lisbon." This cascaded into a placeholder response with a fabricated generic hotel. We added an explicit rule and a few-shot example to anchor on the named city, which fixed it and correctly surfaced Porto and Sintra as nearby options.
What you kept, changed, or threw away — and why:
Confidence calibration was being left entirely to the LLM, which kept it too high even for very vague queries. We narrowed the override to just one case — when a user names a specific place we have no catalog data for — and let everything else stay as-is, since a vague query like "Cheap." is genuinely flexible and medium confidence is defensible there.
should we sepereate the llm-as-judge vs. the inference model for customers? in terms of cost vs. efficiency we might be able to use cheaper models for less critical tasks, though for a judge we can use a better model for more reliablity
What you were trying to do:
I wanted to make this a choice for the user to pick the right model for their usecase. In addition, it helped with my iterations over code and evaluations to be faster as I can use faster models for the inference, and use more reliable models for evaluations What the tool gave you back (briefly): It agreed and executed the task for me What you kept, changed, or threw away — and why: The suggested changes were perfect