hev.dev / reinvent
re:trieval
A curated list of sessions across search and evals · picked from 2,241 sessions · every database, observability and eval booth on the floor
Excited to join for another year at re:Invent. The event has grown beyond being reasonable, so I decided to have my agents navigate the schedule for me, and I'm sharing the results with you.
My focus is to connect with other builders in retrieval and observability, and to share some laughs and insights. This track intentionally skips the keynotes in favor of smaller breakouts with teams demonstrating excellence in search.
To add to the fun, I budgeted for a limo to bring us to happy hour if we can get enough signups. Drop your email if you're interested.
See you in Vegas.
Where you see my head, I'll be there. Say hi, or book 20 minutes if you want to talk retrieval, evals, agent traces or production search. I'm Adam Hevenor, and I run hev mind. On Wednesday evening I'm running a limo from MGM to a happy hour: get on the list.
Reserved seating opened Oct 6 and nearly every chalk talk, builders' session and workshop here is already full. Breakouts and lightning talks are walk-up only, so just show up. Seats checked Oct 15; full rooms still take walk-ups at the door, and reserved seats are released 10 minutes before start.
Getting around
staying at
walk
shuttle, car or monorail
I'll be there
I'll be there
Schematic, roughly to scale. Travel times are estimates for conference week, when Strip traffic is at its worst. re:Invent uses five venues this year; Mandalay Bay isn't one of them.
Where to stay
Both tracks lean toward the south end, while the expo, keynotes and most of the nights are at the north end. I added up the travel between venues for the search track from each end, and it comes out about even: roughly four hours across Monday to Thursday either way. So choose by what you're coming for.
MGM Grand, for the talks
- About 80 of the retrieval session slots are here, against 13 at the Venetian
- About 27 of the eval session slots are here, the most of any venue. The rest spread across Wynn (16), Caesars Palace (12) and the Venetian (10)
- Badge pickup on site
- The closest venue to the airport, about 15 minutes
- A room upstairs on the long session days (Tue afternoon, all of Wed)
Venetian side, for the floor and the nights
- The expo, all the keynotes and the welcome reception are here
- The Venetian, Palazzo, Treasure Island and Wynn are all within walking distance of each other
- Closer to re:Play (Thu night, Festival Grounds)
- Badge pickup on site as well
Flying in? There's also badge pickup at Harry Reid International, in Terminals 1 and 3.
What the catalog says
- Evals moved from the model to the agent. Most of the eval sessions score whole agent runs: tool calls, trajectories and whether a multi-turn conversation reached its goal, not single prompts. AWS's own AgentCore Evaluations comes up in more than a dozen of them.
- The judge needs checking too. LLM-as-judge is the default scorer now, and the better talks are about keeping it honest. Vanguard sends every disagreement between the judge and a subject-matter expert back into calibration. Arize argues the judge itself has to become an agent that reads traces, because production failures arrive after the rubric was written.
- Traces become test sets. The same loop shows up again and again: capture OpenTelemetry traces in production, turn the failures into a versioned dataset, and gate the next release on it.
- Almost nobody evaluates retrieval itself. Only two sessions measure retrieval quality directly: Braintrust's benchmark of Bedrock Knowledge Bases, and a lab that scores RAG answers for groundedness. Relevance judgments, recall against a golden set and ranking metrics barely appear. If you build search, that's the gap, and the hallway is the place to talk about it.
- AWS put vector search in every database it owns. There's S3 Vectors, DynamoDB vector indexes, OpenSearch GPU indexing with cuVS, Aurora pgvector hybrid search, semantic caching on ElastiCache, Neptune GraphRAG, and a new product called AWS Context. Nearly every abstract makes the same pitch: you don't need a separate vector database.
- Vendor highlights include turbopuffer and Zilliz. Notably missing from the expo this year are Pinecone, Weaviate, Qdrant and LanceDB. The retrieval companies with sponsored talks are Elastic, MongoDB, Redis, Exa, Neo4j, Glean and Braintrust.
- RAG really is dead, and the industry now says "agent retrieval" instead. The OpenSearch team says it directly: agents fire hundreds of sub-queries per task. That changes the latency budget, the cache and the cost model.
- Multi-tenant isolation has its own sessions now. Per-tenant indexes, scoped credentials and cache namespacing come up in at least four talks. It's become a real problem people pay to solve.
core pick
alternative
expo time
I'll be there
I'll be there
Mon Nov 30 · S3 Vectors and DynamoDB at MGM, then the welcome reception· judges and guarantees at MGM, then the reception
8:00–10:00lab
Jonathan Dion (Principal Technical Instructor)
The only hands-on session that scores RAG answers. Build an evaluator that checks groundedness and one that measures goal success across whole conversations, then trace failures back to their cause.
Caesars Palace · Palace Ballroom II
9:00–10:00sponsored
Aparna Dhinakaran (Founder and CPO, Arize AI)
A rubric catches the failure it was written for, and agents keep failing in new ways. This is the case for a judge with tool access that reads the trace and labels what went wrong, and how to check that judge against human labels.
MGM Grand · Grand 119
10:30–11:30chalk talk · 500
Weitong Ruan (Senior Applied Science Manager)
Benchmarks and red-teaming say little about how often something goes badly wrong. This is about putting statistical bounds on failure rates instead, in a small Distinguished-level room.
MGM Grand · Boulevard 155
1:00–2:00chalk talk
Vishal Srivastava, Neha Wadhwani
An open-source framework: calibrate an automated judge against human labels, track failure rates in CI, and in a multi-agent system, blame the agent that caused a failure rather than the one where it surfaced. It repeats Wed 9:00 at Wynn.
Caesars Palace · Octavius 11
8:30–9:30breakout
Dhanika Sujan (S3 Vectors PM), Venkata Sistla
The product manager for S3 Vectors. The talk will be on YouTube, but the conversation afterward won't.
MGM Grand · Chairman's 363 · Purple Theater
9:00–10:00breakout
Andrew Ross (Principal Engineer, OpenSearch TSC chair), Dotan Horovits
The OpenSearch 3.x roadmap from the TSC chair. Watch it later unless you want Andrew in person.
Caesars Forum · Forum 120 · Blue Theater
9:00–10:00chalk talk · 400
Don Simpson (Principal Technologist), David Gardner
Treating the context window as an attention budget, and tracing a task as it builds context from Aurora, Valkey and Neptune. It repeats Tue 4:00 and Wed 2:30 at Caesars Palace if STG301 takes this slot.
MGM Grand · Boulevard 165
10:30–11:30chalk talk
Venkata Sistla, Samyak Kathane
Object storage as the long-term memory tier for agents, in a small room: embeddings in S3 Vectors, working state in S3 Files.
MGM Grand · Boulevard 162
11:30–12:30chalk talk
Aakash Upadhyay (Principal Engineer), Hans Nesbitt
A production system serving thousands of tenants, plus the question every retrieval team eventually faces: when to build your own stack and when to use a managed knowledge base.
MGM Grand · Premier 311
1:00–2:00breakoutI'll be there
I'll be thereRashmi Krishnaiah Setty, Abhinav Duggal (Principal Engineers)
Vector indexes that shard with the table, scatter-gather queries, filtering that doesn't wreck recall, and the partition key as the tenant boundary. The most substantial architecture talk in the catalog.
MGM Grand · Chairman's 363 · Purple Theater
3:00–4:00breakoutI'll be there
I'll be thereJordan Dolman (Principal PM, AWS Storage)
S3 Files, S3 Vectors and S3 Metadata pitched as one agent stack, by the product lead. Take the monorail from MGM to Harrah's/LINQ to get here.
Caesars Forum · Forum 120 · Blue Theater
3:00–4:00sponsored
DAT318-S10x faster vector indexes: GPU acceleration in Amazon OpenSearch with NVIDIA cuVSadded Oct 2walk-up
Vamshi Vijay Nakkirtha (Sr Software Engineering Manager, OpenSearch)
How index builds move to the GPU at billion-vector scale. NVIDIA sponsors it, but the speaker manages the engineers who built it.
Caesars Palace · Roman I
4:00–7:00expoI'll be there
I'll be thereWelcome reception, the first night on the floor
About 450 exhibitors, with food and drinks. It's the least crowded the floor will be all week. Start with the vector and search booths (see the floor list) before the people you want to meet are pinned down by demos.
Venetian · Level 2 · Hall B
4:30–5:30code talk · 400
Raviteja Sunkavalli, Sudhir Kumar (Sr Software Development Manager)
Live coding: OpenTelemetry and OpenInference traces on a multi-agent app, LLM-as-judge evaluators, then production traces pulled into a versioned dataset to reproduce a failure. It runs again Wed 3:00 at Wynn, so Monday can go to the reception.
Caesars Forum · Forum 120 · Code Talk
5:00–5:20lightning · expoI'll be there
I'll be thereUK AI Security Institute
How a government safety institute built repeatable eval infrastructure for frontier models: model access, prompt orchestration and result analysis. Twenty minutes, in the middle of the reception.
Venetian · Hall B · Global Impact Theater
6:00–6:20lightning · expoI'll be there
I'll be thereRan Pergamin, on Bright Data's build
1.5 trillion video vectors: partitioning, query routing, metadata filtering, and cost per million queries against dedicated vector databases. It's in the middle of the reception.
Venetian · Hall B · Industry Theater
Tue Dec 1 · keynote and the floor, then the S3 Vectors internals talk· keynote, then three eval talks on the expo floor
8:30–10:30keynote
Opening keynote, Matt Garman
This is where the week's biggest launches get announced. It's also streamed, if you'd rather spend the morning on meetings.
Venetian
10:30–2:15expoI'll be there
I'll be thereThe main expo block
The floor opens at 10, right as the keynote lets out next door. This is the longest window in the track, so use it for the booths you most want. The Braintrust talk below is in the middle of it.
Venetian · Hall B · open 10:00–6:00
10:30–10:50lightning · expoI'll be there
I'll be thereJoel Johnson (Deployed Engineer, LangChain)
What LangChain's deployed engineers see in the field: eval practice, plus the team and process side that decides whether anyone uses the evals.
Venetian · Hall B · Builder Theater
12:00–12:20lightning · expoI'll be there
I'll be thereBraintrust
An independent retrieval eval with p50/p95 latency, cost, relevance, citation support and confidence intervals. That's the level of rigor retrieval benchmarks should have.
Venetian · Hall B · Industry Theater
1:00–2:00code talk
Guille Ojeda
Turn real failure modes into test cases that score both answers and actions, then gate CI/CD on risk. There's a public reference implementation. It repeats Thu 12:00 at Caesars Forum.
Wynn/Encore · Cristal 2 · Code Talk
3:00–3:20lightning · expo
Piyush Saxena (Head of AI/ML), Ritesh Shah (Chief Architect, AI/ML), Vanguard
A judge scores every response, and an expert confirms or overturns it. Agreements sharpen the agent, disagreements recalibrate the judge, and the expert labels grow into a benchmark. The customer is giving the talk, which is the reason to be there.
Venetian · Hall B · Partner Advantage Theater
3:00–4:00sponsored
ClickHouse
Langfuse for traces, trajectories and production quality scores, with failures turned into datasets. Take it if you're already at MGM for the afternoon.
MGM Grand · Chairman's 363 · White Theater
11:30–12:30sponsored
Columnar and vector modes, Jina embeddings, native inference. Elastic has a Diamond booth, so you can also catch them on the floor without leaving the Venetian.
MGM Grand · Grand 117
3:00–4:00chalk talk · 400
STG419-R1Inside Amazon S3 Vectors: architecture and cost optimization for vector searchtop pickrepeats Thufull
Dmitri Pavlichin (Principal Applied Scientist)
A whiteboard walkthrough of the indexing approach behind 2B vectors per index and "90% cheaper", plus what the 100× top-k increase and the new query pricing mean. If you only take one talk this week, take this one.
MGM Grand · Boulevard 162
3:00–5:00workshopI'm booked
I'm bookedWale Akinfaderin
Text, image, video and audio in one embedding space. The S3 Vectors chalk talk filled up, so this is where I'll be at 3:00.
MGM Grand · Premier 314
4:00–5:00sponsored
MongoDB, on ArcXP's build
Hybrid lexical and vector search over a news archive, including a pipeline that made about five million historical images searchable by meaning. A good pick if you stayed at the Venetian.
Venetian · Ballroom F · Red Theater
4:30–5:30builders'
Kevin McGehee, Elad Bernstein (Principal Engineers), Yaron Sananes, Eran Balan
Similarity thresholds, per-user cache namespaces, and keeping one tenant's cached answer from leaking to another. It's the cache layer of retrieval, taught by the people who built it.
MGM Grand · Chairman's 363 · Builders' Session 1
Wed Dec 2 · the first hour on the floor, then the frontier and the ranking problem· the eval booths, then build a pipeline
8:30–10:00keynote
Keynote
It's streamed, so this is a good slot for a breakfast meeting.
Venetian
10:00–11:00expoI'll be there
I'll be thereFirst hour on the floor
Go back to the booths you missed on Tuesday, before the keynote crowd arrives. Leave by 11 to make INV505 at MGM.
Venetian · Hall B · open 10:00–6:00
10:00–12:30expoI'll be there
I'll be thereThe eval booths
Arize (Platinum), Braintrust and LangChain (Gold) and Openlayer (Showcase) are all on the floor. Go before lunch, then walk to Caesars Palace.
Venetian · Hall B · open 10:00–6:00
11:30–12:30chalk talk · 500Full · I'll try walk-up
Full · I'll try walk-upSpeakers not yet announced
The only Distinguished-level retrieval session: multi-modal, multi-step, and keeping accuracy up while holding latency and cost down. It clashes with INV510 (S3's architecture, James Bornholt), which is also excellent.
MGM Grand · Premier 311
12:00–1:00breakout
Meet Bhagdev (Head of Product, ElastiCache, Neptune & MemoryDB), Madelyn Olson
He runs product for three of AWS's vector surfaces.
Caesars Forum · Forum 120 · White Theater
12:00–1:00breakout · 400
Steve Ash (Senior Principal Engineer)
The launch talk for AWS's governed knowledge graph for agents. It clashes with INV505 and it's recorded, so watch it later unless you want Steve in person.
MGM Grand · Chairman's 363 · White Theater
1:30–2:30chalk talk · 400I'm booked
I'm bookedNihilson Gnanadason, Daniel Wirjo
ABAC with STS session tags and a token vending machine that mints per-tenant credentials, instead of hundreds of IAM policies. This is the pattern AWS will tell every multi-tenant AI company to copy.
MGM Grand · Level 3 · Room 353
1:30–2:30builders'
Grant McAlister (Sr Principal Engineer), Radwan Deeb (Principal Engineer), Shayon Sanyal, and others
Full-text search plus pgvector, reciprocal rank fusion, reranking and retrieval diagnostics, all in Postgres. Go for the speakers rather than the lab. There's also a Monday run at 4:00 (MGM, Room 301), during the welcome reception.
MGM Grand · Chairman's 363 · Builders' Session 1
4:00–5:00chalk talk · 400Full · I'll try walk-up
Full · I'll try walk-upOpenSearch team
A bring-your-problems format covering hybrid search, RRF, MMR, neural sparse, quantization, tiered storage and GPU indexing. Come with a real question.
MGM Grand · Boulevard 163
1:00–2:00builders'
Suyin Wang, Sam Palani (WW Leader, 1P Foundation Models), Wale Akinfaderin
Hands-on: evaluation datasets, LLM-as-judge and programmatic scoring, tracking over time, and alerts on regressions. It repeats Thu 11:30 at MGM.
Caesars Palace · Octavius 4
1:00–2:00chalk talk · 500
Rahul Gupta (Senior Manager, Amazon)
Deception that single-turn evals miss: a benchmark that compares a model's output against its chain of thought, and a multi-agent simulation that surfaces deception across long task sequences. Safety evals, not product evals, but Distinguished-level and in a small room.
MGM Grand · Room 350
3:00–4:00code talk · 400
Raviteja Sunkavalli, Sudhir Kumar
The second run of Monday's live-coding talk.
Wynn/Encore · Margaux 1
4:30–5:30chalk talk
Luca Bianchi (CTO, comPVter)
Harvest an eval set from production, set an accuracy floor, benchmark the candidate models, and route each request to the cheapest one that clears the floor. He reports about 80% lower cost. A practitioner, not a vendor.
Caesars Forum · Alliance 305
5:00–7:30hostedI'll be there
I'll be therere:trieval limo + happy hour
Hosted by me, for both tracks
MGM to the north end is the slowest trip of the week, so I'm taking care of it. A limo leaves MGM at 5:00, after ANT442 lets out, and drops everyone at a happy hour near the Venetian from 5:30 to 7:30. If you're on the eval track, it's a short walk over from Caesars Forum. That's early enough that you can still get to the night's parties afterward. There are about a dozen seats in the car and more room at the bar. Sign up and I'll send you the details.
Leaves MGM Grand at 5:00 · bar near the Venetian, announced to signups
Thu Dec 3 · agent retrieval, multi-tenancy, re:Play· MGM all day: the harness, eval-first skills, the evaluator
8:00–10:00workshop
Amit Kalawat, James Wu
Multimodal embeddings tuned for machine recall, hybrid search and an entity graph over video, built for agents rather than people.
Caesars Forum · Forum 110
9:00–10:00chalk talk · 400
Leonardo Gomez Virahonda (Principal Specialist SA)
Carrying existing data permissions into the context layer, so each user's identity flows through the agent. It's the governance half of AWS Context, one door down from ANT322. It also runs Wed 12:00 at Caesars Palace.
MGM Grand · Room 353
10:00–11:00chalk talkFull · I'll try walk-up
Full · I'll try walk-upMuthu Pitchaimani (OpenSearch Specialist SA), Mikhail Vaynshteyn
Hybrid retrieval, MCP access and built-in memory, in a small room. If you missed the S3 Vectors talk on Tuesday, STG419-R runs at the same time on the same floor (Premier 320).
MGM Grand · Room 350
12:30–12:50lightning · expo
Exa
Passage-level results and context budgets, backed by production traces. Most of an agent's bill comes from the context it's given, not the model.
Venetian · Hall B · Builder Theater
12:50–2:30expo
Last pass on the floor
The expo closes at 4 today. This is the time to follow up with anyone you met on Monday or Tuesday.
Venetian · Hall B · open 10:00–4:00
1:00–2:00breakout
Tia White (General Manager, OpenSearch)
The clearest statement in the catalog of why agent retrieval is a different workload. It's a recorded breakout, so it can wait until January.
MGM Grand · Chairman's 363 · Red Theater
3:00–4:00chalk talk
Aakash Upadhyay (Principal Engineer), Hans Nesbitt
The repeat of Monday's talk.
Caesars Palace · Neopolitan III
10:00–11:00code talk · 400
Jonathan Buck (Principal Engineer), Po-Shin Chen
Tools, instructions, context, memory and runtime policy treated as one system you improve with controlled experiments, without overfitting to the benchmark. It also runs Wed 2:30 at Wynn.
MGM Grand · Boulevard 159
11:30–12:30builders'
Suyin Wang, Sam Palani, Wale Akinfaderin
The second run of Wednesday's builders' session.
MGM Grand · Room 301
12:00–2:00workshop · 400
Maxym Kharchenko (Principal Database Engineer), Shayon Sanyal
Write the evals first (lock contention, replica lag, vacuum blocked by a long transaction), then write an agent skill that passes them. Test-driven development for agent skills, taught by a principal database engineer.
MGM Grand · Premier 317
12:00–2:00workshop · 400
Rajesh Kumar Ravi, Abhinav Krishna
Traces, evaluators and offline batch evaluation, then live A/B tests through the gateway until the result is statistically significant.
MGM Grand · Premier 314
2:30–3:30chalk talk · 500
Rohit Malshe (Director, Research Science, Amazon)
LLM-guided code evolution across 403 problems. When the evaluation function gives a smooth signal, state of the art shows up whichever model you use. When it doesn't, no amount of compute helps. The best argument in the catalog that the eval is the product.
MGM Grand · Room 306
2:30–3:30sponsored
Steven Karis (Chief Architect, Resolve AI)
How an AI SRE company decides how far to trust an agent that acts on production. Take it if you're still at the Venetian.
Venetian · Ballroom F · Red Theater
7:30–midnightparty
re:Play
By the last night people have stopped performing, and the conversations get better.
Las Vegas Festival Grounds
Fri Dec 4 · a make-up session and an S3 workshop· nothing on the eval track
9:00–10:00chalk talk · 400
Nihilson Gnanadason, Daniel Wirjo
The Friday run of Thursday's talk, if re:Play took Thursday afternoon.
Wynn/Encore · La Tache 2
10:30–12:30workshop
Samyak Kathane, Kiran Chukkala
Agents on EKS sharing a mounted S3 file system as their workspace, with S3 Vectors as long-term memory. It's the hands-on version of Monday's STG325, with one of the same speakers.
Wynn/Encore · Cristal 1
all day
Nothing on the eval track
There are no eval sessions on Friday. Fly home, or switch to the search track for a make-up session.
On the expo floor · every database, observability and eval company exhibiting
This is from the sponsor catalog. The tier is a rough guide to booth size, from Emerald (largest) through Diamond, Platinum and Gold down to Showcase and Bronze. Booth numbers haven't been assigned yet, and this list will get them when they are. The companies at the core of the track you're viewing are in bold.
Databases
Vector
turbopufferbronzeZillizshowcase
Search engines
ElasticdiamondOpenSearchbronze
Document & key-value
MongoDBemeraldRedisdiamondDragonflybronze
Postgres
Tiger DatagoldpganalyzeshowcaseSupabasebronzeTessellbronzePerconabronze
Distributed SQL
OraclediamondCockroach LabsplatinumYugabyteDBgoldTiDB (PingCAP)gold
Real-time & analytics
ClickHouseplatinumVeloDBshowcaseRisingWaveshowcaseSingleStorebronzeMaterializebronze
Lakehouse & warehouse
DatabricksdiamondSnowflakediamondClouderadiamondStarburstshowcaseOnehousebronze
Graph
Neo4jgoldPuppyGraphshowcase
Database tooling & security
RedgategoldDBeavershowcaseBytebaseshowcaseDataSunrisebronzeFormalbronze
Retrieval APIs & models
ExagoldParallelgoldCoheregoldYou.comshowcaseTwelveLabsshowcase
Observability
Platforms
DatadogemeraldSplunkemeraldDynatracediamondCoralogixdiamondNew RelicplatinumDash0platinumgroundcoverplatinumGrafana LabsgoldHoneycombgoldLogicMonitorgoldSolarWindsgoldBrontogoldApicashowcaseZABBIXshowcaseSumo Logicbronze
AI observability & evals
ArizeplatinumBraintrustgoldLangChaingoldOpenlayershowcaseMonte CarloshowcaseLlamaIndexbronzeScale AIbronze
AI SRE & incident response
TraversaldiamondResolve AIdiamondPagerDutydiamondNeuBirdgoldCiroosgoldincident.iogoldRootlybronzeSelector AIbronze
Telemetry pipelines & runtime
CriblplatinumSentrygoldHudgoldMezmobronzeGreprbronzeChecklybronze
Not exhibiting as of Oct 2: Pinecone, Weaviate, Qdrant, LanceDB, Chroma, Vespa, TopK. Sponsors keep being added until November.
Watch later
- AIM278-S Open-source agent observability and evals with AgentCore and Langfuse. It clashes with Vanguard's talk on Tuesday.
- AIM232-S 4 primitives to evaluate your on-call agent's autonomy, from Resolve AI. It clashes with INV522.
- ANT337 Human search to agent retrieval: rearchitecting with OpenSearch Service, from Tia White, the GM of OpenSearch. It's the clearest statement in the catalog of why agent retrieval is a different workload.
- ANT423 Introducing AWS Context, from Steve Ash. It clashes with INV505.
- STG346 What's new with Amazon S3... and now files!, from Ben Grubin. S3 Files and S3 Vectors in one hour. No time yet.
- OPN311 What's new with OpenSearch, from Andrew Ross, the TSC chair.
- INV510 Inside Amazon S3: how a one-page design reached planetary scale. It clashes with INV505.