Agents, RAG and evals for AI that actually works.
I'm Paulo Vitor Souza, an AI engineer with 4+ years of software engineering experience. I build agents and assistants that answer from a company's own documents, and the benchmarks that show whether they get it right before it reaches the customer.
| Test case | Base model | Fine-tuned |
|---|---|---|
| Cites the right clause | passed | passed |
| Refuses when there is no source | failed | passed |
| Uses the domain vocabulary | failed | passed |
| Answers in the requested format | passed | passed |
| Total | 2 of 4 | 4 of 4 |
Ask my assistant
It was built with the same technique I use for companies: it answers from what I published on this site. It doesn't know what was left out, and it can be wrong.
What I do
- AI built into workflows
- AI where the work happens: agents and automations that plug into company workflows to cut manual work and gain efficiency.
- Custom evals and benchmarks
- Benchmarks specific to your domain and evaluation harnesses to measure quality, compare models and catch regressions before production.
- Fine-tuning and model adaptation
- Adapting models to the company's vocabulary, tone and tasks, with curated data and evaluation before and after.
- RAG and document assistants
- Assistants that answer from the company's documents: ingestion, semantic search and answers that cite their sources.
I also build web products in TypeScript, from React/Next.js to APIs, and take care of observability in production.
Projects
| Project | What it is | Tools |
|---|---|---|
| Eval harness and benchmarks | Domain benchmark with a multi-model runner, per-case scores and a test that blocks quality regressions. It runs the same questions against every model in this site's chat. | LLM evals, Benchmarks, TypeScript |
| Domain fine-tuning | Fine-tuning a small model on a domain dataset, with evaluation before and after to measure the real gain. | Fine-tuning, LoRA, Unsloth |
| RAG over documents | Ingests documents, runs semantic search and answers citing the sources. The playground's hybrid retrieval gets 35 out of 35 questions of the evaluation set right (100% recall@5). | RAG, Embeddings, Vector DB |
| This portfolio | Static Next.js site exported to Cloudflare Pages, an AI assistant in a separate Worker and Three.js experiments. Case study: How the chat works. | Next.js, TypeScript, Cloudflare Workers, Three.js |
Day-to-day tools: TypeScript, Node.js, React / Next.js, Python, PostgreSQL, AWS, Docker, LLM APIs, Embeddings / Vector DB, Evals, Datadog.
Writing
- How the chat works The architecture of this site's assistant: Worker, hybrid RAG, AI Gateway and streaming.
- Postmortem: the eval that took the chat down The Workers AI quota ran out, the chat's search failed silently and the metrics stayed green. How to get to the cause.
- Hobbies Path of Exile, web novels, manhwa and manga: my favorites.