Update: Changelog #0008

Welcome to Butter’s eighth changelog! Just like grandma on Thanksgiving morning, we’ve spent a whole lot of time cooking.

Starting with:

Gramma’s Recipe Book

As a Thanksgiving treat, we launched cookwithbutter.com, which uses Butter’s LLM response caching to help generate, and cache, popular holiday cooking recipes.

Look up popular pre-computed recipes, or try your own! Gramma wants all your favorite recipes, from standard stuffing to creative butter, covered in butter, with a butter garnish.

The remainder of our time has been focused on internal tech improvements, which are still in-progress so we’ll only briefly highlight them below:

Data Engine Rewrite

Part necessity and part premature optimization, we’ve been rewriting our data backend to structure the cache tree in a much more efficient, S3-centric way.

This project, once completed, will include:

As low as 0 round trips to S3 for serving hot-path responses.
Truly stateless servers able to horizontally replicate and achieve high availability.
The ability to say we “rewrote in Rust” (meme).

Automatic Template Induction

The goal of template-aware caching is to expand the generalizability of cache entries by converting literal text messages into a more powerful composition of templates and dynamic variables.

Currently, this is a manual process, where users must explicitly flag their dynamic content in the butter-bindings request headers in order for those variables to be stripped into template placeholders. Powerful, but cumbersome, especially when agent intent isn’t known in at the time of programming.

Our goal is to make the process automatic, using hints such as attention values to map out which parts of the context window are noise (ignored), which are dynamic (templated), and which are structural (cached).

Two days ago, thanks to hard work from teammate Raymond, we’ve got our first proof-of-concept working end to end: