Beta Update August #2 - LLM Evaluation and Beta Expansion

We are nearing the end of the month, so it's time for yet another update from the flight deck. With this time: new documentation, a chatbot video tutorial, a poll to decide which LLMs we are adding next, and much more. Enjoy!

New Documentation

In the last update, we reported on all the new features in Seaplane. It’s about time we explain how you can use all those awesome features in production today. We released documentation on the following:

• Langchain Integration
• Foundational Model Support (GPT-3.5 and MPT-30B with more models coming soon! Keep reading to learn more.)
• Multipart form data - Enabling support for file uploads in POST requests
• Text Embeddings

Adding support for new LLMs and other models

We are soon adding support for more foundational models but need your input. Which models should we add first to create the most value for our beta users?

Vote on this poll on LinkedIn!

New Blog: Evaluating and Measuring LLM Performance: A Quantitative Approach

In a previous blog post, we compared two LLMs (GPT-3.5 and MPT-30B) by asking all of our co-workers which output they preferred on prompts in three categories. While it's a valid method, it remains a qualitative experiment heavily influenced by individual opinions.

This time we will evaluate LLMs quantitatively by looking at the cosine similarity of different LLMs outputs and compared to a reference answer. In the end, we propose a framework that you can use to evaluate the performance of any LLM over time.

Take off HERE

New Video Tutorial: Chatbot with in-context learning of Notion pages

With all the great new features, such as our Langchain integration and vector store, we have enabled production-grade chatbot use cases. In this video tutorial, we will show you how you can use them to train a chatbot (in-context learning) on any Notion repository.

Watch HERE.

Beta Expanding

We are expanding the number of users in our beta. If you have not already, sign up here to get early access.

Join the Seaplane Beta.

That was all for the second update of the month. Thank you for flying with Seaplane, and until next time.

Cheers,

Fokke and the rest of the Seaplane team.

‍