I’ve been using Azure OpenAI (AOAI) since its release and am here to fill you in on the 3 mistakes I made when I first started out. Let’s go!
1. Going too deep too fast
Like most people who are onboarded into the Azure OpenAI service, I tried to do everything at once. Burning the midnight oil testing things that I had no business testing and had no idea what I was doing. While this provided some fun nights tinkering with things, it also slowed down my ability to generate value quickly. To those of you who are new to AOAI: enjoy it and take your time. It’s not going anywhere.
2. Why am I Getting Throttled?
As I started to ramp up and become more familiar with the studio, I got hit with ‘Too many requests’ or ‘Token Length Exceeds…’ type of errors. I was unknowingly calling the API too many times or passing in too long of a string. This is a common occurrence when you start working with loops and batch processing. These were infuriating to try and develop off of. Here’s what it taught me:
- The token counts matter – don’t pass in everything just because you can
- Throttling is real – spread out your calls or add a wait step in your code to make sure you don’t send 100’s of calls in a matter of seconds
3. Sticker Shock
The pricing model is based on a dollar amount per 1,000 tokens and it varies by model. For reference, 1,000 tokens is roughly 800 words give or take and the gpt-4 family of models is more expensive than the gpt-3 family of models. A basic application built on gpt-3.5-turbo won’t cause you any heartburn and costs as much as your usual Chipotle order, but as you progress into the gpt-4 family, things change. As you begin to scale your application, add more users, and feed in larger amounts of tokens, you can easily jump from a Chipotle order to a budget trip to Disneyworld without even knowing it. This is not all bad though!
What you gain moving from the 3.5 to 4 models is significant enough to make the change… you gain better formatting, larger token amounts, greater context windows, and more. This is purely a word of caution. These tools are extremely powerful and can be game changers, just be aware of the implications.
