Skip to main content

Streaming chat response

Middlebop also supports streaming chat response for that "word for word" live writing look. When revieving the response you'll get it as soon as the AI models has processed it. Every bit/event is called a chunk. To simplify the stream data structure it follows the same message type syntax as a non streaming chat response.

See our supported models

OpenAI​

curl -X POST "https://api.example.com/v1/streaming/chat/openai/" \
-H "Content-Type: application/json" \
-H "x-api-key: your_api_key_here" \
-d '{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": {
        "type": "text",
        "text": "Hello, who are you?"
      }
    }
  ]
}'

Response​

The response is a MiddlebopChatCompletionStreamChunk that is passed to your response handler one at a time. It looks like this

// first chunk
{
  "type": "chunk",
  "model": "gpt-4",
  "messages": [
    {
      "role": "assistant",
      "content": {
        "type": "text",
        "text": "Hello"
      }
    }
  ]
}

// second chunk
{
  "type": "chunk",
  "model": "gpt-4",
  "messages": [
    {
      "role": "assistant",
      "content": {
        "type": "text",
        "text": "!"
      }
    }
  ]
}

// third chunk
{
  "type": "chunk",
  "model": "gpt-4",
  "messages": [
    {
      "role": "assistant",
      "content": {
        "type": "text",
        "text": " I"
      }
    }
  ]
}

// etc...

Google​

Google streaming chat works pretty much the same way. It can use the same Middlebop API key and input and response looks the same.

curl -X POST "https://api.example.com/v1/streaming/chat/google/" \
-H "Content-Type: application/json" \
-H "x-api-key: your_api_key_here" \
-d '{
  "model": "gemini-pro",
  "messages": [
    {
      "role": "user",
      "content": {
        "type": "text",
        "text": "Hello, who are you?"
      }
    }
  ]
}'

Response​

The response ofc looks the same

// first chunk
{
  "type": "chunk",
  "model": "gemini-pro",
  "messages": [
    {
      "role": "assistant",
      "content": {
        "type": "text",
        "text": "Hello"
      }
    }
  ]
}

// second chunk
{
  "type": "chunk",
  "model": "gemini-pro",
  "messages": [
    {
      "role": "assistant",
      "content": {
        "type": "text",
        "text": "!"
      }
    }
  ]
}

// third chunk
{
  "type": "chunk",
  "model": "gemini-pro",
  "messages": [
    {
      "role": "assistant",
      "content": {
        "type": "text",
        "text": " I"
      }
    }
  ]
}

// etc...