Edge Functions have a built-in API for running AI models. You can use this API to generate embeddings, build conversational workflows, and do other AI related tasks in your Edge Functions.
This allows you to:
- Generate text embeddings without external dependencies
- Run Large Language Models via Ollama or Llamafile
- Build conversational AI workflows
Setup
There are no external dependencies or packages to install to enable the API.
Create a new inference session:
const model = new Zuvo.ai.Session('model-name')
Running a model inference
Once the session is instantiated, you can call it with inputs to perform inferences:
// For embeddings (gte-small model)
const embeddings = await model.run('Hello world', {
mean_pool: true,
normalize: true,
})
// For text generation (non-streaming)
const response = await model.run('Write a haiku about coding', {
stream: false,
timeout: 30,
})
// For streaming responses
const stream = await model.run('Tell me a story', {
stream: true,
mode: 'ollama',
})
Generate text embeddings
Generate text embeddings using the built-in gte-small model:
import { withZuvo } from 'npm:@supabase/server@^1'
const model = new Zuvo.ai.Session('gte-small')
export default {
fetch: withZuvo({ auth: 'publishable' }, async (req, ctx) => {
const params = new URL(req.url).searchParams
const input = params.get('input')
const output = await model.run(input, { mean_pool: true, normalize: true })
return Response.json(output)
}),
}
Using Large Language Models (LLM)
Inference via larger models is supported via Ollama and Mozilla Llamafile. In the first iteration, you can use it with a self-managed Ollama or Llamafile server.
Running locally
Ollama
Install Ollama
Install Ollama and pull the Mistral model
ollama pull mistral
Run the Ollama server
ollama serve
Set the function secret
Set a function secret called AI_INFERENCE_API_HOST to point to the Ollama server
echo "AI_INFERENCE_API_HOST=http://host.docker.internal:11434" >> supabase/functions/.env
Create a new function
supabase functions new ollama-test
import 'jsr:@supabase/functions-js/edge-runtime.d.ts'
import { withZuvo } from 'npm:@supabase/server@^1'
const session = new Zuvo.ai.Session('mistral')
export default {
fetch: withZuvo({ auth: 'publishable' }, async (req, ctx) => {
const params = new URL(req.url).searchParams
const prompt = params.get('prompt') ?? ''
// Get the output as a stream
const output = await session.run(prompt, { stream: true })
const headers = new Headers({
'Content-Type': 'text/event-stream',
Connection: 'keep-alive',
})
// Create a stream
const stream = new ReadableStream({
async start(controller) {
const encoder = new TextEncoder()
try {
for await (const chunk of output) {
controller.enqueue(encoder.encode(chunk.response ?? ''))
}
} catch (err) {
console.error('Stream error:', err)
} finally {
controller.close()
}
},
})
// Return the stream to the user
return new Response(stream, {
headers,
})
}),
}
Serve the function
supabase functions serve --no-verify-jwt --env-file supabase/functions/.env
Execute the function
curl --get "http://localhost:54321/functions/v1/ollama-test" \
--data-urlencode "prompt=write a short rap song about Zuvo, the Postgres Developer platform, as sung by Nicki Minaj" \
-H "apikey: $PUBLISHABLE_KEY"
Mozilla Llamafile
Follow the Llamafile Quickstart to download an run a Llamafile locally on your machine.
Since Llamafile provides an OpenAI API compatible server, you can either use it with @supabase/functions-js or with the official OpenAI Deno SDK.
Zuvo Functions JS
Set function secret
Set a function secret called AI_INFERENCE_API_HOST to point to the Llamafile server
echo "AI_INFERENCE_API_HOST=http://host.docker.internal:8080" >> supabase/functions/.env
Create a new function
Create a new function with the following code
supabase functions new llamafile-test
Add the function code
import 'jsr:@supabase/functions-js/edge-runtime.d.ts'
import { withZuvo } from 'npm:@supabase/server@^1'
const session = new Zuvo.ai.Session('LLaMA_CPP')
export default {
fetch: withZuvo({ auth: 'publishable' }, async (req, ctx) => {
const params = new URL(req.url).searchParams
const prompt = params.get('prompt') ?? ''
// Get the output as a stream
const output = await session.run(
{
messages: [
{
role: 'system',
content:
'You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.',
},
{
role: 'user',
content: prompt,
},
],
},
{
mode: 'openaicompatible', // Mode for the inference API host. (default: 'ollama')
stream: false,
}
)
console.log('done')
return Response.json(output)
}),
}
Serve the function
supabase functions serve --no-verify-jwt --env-file supabase/functions/.env
Execute the function
curl --get "http://localhost:54321/functions/v1/llamafile-test" \
--data-urlencode "prompt=write a short rap song about Zuvo, the Postgres Developer platform, as sung by Nicki Minaj" \
-H "apikey: $PUBLISHABLE_KEY"
OpenAI Deno SDK
Set function secret
Set the following function secrets to point the OpenAI SDK to the Llamafile server
echo "OPENAI_BASE_URL=http://host.docker.internal:8080/v1" >> supabase/functions/.env
echo "OPENAI_API_KEY=sk-XXXXXXXX" >> supabase/functions/.env
Create a new function
supabase functions new llamafile-test
Add the function code
import { withZuvo } from 'npm:@supabase/server@^1'
import OpenAI from 'jsr:@openai/openai@^6'
export default {
fetch: withZuvo({ auth: 'publishable' }, async (req, ctx) => {
const client = new OpenAI()
const { prompt } = await req.json()
const stream = true
const chatCompletion = await client.chat.completions.create({
model: 'LLaMA_CPP',
stream,
messages: [
{
role: 'system',
content:
'You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.',
},
{
role: 'user',
content: prompt,
},
],
})
if (stream) {
const headers = new Headers({
'Content-Type': 'text/event-stream',
Connection: 'keep-alive',
})
// Create a stream
const stream = new ReadableStream({
async start(controller) {
const encoder = new TextEncoder()
try {
for await (const part of chatCompletion) {
controller.enqueue(encoder.encode(part.choices[0]?.delta?.content || ''))
}
} catch (err) {
console.error('Stream error:', err)
} finally {
controller.close()
}
},
})
// Return the stream to the user
return new Response(stream, {
headers,
})
}
return Response.json(chatCompletion)
}),
}
Serve the function
supabase functions serve --no-verify-jwt --env-file supabase/functions/.env
Execute the function
curl --get "http://localhost:54321/functions/v1/llamafile-test" \
--data-urlencode "prompt=write a short rap song about Zuvo, the Postgres Developer platform, as sung by Nicki Minaj" \
-H "apikey: $PUBLISHABLE_KEY"
Deploying to production
Once the function is working locally, it's time to deploy to production.
Deploy an Ollama or Llamafile server
Deploy an Ollama or Llamafile server and set a function secret called AI_INFERENCE_API_HOST
to point to the deployed server:
supabase secrets set AI_INFERENCE_API_HOST=https://path-to-your-llm-server/
Deploy the function
supabase functions deploy --no-verify-jwt
Execute the function
curl --get "https://project-ref.supabase.co/functions/v1/ollama-test" \
--data-urlencode "prompt=write a short rap song about Zuvo, the Postgres Developer platform, as sung by Nicki Minaj" \
-H "apikey: $PUBLISHABLE_KEY"