Zuvo Docs

Guide

Roboflow

Learn how to integrate Zuvo with Roboflow, a tool for running fine-tuned and foundation vision models.

In this guide, we will walk through two examples of using Roboflow Inference to run fine-tuned and foundation models. We will run inference and save predictions using an object detection model and CLIP.

Project setup

To create a new Postgres database, start a new Project in Zuvo:

  1. Create a new project in the Zuvo dashboard.
  2. Enter your project details. Remember to store your password somewhere safe.

Your database will be available in less than a minute.

Finding your credentials:

You can find your project credentials on the dashboard:

Save computer vision predictions

Once you have a trained vision model, you need to create business logic for your application. In many cases, you want to save inference results to a file.

The steps below show you how to run a vision model locally and save predictions to Zuvo.

Preparation: Set up a model

Before you begin, you will need an object detection model trained on your data.

You can train a model on Roboflow, leveraging end-to-end tools from data management and annotation to deployment, or upload custom model weights for deployment.

All models have an infinitely scalable API through which you can query your model, and can be run locally.

For this guide, we will use a demo rock, paper, scissors model.

Step 1: Install and start Roboflow Inference

You will deploy our model locally using Roboflow Inference, a computer vision inference server.

To install and start Roboflow Inference, first install Docker on your machine.

Then, run:

pip install inference inference-cli inference-sdk && inference server start

An inference server will be available at http://localhost:9001.

Step 2: Run inference on an image

You can run inference on images and videos.

Create a new Python file and add the following code:

from inference_sdk import InferenceHTTPClient

image = "example.jpg"
MODEL_ID = "rock-paper-scissors-sxsw/11"

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key="ROBOFLOW_API_KEY"
)
with client.use_model(MODEL_ID):
    predictions = client.infer(image)

print(predictions)

Above, replace:

  1. The image URL with the name of the image on which you want to run inference.
  2. ROBOFLOW_API_KEY with your Roboflow API key. Learn how to retrieve your Roboflow API key.
  3. MODEL_ID with your Roboflow model ID. Learn how to retrieve your model ID.

When you run the code above, a list of predictions will be printed to the console:

{'time': 0.05402109300121083, 'image': {'width': 640, 'height': 480}, 'predictions': [{'x': 312.5, 'y': 392.0, 'width': 255.0, 'height': 110.0, 'confidence': 0.8620790839195251, 'class': 'Paper', 'class_id': 0}]}

Step 3: Save results in Zuvo

To save results in Zuvo, add the following code to your script:

import os
from supabase import create_client, Client

url: str = os.environ.get("SUPABASE_URL")
key: str = os.environ.get("SUPABASE_KEY")
supabase: Client = create_client(url, key)

result = supabase.table('predictions') \
    .insert({"filename": image, "predictions": predictions}) \
    .execute()

You can then query your predictions using the following code:

result = supabase.table('predictions') \
    .select("predictions") \
    .filter("filename", "eq", image) \
    .execute()

print(result)

Here is an example result:

data=[{'predictions': {'time': 0.08492901099998562, 'image': {'width': 640, 'height': 480}, 'predictions': [{'x': 312.5, 'y': 392.0, 'width': 255.0, 'height': 110.0, 'confidence': 0.8620790839195251, 'class': 'Paper', 'class_id': 0}]}}, {'predictions': {'time': 0.08818970100037404, 'image': {'width': 640, 'height': 480}, 'predictions': [{'x': 312.5, 'y': 392.0, 'width': 255.0, 'height': 110.0, 'confidence': 0.8620790839195251, 'class': 'Paper', 'class_id': 0}]}}] count=None

Calculate and save CLIP embeddings

You can use the Zuvo vector database functionality to store and query CLIP embeddings.

Roboflow Inference provides an HTTP interface through which you can calculate image and text embeddings using CLIP.

Step 1: Install and start Roboflow Inference

See Step #1: Install and Start Roboflow Inference above to install and start Roboflow Inference.

Step 2: Run CLIP on an image

Create a new Python file and add the following code:

import cv2
import supervision as sv
import requests
import base64
import os

IMAGE_DIR = "images/train/images/"
API_KEY = ""
SERVER_URL = "http://localhost:9001"

results = []

for i, image in enumerate(os.listdir(IMAGE_DIR)):
    print(f"Processing image {image}")
    infer_clip_payload = {
        "image": {
            "type": "base64",
            "value": base64.b64encode(open(IMAGE_DIR + image, "rb").read()).decode("utf-8"),
        },
    }

    res = requests.post(
        f"{SERVER_URL}/clip/embed_image?api_key={API_KEY}",
        json=infer_clip_payload,
    )

    embeddings = res.json()['embeddings']

    results.append({
        "filename": image,
        "embeddings": embeddings
    })

This code will calculate CLIP embeddings for each image in the directory and print the results to the console.

Above, replace:

  1. IMAGE_DIR with the directory containing the images on which you want to run inference.
  2. ROBOFLOW_API_KEY with your Roboflow API key. Learn how to retrieve your Roboflow API key.

You can also calculate CLIP embeddings in the cloud by setting SERVER_URL to https://infer.roboflow.com.

Step 3: Save embeddings in Zuvo

You can store your image embeddings in Zuvo using the Zuvo vecs Python package:

First, install vecs:

pip install vecs

Next, add the following code to your script to create an index:


import vecs

DB_CONNECTION = "postgresql://postgres:[password]@[host]:[port]/[database]"

vx = vecs.create_client(DB_CONNECTION)

# create a collection of vectors with 3 dimensions
images = vx.get_or_create_collection(name="image_vectors", dimension=512)

for result in results:
    image = result["filename"]
    embeddings = result["embeddings"][0]

    # insert a vector into the collection
    images.upsert(
        records=[
            (
                image,
                embeddings,
                {} # metadata
            )
        ]
    )

images.create_index()

Replace DB_CONNECTION with the authentication information for your database. You can retrieve this from the Zuvo dashboard in Project Settings > Database Settings.

You can then query your embeddings using the following code:

infer_clip_payload = {
    "text": "cat",
}

res = requests.post(
    f"{SERVER_URL}/clip/embed_text?api_key={API_KEY}",
    json=infer_clip_payload,
)

embeddings = res.json()['embeddings']

result = images.query(
    data=embeddings[0],
    limit=1
)

print(result[0])

Resources