User guide

Installation

CAVlib can be installed through pip.

$ pip install cavlib

CAVlib supports Python 3.7, 3.8, and 3.9.

Note: Note: Due to limited availability of tflite-runtime wheels on PyPI, you might need to add an --extra-index-url argument to pip, like this:

$ pip install --extra-index-url https://google-coral.github.io/py-repo/ cavlib

How to use

Here’s a program that calculates the CAV score for a few images on your computer:

from pathlib import Path
from cavlib import CAV

images_dir = Path('examples/images')
my_cav = CAV.load('examples/roundness.cav')

for image in images_dir.iterdir():
    print(image.name, my_cav.score(image))

Here’s a program that searches a folder of images for the top 3 according a CAV:

from pathlib import Path
from cavlib import CAV

images_dir = Path('examples/images')
image_files = list(images_dir.iterdir())

my_cav = CAV.load('examples/roundness.cav')
sorted_images = my_cav.sort(image_files, reverse=True)
print('top 3 images:', sorted_images[0:3])

Or here’s a program that takes a CAV from CAVstudio and runs it on a live video feed from your webcam:

import cv2
from cavlib import CAV

webcam = cv2.VideoCapture(0)
my_cav = CAV.load('examples/roundness.cav')

while True:
    _, frame = webcam.read()
    print(my_cav.score(frame))
    cv2.imshow('webcam', frame)
    cv2.waitKey(1)

These examples are also available in a IPython notebook.

How does it work?

CAVs are a direction in ‘embedding space’ - a vector. It works like this - if you take a layer of a neural network ML model and extract all the values in the neurons at that point (the ‘activations’), you can treat these values as a point in embedding space. Every different input image is mapped to a different location, with similar images close together, and different images far apart.

---

'Embedding space' is a high-dimensional space - perhaps thousands of dimensions. Embedding projector is a visual way to explore embedding spaces, by projecting each point to 3 dimensions using clever mathematical techniques like UMAP and T-SNE.

---

To create a CAV, you need positive and negative training images. Positive images show the concept you’re trying to express, and negative ones do not (or even better, show the opposite).

For example, if you were trying to create a CAV for the concept ‘Roundness’, your positive training images would be pictures of round things, and the negative images would be not-round things, or just random images.

CAVs are created by finding a direction in this high dimensional space that differentiates the positive from the negative training images. In CAVlib, you can do this with the cavlib.train_cav function, or you can download a CAV from CAVstudio.

Once formed, the CAV can be compared with any new image, by looking at the angle between the CAV and the new image’s activations, a technique known as cosine similarity. We call the resulting number the ‘CAV score’ - in our example, a measure of ‘Roundness’ - as a number from -1 to 1.