Terran's face recognition meets Streamlit Components

We can’t believe Streamlit is barely one year old now. Today, we can’t think of any better tool for creating a fast and interactive dashboard, demos, and visualizations. On top of that, they recently launched a new awesome feature called Streamlit Components. This new feature enables developers to create custom visualizations using your favorite front-end framework, namely React, Vue, or just plain HTML.

Especially, Streamlit is really convenient when starting new machine learning projects, making it super easy and fast to go from nothing to a proof of concept of a ML solution. Because of this, we use Streamlit pretty frequently: for project prototyping, dynamic reports, and for building cool demos 💅

Particularly, our open-source project Terran was not an exception to this. We've built a face-recognition timeline demo to demonstrate how easy it is to use Terran on YouTube videos, and additionally, how straightforward it is to create amazing visualizations using the new Streamlit Components! 🚀

In this blog post, we'll share how the process of building this component was, and hopefully, inspire you to build your own component or demo using Terran!

Our demo consists of the following:

  1. You take any video from YouTube that has people in it, for example, this Big Bang Theory video.
  2. Then, we use Terran to detect all the faces that appear in the video, and when they actually appear on the screen.
  3. Finally, we create a timeline visualization that shows in green when someone is present in the video.
  4. Optionally, we can make it dynamic so that when the user clicks on a specific time of the timeline, a video player is rendered and it'll begin playing starting from the second the user clicked.

The final result looks like this

What is Terran

Terran face-recognition timeline

Terran is our open-source human perception library. We focus on providing open, auditable, and performant alternatives to closed approaches. Solutions that can run on your computer or device and that will adapt to the environment that it's in. But most importantly, code that you can test (without piling-up costs), understand and modify to your needs.

You can check out the Terran's introduction blog post to know more about this project! Also, you can support our project and give it a try and why not a star in Github 💫

Using Terran's face-recognition modules

The demo consists of automatically detecting and tracking faces on YouTube videos. With this information, we can generate timeline charts showing specifically when each person appears on the screen or not. The idea then is to use Terran's face-recognition library to generate all the data needed to create a timeline visualization, and then we can create a new Streamlit Component to create a fancy and interactive timeline chart!

We can split the processes into the following steps:

  1. Generate the timeline data from a YouTube video using Terran
  2. Bootstrapping your Streamlit Component and declare the input/output data bindings
  3. And creating a custom and interactive visualization tool using React and D3

Let's dive in!

Generate the timeline data from a YouTube video using Terran

This is pretty simple. Using Terran's open_video we will iterate through the video's frames, and using the functions face_detection and extract_features we'll detect for each frame where the faces are and a vector representation -features- of each face.

Terran Streamlit timeline generator diagram

Then, we can start comparing the extracted features to know if we're talking about the same person or not, we can do a simple cosine_distance threshold to know if that's the case.

Finally, we create a data structure that stores for each detected person all the frame indexes in which they appear, and that's it!

Terran Streamlit timeline generator data structure

We created a function generate_timeline that does this exact same logic, plus some other minor features. When being used, it looks as simple as this:

from streamlit_terran_timeline import generate_timeline

timeline = generate_timeline("https://youtu.be/mbdOqxdiSZc")

Bootstrap your Streamlit Component and declare the input/output data bindings

This step might work differently for you depending on your frontend skills. In particular, we're more into working with React, so this Streamlit Component template was great for setting up the demo.

We think the way of integrating new components to Streamlit is very smart: just assume that you can send all the data you need to build the component as long as this data can be JSON serialized. Additionally, our template will declare our component in a developing stage, so that it is easier for us to adjust our frontend with hot reloading as we make changes. You can go to the official Streamlit documentation for more details.

To declare the bidings of our component, we create our component function called terran_timeline like this:

def terran_timeline(timeline, key=None):
    component_value = _component_func(timeline=timeline, key=key, default=0)

    return component_value

...where the _component_func is the one that declares our Streamlit Component and we are saying that we want to send a data structured by the name of timeline to this component. The additional key argument makes it possible to have multiple components of the same kind by specifying different keys, and the default argument stands for the default return value of the component, which in this case means starting second value for the video.

Create a custom and interactive visualization tool using React and D3

Then we have to move into the React world and start building our component. The good news is that we can retrieve the timeline object inside our component super easily!

public render = (): ReactNode => {
    const timeline = this.props.args["timeline"]
    ...
}

Also, we can make our component "return" a value using the Streamlit.setComponentValue function they provide in the template, for example, making that when the user clicks on the timeline widget

//
// If the user clicks on the timeline, we set the component value to the
// time in seconds where the user clicked
//
Streamlit.setComponentValue(seekTimeSecond);

The rest is mostly plain React and D3 stuff we think it's not worth focusing on this blog, but if you are interested you can check how it was implemented in our streamlit-terran-timeline repo.

Here's an example of what it looks like to use this new Streamlit Component:

import streamlit as st
from streamlit_terran_timeline import generate_timeline, terran_timeline

# Generate the timeline information
video_url = "https://youtu.be/mbdOqxdiSZc"
timeline = generate_timeline(video_url)

#
# Display the timeline. If the users click, you'll get the exact second of
# the part of the timeline video. By default, it returns 0.
#
start_time = terran_timeline(timeline)

#
# Display the video. Optionally, the user might click the timeline and start the video
# from a different time
#
st.video(video_url, start_time=start_time)

In summary, the Streamlit Components feature is a game-changer. It was never more straightforward for us to create dynamic, reliable, and nice-looking pilot projects or demos from scratch. It's also really amazing that the Streamlit community is building new components and we're really thrilled to contribute with ours! If you are interested, you can see a full list of published and upcoming components in this community post .

This is the first of many upcoming Terran demos, so stay in touch to know what other cool stuff we're building! You can follow us on Twitter and LinkedIn, and you can also subscribe to our newsletter!

Do you like our content?

Sign up for our newsletter to stay up to date.