Identifying highlights in audio and video files

The Key Phrases model identifies significant words and phrases in your transcript and lets you to extract the most important concepts or highlights from your audio or video file. For example, if you’re a call center, you can analyze highlights from recorded phone calls. In this step-by-step guide, you’ll learn how to apply the model. You’ll send the auto_highlights parameter in your request, and then use the auto_highlights_result property in the response.

Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can sign up for a free account and get your API key from your dashboard. The complete source code for this guide can be viewed here. Here’s an audio sample for this guide:

https://assembly.ai/wildfires.mp3

Step-by-step instructions

Python (requests)
Python SDK
JavaScript

Create a new file and request.

import requests
import time

Install the SDK.

pip install -U assemblyai

Create a new file and request.

import fs from "fs-extra";

Python
Python SDK
JavaScript

Set up the API endpoint and headers. The headers should include your API key.

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

Import the assemblyai package and set the API key.

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

Set up the API endpoint and headers. The headers should include your API key.

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

Python
Python SDK
JavaScript

Upload your local file to the AssemblyAI API.

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

Create a TranscriptionConfig with auto_highlights set to True.

# highlight-next-line
config = aai.TranscriptionConfig(
    speech_models=["universal-3-pro", "universal-2"],
    language_detection=True,
    auto_highlights=True
)

Upload your local file to the AssemblyAI API.

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

Python
Python SDK
JavaScript

Use the upload_url returned by the AssemblyAI API to create a JSON payload containing the audio_url parameter and the auto_highlights parameter set to True.

data = {
    "audio_url": upload_url,
    "speech_models": ["universal-3-pro", "universal-2"],
    "language_detection": True,
    "auto_highlights": True
}

Create a Transcriber object and pass in the configuration.

transcriber = aai.Transcriber(config=config)

Use the upload_url returned by the AssemblyAI API to create a JSON payload containing the audio_url parameter and the auto_highlights parameter set to True.

const data = {
  audio_url: uploadUrl,
  speech_models: ["universal-3-pro", "universal-2"],
  language_detection: true,
  auto_highlights: true,
};

Python
Python SDK
JavaScript

Make a POST request to the AssemblyAI API endpoint with the payload and headers.

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

Pass the URL or file path to Transcriber.transcribe(). You can access the transcript from the returned Transcript object.

FILE_URL = "https://assembly.ai/wildfires.mp3"

transcript = transcriber.transcribe(FILE_URL)

Make a POST request to the AssemblyAI API endpoint with the payload and headers.

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

Python
Python SDK
JavaScript

After making the request, you’ll receive an ID for the transcription. Use it to poll the API every few seconds to check the status of the transcript job. Once the status is completed, you can retrieve the transcript from the API response, as well as the auto highlight results.

transcription_id = response.json()['id']
polling_endpoint = f"https://api.assemblyai.com/v2/transcript/{transcription_id}"

while True:
    transcription_result = requests.get(polling_endpoint, headers=headers).json()

    if transcription_result['status'] == 'completed':
        auto_highlights_result = transcription_result['auto_highlights_result']
        for highlight in auto_highlights_result['results']:
            print(f"Highlight: {highlight['text']}, Count: {highlight['count']}, Rank: {highlight['rank']}, Timestamps: {highlight['timestamps']}")
        break

    elif transcription_result['status'] == 'error':
        raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

    else:
        time.sleep(3)

You can access automatic highlights from transcript.auto_highlights.results.

for result in transcript.auto_highlights.results:
  print(f"Highlight: {result.text}, Count: {result.count}, Rank: {result.rank}, Timestamps: {result.timestamps}")

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  let res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    const autoHighlightsResult = transcriptionResult.auto_highlights_result;
    for (const highlight of autoHighlightsResult.results) {
      const timestamps = highlight.timestamps
        .map((timestamp) => `${timestamp.start}ms-${timestamp.end}ms`)
        .join(", ");
      console.log(
        `Highlight: ${highlight.text}, Count: ${highlight.count}, Rank: ${highlight.rank}, Timestamps: ${timestamps}`
      );
    }
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

Understanding the response

The auto_highlights_result key in the response contains a list of all the highlights found in the transcription text. For each entry, the results include the text of the phrase or word detected (text), how many times it occurred in the text (count), its relevancy score (rank), and a list of all the timestamps (timestamps), in milliseconds, in the audio where the phrase or word is spoken.

{
      auto_highlights_result: {
        status: "success",
        results: [
          {
            count: 2,
            rank: 0.04,
            text: "months",
            timestamps: [
              {
                start: 24688,
                end: 24922,
              },
              {
                start: 80408,
                end: 80594,
              },
            ],
          },
          {
            count: 1,
            rank: 0.04,
            text: "first fee",
            timestamps: [
              {
                start: 78948,
                end: 79766,
              },
            ],
          },
        ],
      },
      auto_highlights: true,
    
}

For more information about the API response, see API/Model reference.

Conclusion

Automatically highlighting relevant phrases in calls is a great way to focus on important information at a glance. In general, adding AI to Conversation Intelligence tools can augment them by generating actionable summaries to speed up call review, generating insights, monitoring for concerns, increasing engagement, and more. Our AI summarization model has several customizable parameters that you can experiment with for other types of recordings. To learn more about how to use AI summarization for call coaching, see AssemblyAI blog.

Documentation Index

​Get started

​Step-by-step instructions

​Understanding the response

​Conclusion

Get started

Step-by-step instructions

Understanding the response

Conclusion