Identifying hate speech in audio or video files

Our Content Moderation model can help you ensure that your content is safe and appropriate for all audiences. The model pinpoints sensitive discussions in spoken data and provides information on the severity to which they occurred. In this guide, we’ll learn how to use the Content Moderation model, and look at an example response to understand its structure.

Get started

Before we begin, make sure you have an AssemblyAI account and an API key. You can sign up for a free account and get your API key from your dashboard. The complete source code for this guide can be viewed here. Here is an audio example for this guide:

https://assembly.ai/wildfires.mp3

Step-by-step instructions

Python
Python SDK
JavaScript

Create a new file and request.

import requests
import time

Install the SDK.

pip install -U assemblyai

Create a new file and request.

import fs from "fs-extra";

Python
Python SDK
JavaScript

Set up the API endpoint and headers. The headers should include your API key.

base_url = "https://api.assemblyai.com"

headers = {
    "authorization": "<YOUR_API_KEY>"
}

Import AssemblyAI package and set your API key.

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

Set up the API endpoint and headers. The headers should include your API key.

const baseUrl = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

Python
Python SDK
JavaScript

Upload your local file to the AssemblyAI API.

with open("./my-audio.mp3", "rb") as f:
  response = requests.post(base_url + "/v2/upload",
                          headers=headers,
                          data=f)

upload_url = response.json()["upload_url"]

Create a TranscriptionConfig with content_safety set to True.

# highlight-next-line
config = aai.TranscriptionConfig(content_safety=True)

Upload your local file to the AssemblyAI API.

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${baseUrl}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const urlResponse = await res.json();
const uploadUrl = urlResponse.upload_url;

Python
Python SDK
JavaScript

Use the upload_url returned by the AssemblyAI API to create a JSON payload containing the audio_url parameter and the content_safety parameter set to True.

data = {
    "audio_url": upload_url,
    "content_safety": True
}

Create a Transcriber object and pass in the configuration.

transcriber = aai.Transcriber(config=config)

Use the upload_url returned by the AssemblyAI API to create a JSON payload containing the audio_url parameter and the content_safety parameter set to True.

const data = {
  audio_url: uploadUrl,
  content_safety: true,
};

Python
Python SDK
JavaScript

Make a POST request to the AssemblyAI API endpoint with the payload and headers.

url = base_url + "/v2/transcript"
response = requests.post(url, json=data, headers=headers)

Use the Transcriber object’s transcribe method and pass in the audio file’s path as a parameter. The transcribe method saves the results of the transcription to the Transcriber object’s transcript attribute.

FILE_URL = "https://assembly.ai/wildfires.mp3"

transcript = transcriber.transcribe(FILE_URL)

Make a POST request to the AssemblyAI API endpoint with the payload and headers.

const url = `${baseUrl}/v2/transcript`;
let res = await fetch(url, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

Python
Python SDK
JavaScript

After making the request, you’ll receive an ID for the transcription. Use it to poll the API every few seconds to check the status of the transcript job. Once the status is completed, you can retrieve the transcript from the API response, using the content_safety_labels key to view the results.

transcript_id = response.json()['id']
polling_endpoint = f"https://api.assemblyai.com/v2/transcript/{transcript_id}"

while True:
   transcription_result = requests.get(polling_endpoint, headers=headers).json()

   if transcription_result['status'] == 'completed':
       # Uncomment the next line to print everything
       # print(transcription_result['content_safety_labels'])

       content_safety_labels = transcription_result['content_safety_labels']['results']
       for results in content_safety_labels:
           labels = results['labels']
           for label in labels:
             # The severity score measures how severe the flagged content is on a scale of 0-1, with 1 being the most severe.
             if label['label'] == 'hate_speech' and label['severity'] >= 0.5:
                 print("Hate speech detected with severity score:", label['severity'])
                 # Do something with this information, such as flagging the transcription for review
       break

   elif transcription_result['status'] == 'error':
       raise RuntimeError(f"Transcription failed: {transcription_result['error']}")

   else:
       time.sleep(3)

You can access the content moderation results through the Transcriber object’s content_safety attribute.

# Get the parts of the transcript which were flagged as sensitive
for result in transcript.content_safety.results:
  print(result.text)  # sensitive text snippet
  print(result.timestamp.start)
  print(result.timestamp.end)

  for label in result.labels:
    print(label.label)  # content safety category
    print(label.confidence) # model's confidence that the text is in this category
    print(label.severity) # severity of the text in relation to the category

# Get the confidence of the most common labels in relation to the entire audio file
for label, confidence in transcript.content_safety.summary.items():
  print(f"{confidence * 100}% confident that the audio contains {label}")

# Get the overall severity of the most common labels in relation to the entire audio file
for label, severity_confidence in transcript.content_safety.severity_score_summary.items():
  print(f"{severity_confidence.low * 100}% confident that the audio contains low-severity {label}")
  print(f"{severity_confidence.medium * 100}% confident that the audio contains mid-severity {label}")
  print(f"{severity_confidence.high * 100}% confident that the audio contains high-severity {label}")

const transcriptId = response.id;
const pollingEndpoint = `${baseUrl}/v2/transcript/${transcriptId}`;

while (true) {
  let res = await fetch(pollingEndpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcriptionResult = await res.json();

  if (transcriptionResult.status === "completed") {
    // Uncomment the next line to print everything
    // console.log(transcriptionResult.content_safety_labels)

    const contentSafetyLabels =
      transcriptionResult.content_safety_labels.results;
    contentSafetyLabels.forEach((label) => {
      const labels = label.labels;
      labels.forEach((label) => {
        // The severity score measures how severe the flagged content is on a scale of 0-1, with 1 being the most severe.
        if (label.label === "hate_speech" && label.severity >= 0.5) {
          console.log(
            `Hate speech detected with severity score: ${label.severity}`
          );
          // Do something with this information, such as flagging the transcription for review
        }
      });
    });
    break;
  } else if (transcriptionResult.status === "error") {
    throw new Error(`Transcription failed: ${transcriptionResult.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

Understanding the response

In the JSON response, there’ll be an additional key called content_safety_labels that contains information about any sensitive content detected. The full text is contained in the text key, and each problematic utterance has its own labels and timestamp. The entire audio is assigned a summary and a severity_score_summary for each category of unsafe content. Each label is returned with a confidence score and a severity score.

{
      content_safety_labels: {
        status: "success",
        results: [
          {
            text: "You know, the problem with...",
            labels: [
              {
                label: "sensitive_social_issues",
                confidence: 0.9962000250816345,
                severity: null,
              },
              {
                label: "hate_speech",
                confidence: 0.9268788695335388,
                severity: 0.5201690793037415,
              },
            ],
            timestamp: {
              start: 650,
              end: 4970,
            },
          },
        ],
        summary: {
          sensitive_social_issues: 0.9999820200945412,
          hate_speech: 0.9999516283032464,
        },
        severity_score_summary: {
          hate_speech: {
            low: 0,
            medium: 1,
            high: 0,
          },
        },
      },
    
}

For more information, see Content Moderation model documentation and API reference.

Conclusion

The AssemblyAI API supports many different content safety labels. Identifying hate speech is only a single, important use case for automated content moderation, and you can learn about others on the AssemblyAI blog.

Documentation Index

​Get started

​Step-by-step instructions

​Understanding the response

​Conclusion

Get started

Step-by-step instructions

Understanding the response

Conclusion