Rust Packages for Audio and Video Processing: A Complete Guide

Are you tired of using slow and unreliable libraries for audio and video processing in your Rust projects? Look no further! Rust has a growing ecosystem of packages for audio and video processing that are fast, reliable, and easy to use. In this guide, we will explore some of the best Rust packages for audio and video processing, and how you can use them in your projects.

Audio Processing

rodio

Rodio is a Rust audio playback library that provides a simple and easy-to-use API for playing audio files. It supports a wide range of audio formats, including MP3, WAV, FLAC, and more. With rodio, you can easily play audio files in the background of your Rust applications, without worrying about the details of audio playback.

use rodio::Sink;
use std::fs::File;

fn main() {
    let file = File::open("path/to/audio/file.mp3").unwrap();
    let sink = Sink::new(&rodio::default_output_device().unwrap());
    sink.append(rodio::Decoder::new(file).unwrap());
    sink.sleep_until_end();
}

In the example above, we open an audio file, create a new sink, and append the audio file to the sink. We then call sleep_until_end() to block the main thread until the audio file has finished playing. It's that simple!

hound

Hound is a Rust library for reading and writing audio files in various formats, including WAV, AIFF, and FLAC. It provides a simple and easy-to-use API for reading and writing audio files, making it a great choice for audio processing in Rust.

use hound::{WavReader, WavWriter};

fn main() {
    let mut reader = WavReader::open("path/to/audio/file.wav").unwrap();
    let spec = reader.spec();
    let mut writer = WavWriter::create("path/to/new/audio/file.wav", spec).unwrap();
    for sample in reader.samples::<i16>() {
        let sample = sample.unwrap();
        writer.write_sample(sample).unwrap();
    }
}

In the example above, we open an audio file using WavReader, get the audio file's specification, create a new WavWriter with the same specification, and write each sample to the new audio file. It's that easy!

cpal

Cpal is a low-level Rust library for audio playback and recording. It provides a simple and easy-to-use API for interacting with audio devices, making it a great choice for audio processing in Rust.

use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};

fn main() {
    let host = cpal::default_host();
    let device = host.default_output_device().unwrap();
    let config = device.default_output_config().unwrap();
    let sample_rate = config.sample_rate().0;
    let channels = config.channels();
    let mut buffer = vec![0.0; 4096];
    let stream = device.build_output_stream(
        &config.into(),
        move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
            for sample in data.iter_mut() {
                *sample = buffer.remove(0);
            }
        },
        move |err| {
            eprintln!("an error occurred on the output audio stream: {}", err);
        },
    ).unwrap();
    stream.play().unwrap();
}

In the example above, we get the default audio output device, get its default output configuration, create a buffer of audio samples, and build an output stream with a callback function that fills the audio buffer with audio samples. We then play the audio stream. It's that simple!

Video Processing

ffmpeg-next

FFmpeg-next is a Rust library for interacting with the FFmpeg multimedia framework. It provides a simple and easy-to-use API for decoding and encoding video files, making it a great choice for video processing in Rust.

use ffmpeg_next::{codec, format, media, software::scaling::context::Context};
use std::{fs::File, io::Write};

fn main() {
    let input_path = "path/to/video/file.mp4";
    let output_path = "path/to/new/video/file.mp4";
    let mut input_format_context = format::input(&input_path).unwrap();
    let input_stream = input_format_context.streams().best(media::Type::Video).unwrap();
    let input_codec = codec::decoder::find(input_stream.codec_parameters().codec_id()).unwrap();
    let mut input_codec_context = codec::decoder::Context::new(&input_codec);
    input_codec_context.apply_parameters(input_stream.codec_parameters()).unwrap();
    let mut output_format_context = format::output(&output_path).unwrap();
    let output_codec = codec::encoder::find(codec::Id::H264).unwrap();
    let mut output_stream = output_format_context.add_stream(output_codec).unwrap();
    let mut output_codec_context = codec::encoder::Context::new(&output_codec);
    output_codec_context.set_width(input_codec_context.width());
    output_codec_context.set_height(input_codec_context.height());
    output_codec_context.set_time_base(input_stream.time_base());
    output_codec_context.set_format(format::Pixel::YUV420P);
    output_codec_context.open(None).unwrap();
    output_stream.set_time_base(input_stream.time_base());
    output_stream.set_codecpar(output_codec_context.parameters().clone());
    output_format_context.write_header().unwrap();
    let mut packet = ffmpeg_next::util::packet::Packet::empty();
    let mut frame = ffmpeg_next::util::frame::Video::empty();
    let mut scaler = Context::get(
        input_codec_context.pix_fmt(),
        input_codec_context.width(),
        input_codec_context.height(),
        output_codec_context.pix_fmt(),
        output_codec_context.width(),
        output_codec_context.height(),
        ffmpeg_next::software::scaling::flag::Flags::BILINEAR,
    ).unwrap();
    let mut output_frame = ffmpeg_next::util::frame::Video::empty();
    output_frame.set_width(output_codec_context.width());
    output_frame.set_height(output_codec_context.height());
    output_frame.set_format(output_codec_context.pix_fmt());
    output_frame.set_pts(0);
    let mut output_packet = ffmpeg_next::util::packet::Packet::empty();
    while input_format_context.read_packet(&mut packet).unwrap() {
        if packet.stream_index() == input_stream.index() {
            input_codec_context.send_packet(&packet).unwrap();
            while let Ok(decoded) = input_codec_context.receive_frame(&mut frame) {
                scaler.run(&frame, &mut output_frame).unwrap();
                output_codec_context.send_frame(&output_frame).unwrap();
                while let Ok(encoded) = output_codec_context.receive_packet(&mut output_packet) {
                    output_format_context.write_packet(&mut output_packet).unwrap();
                }
            }
        }
    }
    input_codec_context.send_eof().unwrap();
    while let Ok(decoded) = input_codec_context.receive_frame(&mut frame) {
        scaler.run(&frame, &mut output_frame).unwrap();
        output_codec_context.send_frame(&output_frame).unwrap();
        while let Ok(encoded) = output_codec_context.receive_packet(&mut output_packet) {
            output_format_context.write_packet(&mut output_packet).unwrap();
        }
    }
    output_codec_context.send_eof().unwrap();
    while let Ok(encoded) = output_codec_context.receive_packet(&mut output_packet) {
        output_format_context.write_packet(&mut output_packet).unwrap();
    }
    output_format_context.write_trailer().unwrap();
}

In the example above, we open an input video file, get its video stream, get its video codec, create a new video decoder context, open an output video file, get its video codec, create a new video encoder context, and write the video header. We then read packets from the input video file, decode frames, scale frames, encode frames, and write packets to the output video file. It's that complex!

gstreamer

GStreamer is a Rust library for building multimedia applications. It provides a simple and easy-to-use API for creating pipelines that process audio and video data, making it a great choice for video processing in Rust.

use gstreamer::{Element, ElementFactory, Pipeline, State};

fn main() {
    let pipeline = Pipeline::new(None);
    let source = ElementFactory::make("filesrc", None).unwrap();
    let decoder = ElementFactory::make("decodebin", None).unwrap();
    let videoconvert = ElementFactory::make("videoconvert", None).unwrap();
    let videoscale = ElementFactory::make("videoscale", None).unwrap();
    let videorate = ElementFactory::make("videorate", None).unwrap();
    let encoder = ElementFactory::make("x264enc", None).unwrap();
    let muxer = ElementFactory::make("mp4mux", None).unwrap();
    let sink = ElementFactory::make("filesink", None).unwrap();
    source.set_property("location", &"path/to/video/file.mp4").unwrap();
    encoder.set_property_from_str("tune", "zerolatency").unwrap();
    sink.set_property("location", &"path/to/new/video/file.mp4").unwrap();
    pipeline.add_many(&[&source, &decoder, &videoconvert, &videoscale, &videorate, &encoder, &muxer, &sink]).unwrap();
    Element::link_many(&[&decoder, &videoconvert]).unwrap();
    Element::link_many(&[&videoconvert, &videoscale]).unwrap();
    Element::link_many(&[&videoscale, &videorate]).unwrap();
    Element::link_many(&[&videorate, &encoder]).unwrap();
    Element::link_many(&[&encoder, &muxer]).unwrap();
    Element::link_many(&[&muxer, &sink]).unwrap();
    pipeline.set_state(State::Playing).unwrap();
    let bus = pipeline.bus().unwrap();
    for msg in bus.iter_timed(gstreamer::CLOCK_TIME_NONE) {
        match msg.view() {
            gstreamer::MessageView::Eos(..) => break,
            gstreamer::MessageView::Error(err) => {
                eprintln!("Error from {:?}: {}", err.get_src().map(|s| s.get_path_string()), err.get_error());
                break;
            }
            _ => (),
        }
    }
    pipeline.set_state(State::Null).unwrap();
}

In the example above, we create a pipeline that reads a video file, decodes the video, scales the video, encodes the video, and writes the video to a new file. It's that easy!

Conclusion

In this guide, we explored some of the best Rust packages for audio and video processing, and how you can use them in your projects. Whether you need to play audio files, read and write audio files, interact with audio devices, decode and encode video files, or build multimedia pipelines, Rust has you covered. So why not give Rust a try for your next audio or video processing project? You won't be disappointed!

Additional Resources