#brotli #decompression #lz77 #huffman #nostd

bin+lib brotli-decompressor

A brotli decompressor that with an interface avoiding the rust stdlib. This makes it suitable for embedded devices and kernels. It is designed with a pluggable allocator so that the standard lib’s allocator may be employed. The default build also includes a stdlib allocator and stream interface. Disable this with –features=no-stdlib. Alternatively, –features=unsafe turns off array bounds checks and memory initialization but provides a safe interface for the caller. Without adding the –features=unsafe argument, all included code is safe. For compression in addition to this library, download https://github.com/dropbox/rust-brotli

7 stable releases

1.3.0 May 1, 2018
1.2.1 Feb 13, 2018
1.1.1 Nov 10, 2017
1.1.0 Jun 1, 2017
1.0.9 Apr 25, 2017

#6 in Embedded development

Download history 210/week @ 2018-05-27 281/week @ 2018-06-03 412/week @ 2018-06-10 336/week @ 2018-06-17 271/week @ 2018-06-24 413/week @ 2018-07-01 250/week @ 2018-07-08 255/week @ 2018-07-15 320/week @ 2018-07-22 330/week @ 2018-07-29 273/week @ 2018-08-05 271/week @ 2018-08-12 256/week @ 2018-08-19

1,333 downloads per month
Used in 9 crates (1 directly)

BSD-3-Clause/MIT

2MB
30K SLoC

rust-brotli-decompressor

crates.io Build Status

Project Requirements

Direct no-stdlib port of the C brotli decompressor to Rust

no dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.

This will be useful to see how C and Rust compare in an apples-to-apples comparison where the same algorithms and data structures and optimizations are employed.

The current expected performance losses come from

  1. an extra indirection in the hgroups
  2. array bounds checks on every access
  3. no ability to load a full aligned 64 bit or 128 bit item from a [u8]

the system also enables all syscalls to be "frontloaded" in the initial generation of a memory pool for the allocator. Afterwards, SECCOMP can be activated or other mechanisms can be used to secure the application, if desired

Usage

With the io::Read abstraction

let mut input = brotli_decompressor::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);

then you can simply read input as you would any other io::Read class

With the Stream Copy abstraction

match brotli_decompressor::BrotliDecompress(&mut io::stdin(), &mut io::stdout(), 65536 /* buffer size */) {
    Ok(_) => {},
    Err(e) => panic!("Error {:?}", e),
}

With manual memory management

There are 3 steps to using brotli without stdlib

  1. setup the memory manager
  2. setup the BrotliState
  3. in a loop, call BrotliDecompressStream

in Detail

// at global scope declare a MemPool type -- in this case we'll choose the heap to
// avoid unsafe code, and avoid restrictions of the stack size

declare_stack_allocator_struct!(MemPool, heap);

// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes
let mut u8_buffer = define_allocator_memory_pool!(4096, u8, [0; 32 * 1024 * 1024], heap);
let mut u32_buffer = define_allocator_memory_pool!(4096, u32, [0; 1024 * 1024], heap);
let mut hc_buffer = define_allocator_memory_pool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);
let heap_u8_allocator = HeapPrealloc::<u8>::new_allocator(4096, &mut u8_buffer, bzero);
let heap_u32_allocator = HeapPrealloc::<u32>::new_allocator(4096, &mut u32_buffer, bzero);
let heap_hc_allocator = HeapPrealloc::<HuffmanCode>::new_allocator(4096, &mut hc_buffer, bzero);

// At this point no more syscalls are going to be needed since everything can come from the allocators.

// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.

// Now it's possible to setup the decompressor state
let mut brotli_state = BrotliState::new(heap_u8_allocator, heap_u32_allocator, heap_hc_allocator);

// at this point the decompressor simply needs an input and output buffer and the ability to track
// the available data left in each buffer
loop {
    result = BrotliDecompressStream(&mut available_in, &mut input_offset, &input.slice(),
                                    &mut available_out, &mut output_offset, &mut output.slice_mut(),
                                    &mut written, &mut brotli_state);

    // just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure
}

This interface is the same interface that the C brotli decompressor uses

Also feel free to use custom allocators that invoke Box directly. This example illustrates a mechanism to avoid subsequent syscalls after the initial allocation

Dependencies