23 releases (8 stable)
|3.0.3||Aug 2, 2018|
|3.0.2||Jul 29, 2018|
|3.0.0-0.2.0||Jan 25, 2018|
|3.0.0-0.1.0||Oct 1, 2017|
|0.3.0||Jul 1, 2016|
#33 in Compression
281 downloads per month
See wiki for current project status.
rdedup is a data deduplication engine and a backup software.
rdedup is generally similar to existing software like
rdedup is written in Rust and provides both command line tool
and library API (
- simple but solid cryptography:
- libsodium based
- public-key encryption mode (the only tool like that I'm aware of,
and primary reason
- flat-file synchronization friendly (Dropbox/syncthing, rsync, rclone)
- immutable data-conflict-free data store
- cloud backends are WIP
- incremental, scalable garbage collection
- variety of supported algorithms:
- chunking: fastcdc, gear, bup
- hashing: blake2b, sha256
- compression: zstd, deflate, xz2, bzip2, none
- encryption: curve25519, none
- very easy to add new ones
rdedup init --helpoutput for up-to-date list
- extreme performance and parallelism - see
Rust fearless concurrency in
- reliability focus (eg.
renameto avoid data corruption even in case of a hardware crash)
- built-in time/performance profiler
It's written in Rust. It's a modern language, that is actually really nice to use. Rust makes it easy to have a very robust and fast software.
The author is a nice person, welcomes contributions, and helps users. Or at least he's trying... :)
rdedup currently does not implement own backup/restore functionality (own
directory traversal), and because of that it's typically paired with
rdup tools. Built-in directory traversal could improve deduplication
ratio for workloads with many small, frequently changing files.
Cloud storage integrations are missing. The architecture to support it is mostly implemented, but the actual backends are not.
If you have
cargo install rdedup
If not, I highly recommend installing rustup (think
but for Rust)
If you're interested in running
rdedup with maximum possible performance,
RUSTFLAGS="-C target-cpu=native" cargo install rdedup --vers ...
In case of troubles, check rdedup building issues or report a new one (sorry)!
rdedup -h for help.
Rdedup always operates on a repo, that you provide as an argument
--dir <DIR>), or via environment variable (eg.
rdedup init- create a new repo.
rdedup init --helpfor repository configuration options.
rdedup store <name>- store data from standard input under a given name.
rdedup load <name>- load data stored under given name and write it to standard output.
rdedup rm <name>- remove the given name.
rdedup ls- list all stored names.
rdedup gc- remove any no longer reachable data.
In combination with rdup this can be used to store and restore your backup like this:
rdup -x /dev/null "$HOME" | rdedup store home rdedup load home | rdup-up "$HOME.restored"
rdedup is data agnostic, so formats like
cpio and other will
but to get benefits of deduplication, archive format should not be
or encrypted already.
While it's not advised, if
RDEDUP_PASSPHRASE is defined, it will be used
instead of interactively asking user for password.
rdedup is licensed under: MPL-2.0