fix(CI): fix test suite and add CI steps (#1808) ## Problem Badger tests are run through a bash script called test.sh. It is currently not working, and there are no CI workflows on the Badger repository. ## Solution We update test.sh and add CI workflow steps. Test suite now works for Linux and Mac (x86, M1). To run the test suite, simply run `make test`. If you are on Mac, see remark below. ### Why a Makefile? Badger depends on jemalloc for efficient memory allocation (see [here](https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/), and z package in Ristretto). While Badger can be built without jemalloc, many users will probably want to benefit from it. The makefile makes it easy to install the jemalloc dependency with `make jemalloc`. Also now the test.sh script contains only test related functionality. This also has the advantage that now the CI workflow only needs essentially two steps: ``` make dependency make test ``` ### Remarks - In pb/gen.sh, `go get` is [deprecated](https://go.dev/doc/go-get-install-deprecation) as a way to retrieve and install an executable into our $GOBIN, which is what we want to do here. We use `go install` instead. - In test.sh, Teamcity flags no longer needed - In test.sh, Installjemalloc no longer needed as it is in the Makefile, simplifying the test script - In test.sh, tests are now run sequentially and not in parallel (currently broken) - We remove .travisci because it is unused - Important note for Mac users: for historical reasons, certain tools on MacOS are different from the standard GNU tools everyone else uses. One of these is the `mktemp` command, which is used in the test suite. In order to get the GNU version of this tool, you can run: ``` brew install coreutils export PATH="$(brew --prefix)/opt/coreutils/libexec/gnubin:$PATH" make test ``` This will temporarily modify your path so that the GNU version of mktemp is called in the script. Without this fix the script will complain that it doesn't recognize the p flag in `mktemp -d -p .` #### To-do - tune lint tests - add code coverage - potentially add build target for badger in makefile - bring test suite to parity with [old teamcity setup](https://teamcity.dgraph.io/project.html?projectId=Badger) - prune dependencies
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast, distributed graph database. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.
Use Discuss Issues for reporting issues about this repository.
Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supports concurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-style bank test runs nightly for 8h, with --race flag and ensures the maintenance of transactional guarantees. Badger has also been tested to work with filesystem level anomalies, to ensure persistence and consistency. Badger is being used by a number of projects which includes Dgraph, Jaeger Tracing, UsenetExpress, and many more.
The list of projects using Badger can be found here.
Badger v1.0 was released in Nov 2017, and the latest version that is data-compatible with v1.0 is v1.6.0.
Badger v2.0 was released in Nov 2019 with a new storage format which won't be compatible with all of the v1.x. Badger v2.0 supports compression, encryption and uses a cache to speed up lookup.
The Changelog is kept fairly up-to-date.
For more details on our version naming schema please read Choosing a version.
To start using Badger, install Go 1.12 or above. Badger v2 needs go modules. Run the following command to retrieve the library.
$ go get github.com/dgraph-io/badger/v3
This will retrieve the library.
Download and extract the latest Badger DB release from https://github.com/dgraph-io/badger/releases and then run the following commands.
$ cd badger-<version>/badger $ go install
This will install the badger command line utility into your $GOBIN path.
BadgerDB is a pretty special package from the point of view that the most important change we can make to it is not on its API but rather on how data is stored on disk.
This is why we follow a version naming schema that differs from Semantic Versioning.
Following these rules:
For a longer explanation on the reasons behind using a new versioning naming schema, you can read VERSIONING.md.
Badger Documentation is available at https://dgraph.io/docs/badger
Badger was written with these design goals in mind:
Badger’s design is based on a paper titled WiscKey: Separating Keys from Values in SSD-conscious Storage.
| Feature | Badger | RocksDB | BoltDB |
|---|---|---|---|
| Design | LSM tree with value log | LSM tree only | B+ tree |
| High Read throughput | Yes | No | Yes |
| High Write throughput | Yes | Yes | No |
| Designed for SSDs | Yes (with latest research 1) | Not specifically 2 | No |
| Embeddable | Yes | Yes | Yes |
| Sorted KV access | Yes | Yes | Yes |
| Pure Go (no Cgo) | Yes | No | Yes |
| Transactions | Yes, ACID, concurrent with SSI3 | Yes (but non-ACID) | Yes, ACID |
| Snapshots | Yes | Yes | Yes |
| TTL support | Yes | Yes | No |
| 3D access (key-value-version) | Yes4 | No | No |
1 The WISCKEY paper (on which Badger is based) saw big wins with separating values from keys, significantly reducing the write amplification compared to a typical LSM tree.
2 RocksDB is an SSD optimized version of LevelDB, which was designed specifically for rotating disks. As such RocksDB‘s design isn’t aimed at SSDs.
3 SSI: Serializable Snapshot Isolation. For more details, see the blog post Concurrent ACID Transactions in Badger
4 Badger provides direct access to value versions via its Iterator API. Users can also specify how many versions to keep per key via Options.
We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and the detailed logs for the benchmarks can be found in the badger-bench repo. More explanation, including graphs can be found the blog posts (linked above).
Below is a list of known projects that use Badger:
badger.Iterator, simplifying from-to, and prefix mechanics.If you are using Badger in a project please send a pull request to add it to the list.
If you're interested in contributing to Badger see CONTRIBUTING.md.