ESSENTIA: AN AUDIO ANALYSIS LIBRARY FOR MUSIC INFORMATION RETRIEVAL
📜 Abstract
We present Essentia 2.0, an open-source C++ library for audio analysis and audio-based music information retrieval released under the Affero GPL license. It contains an extensive collection of reusable algorithms which implement audio input/output functionality, standard digital signal processing blocks, statistical characterization of data, and a large set of spectral, temporal, tonal and high-level music descriptors. The library is also wrapped in Python and includes a number of predefined executable extractors for the available music descriptors, which facilitates its use for fast prototyping and allows setting up research experiments very rapidly. Furthermore, it includes a Vamp plugin to be used with Sonic Visualiser for visualization purposes. The library is cross-platform and currently supports Linux, Mac OS X, and Windows systems. Essentia is designed with a focus on the robustness of the provided music descriptors and is optimized in terms of the computational cost of the algorithms. The provided functionality, specifically the music descriptors included in-the-box and signal processing algorithms, is easily expandable and allows for both research experiments and development of large-scale industrial applications.
✨ Summary
Summary
The paper introduces Essentia 2.0, a cross-platform, open-source C++ library intended to unify a broad range of reusable audio-analysis and music-information-retrieval algorithms. Its capabilities cover audio input/output, DSP, spectral and temporal descriptors, tonal analysis, rhythm analysis, statistical aggregation, high-level classification, and executable feature extractors. Python bindings support rapid prototyping, while Vamp integration supports visualization in Sonic Visualiser. The architecture represents processing components as composable algorithms and provides both imperative standard processing and scheduler-driven streaming processing, allowing a trade-off between prototyping simplicity, memory use, and processing throughput.
The paper documents applications in music classification, semantic autotagging, similarity and recommendation, sound indexing, cover detection, beat detection, instrument analysis, and multimodal music-performance research. It also reports industrial or service use in systems associated with Freesound, BMAT, Stromatolite, Yamaha BODiBEAT, and Steinberg LoopMash.
Subsequent evidence indicates that Essentia became a continuing research platform rather than only a one-time software release. The project maintains a catalog of academic studies using Essentia across music analysis, classification, recommendation, segmentation, tonality, rhythm, source separation, speech, bioacoustics, and related areas. (essentia.upf.edu) The library’s official repository continues to describe the same core role and documents support for additional platforms and Python-based installation. (github.com) Essentia.js extended the library’s algorithms to browser and Node.js environments through WebAssembly, enabling web-based and real-time audio analysis. (github.com) Concrete research applications include AcousticBrainz, which used Essentia to gather audio-derived music information, and Dunya, which used it for culturally specific music interaction and analysis. (essentia.upf.edu)