MARSYAS: A framework for audio analysis
đź“ś Abstract
Existing audio tools handle the increasing amount of computer audio data inadequately. The typical tape-recorder paradigm for audio interfaces is inflexible and time consuming, especially for large data sets. On the other hand, completely automatic audio analysis and annotation is impossible using current techniques. Alternative solutions are semi-automatic user interfaces that let users interact with sound in flexible ways based on content. This approach offers significant advantages over manual browsing, annotation and retrieval. Furthermore, it can be implemented using existing techniques for audio content analysis in restricted domains. This paper describes MARSYAS, a framework for experimenting, evaluating and integrating such techniques. As a test for the architecture, some recently proposed techniques have been implemented and tested. In addition, a new method for temporal segmentation based on audio texture is described. This method is combined with audio analysis techniques and used for hierarchical browsing, classification and annotation of audio files.
✨ Summary
The paper presents MARSYAS, a modular framework for audio analysis that combines signal transformations, feature extraction, temporal memories, iterators, classifiers, segmentors, and hierarchical annotation structures. Its client–server architecture separates a C++ signal-processing engine from a Java user interface, enabling alternative interfaces and rapid experimentation with new algorithms.
The principal technical contribution is a texture-based segmentation method. It computes frame-level feature vectors, measures changes between successive vectors using a covariance-aware distance, detects peaks in the derivative of that distance signal, and converts selected peaks into annotated time regions. These regions support faster browsing, similarity retrieval, classification, and hierarchical time-tree annotation. The paper also demonstrates a music/speech discriminator, reporting 90.1% accuracy with a five-neighbor K-NN classifier on a roughly two-hour evaluation collection.
Subsequent work documents MARSYAS’s continuation as an open-source music-information-retrieval framework. Marsyas-0.2 extended the architecture toward runtime dataflow composition and was used to prototype distributed audio-feature extraction across multiple computers. (webhome.cs.uvic.ca) Later development introduced MarsyasX, extending the framework’s dataflow model beyond audio toward cross-modal multimedia processing. (researchgate.net) Other research redeployed MARSYAS for semantic extraction from mixed soundtracks containing overlapping speech and music, demonstrating reuse of the framework in a broader audio-scene-analysis application. (researchgate.net) These sources establish continued academic development and application of the framework; they do not, by themselves, establish specific commercial products based on the original paper. The publication metadata is confirmed by Cambridge University Press: December 2000, Organised Sound, volume 4, issue 3, pages 169–175. (cambridge.org)