publications

2023

1. OSF

   Pre-Registration Effect of Stress on the Expression of Signs of Understanding During Game Explanations

   Johnson, David, Paletschek, Jonas, Bleimling, Jan, and Drimalla, Hanna

   2023

   Abs HTML

   Automated explaining agents would benefit from monitoring the understanding of human interactants while providing explanations. Individuals, however, differ regarding their expressivity of emotions and mental states, and in certain cases, these social interaction signals deviate strongly from so-called “typical” signals. Therefore it is necessary to investigate inter- and intra-individual variations of nonverbal multimodal signals of a receiver of an explanation (“explainee”) to enhance the inclusiveness of automated monitoring of an explainee’s understanding. While this study is part of a larger research project that investigates these differences, the focus of this study is on understanding the intra-individual differences by investigating how the interaction situation itself influences signal production and expression. A common situational confound of explanation scenarios is stress (e.g., due to time pressure or power imbalances). Stress affects non-verbal behavior in many ways, such as altering voice characteristics or suppressing facial behavior. Expressions of (mis)understanding are likely confounded by such stress-related behavioral signs. As far as we know, this has not been systematically investigated. Therefore, we will study the effect stress has on the expression of signs of understanding. This will be done in a within-subjects study in which we video record participants’ facial behavior while they receive explanations of board game rules, once during a scenario in which stress is induced (the stressed condition) and once in a relaxed setting with no induction of stress (the neutral condition). Participants will self-annotate the recordings from both conditions by continuously annotating changes in a felt state of understanding, or confusion while watching the recordings. The annotation process results in annotations for three mental states: understanding, confusion, or neutral (in which no change from their previous state is taking place). The video recordings will be used to measure specific facial behaviors for each of the states to compare expressivity between the neutral and stressed conditions.

2. ICAD

   Facial Behavior Sonification with the Interactive Sonification Framework Panson

   Nalli, Michele, Johnson, David, and Hermann, Thomas

   In Proceedings of the 28th International Conference on Auditory Display (ICAD 2023) 2023

   Abs HTML

   Facial behavior occupies a central role in social interaction. Its auditory representation is useful for various applications such as for supporting the visually impaired, for actors to train emotional expression, and for supporting annotation of multi-modal behavioral corpora. In this paper we present a prototype system for interactive sonification of facial behavior that works both in real-time mode, using a webcam, and offline mode, analyzing a video file. The system is based on python and Jupyter notebooks, and relies on the python module sc3nb for sonification-related functionalities. Facial feature extraction is realized using OpenFace 2.0. Designing the system led to the development of a framework of reusable components to develop interactive sonification applications, called Panson, which can be used to easily design and adapt sonifications for different use cases. We present the main concepts behind the facial behavior sonification system and the Panson framework. Furthermore, we introduce and discuss novel sonifications developed using Panson, and demonstrate them with a set of sonified videos. The sonifications and Panson are Open Source reproducible research available on GitHub.

3. Arxiv

   Towards Interpretability in Audio and Visual Affective Machine Learning: A Review

   Johnson, David S., Hakobyan, Olya, and Drimalla, Hanna

   Abs PDF

   Machine learning is frequently used in affective computing, but presents challenges due the opacity of stateof-the-art machine learning methods. Because of the impact affective machine learning systems may have on an individual’s life, it is important that models be made transparent to detect and mitigate biased decision making. In this regard, affective machine learning could benefit from the recent advancements in explainable artificial intelligence (XAI) research. We perform a structured literature review to examine the use of interpretability in the context of affective machine learning. We focus on studies using audio, visual, or audiovisual data for model training and identified 29 research articles. Our findings show an emergence of the use of interpretability methods in the last five years. However, their use is currently limited regarding the range of methods used, the depth of evaluations, and the consideration of use-cases. We outline the main gaps in the research and provide recommendations for researchers that aim to implement interpretable methods for affective machine learning.

2022

1. SAMOS

   SEC-Learn: Sensor Edge Cloud for Federated Learning

   Aichroth, Patrick, Antes, Christoph, Gembatzka, Pierre, Graf, Holger, Johnson, David S., Jung, Matthias, Kämpfe, Thomas, Kleinberger, Thomas, Köllmer, Thomas, Kuhn, Thomas, Kutter, Christoph, Krüger, Jens, Loroch, Dominik M., Lukashevich, Hanna, Laleni, Nellie, Zhang, Lei, Leugering, Johannes, Martín Fernández, Rodrigo, Mateu, Loreto, Mojumder, Shaown, Prautsch, Benjamin, Pscheidl, Ferdinand, Roscher, Karsten, Schneickert, Sören, Vanselow, Frank, Wallbott, Paul, Walter, Oliver, and Weber, Nico

   In Embedded Computer Systems: Architectures, Modeling, and Simulation

   Abs

   Due to the slow-down of Moore’s Law and Dennard Scaling, new disruptive computer architectures are mandatory. One such new approach is Neuromorphic Computing, which is inspired by the functionality of the human brain. In this position paper, we present the projected SEC-Learn ecosystem, which combines neuromorphic embedded architectures with Federated Learning in the cloud, and performance with data protection and energy efficiency.

2021

1. InterNoise

   Partial discharge monitoring using deep neural networks with acoustic emission

   Gourishetti, Saichand, Johnson, David S., Werner, Sara, Kátai, András, and Holstein, Peter

   In INTER-NOISE and NOISE-CON Congress and Conference Proceedings 2021
2. DAGA

   Assessment and Evaluation of an Unsupervised Machine Learning Model for Automotive and Industrial NVH Applications

   Paracha, Abdul Haq Azeem, Blickensdorff, Johannes, and Johnson, David S.

   In Jahrestagung der Akustik - DAGA 2020
3. EUSIPCO

   DESED-FL and URBAN-FL: Federated Learning Datasets for Sound Event Detection

   Johnson, David S., Lorenz, Wolfgang, Taenzer, Michael, Mimilakis, Stylianos, Grollmisch, Sascha, Abeßer, Jakob, and Lukashevich, Hanna

   In 2021 29th European Signal Processing Conference (EUSIPCO)

   Abs

   Research on sound event detection (SED) in environmental settings has seen increased attention in recent years. The large amounts of (private) domestic or urban audio data needed raise significant logistical and privacy concerns. The inherently distributed nature of these tasks, make federated learning (FL) a promising approach to take advantage of large-scale data while mitigating privacy issues. While FL has also seen increased attention recently, to the best of our knowledge there is no research towards FL for SED. To address this gap and foster further research in this field, we create and publish novel FL datasets for SED in domestic and urban environments. Furthermore, we conduct baseline results on the datasets in a FL context for three deep neural network architectures. The results indicate that FL is a promising approach for SED, but faces challenges with divergent data distributions inherent to distributed client edge devices.

4. IEEE

   Challenges and Strategies for Managing Requirements Selection in Software Ecosystems

   Damian, D., Linåker, J., Johnson, D., Clear, T., and Blincoe, K.

   IEEE Software (Accepted) 2021

2020

1. EUSIPCO

   Techniques Improving the Robustness of Deep Learning Models for Industrial Sound Analysis

   Johnson, D., and Grollmisch, S.

   In Proceedings of the 28th European Signal Processing Conference (EUSIPCO) 2020

   Abs PDF

   The field of Industrial Sound Analysis (ISA) aims to automatically identify faults in production machinery or manufactured goods by analyzing audio signals. Publications in this field have shown that the surface condition of metal balls and different types of bulk materials (screws, nuts, etc.) sliding down a tube can be classified with a high accuracy using audio signals and deep neural networks. However, these systems suffer from domain shift, or dataset bias, due to minor changes in the recording setup which may easily happen in real-world production lines. This paper aims at finding methods to increase robustness of existing detection systems to domain shift, ideally without the need to record new data or retrain the models. Through five experiments, we implement a convolutional neural network (CNN) for two publicly available ISA datasets and evaluate transfer learning, data normalization and data augmentation as approaches to deal with domain shift. Our results show that while supervised methods with additional labeled data are the best approach, an unsupervised method that implements data augmentation with adaptive normalization is able to improve the performance by a large margin without the need of retraining neural networks.

2. DCASE

   IAEO3 - Combining OpenL3 Embeddings and Interpolation Autoencoder for Anomalous Sound Detection

   Grollmisch, S., Johnson, D., AbeBer, J., and Lukashevich, H.

   2020

   Abs HTML

   In this technical report, we present our system for task 2 of the IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events (DCASE2020 Challenge): Unsupervised Detection of Anomalous Sounds for Machine Condition Monitoring. The focus of this task is to detect anomalous industrial machine sounds using an acoustic quality control system, which is only trained with sound samples from the normal (machine) condition. The dataset covers a variety of machines ranging from stable sound sources such as car engines, to transient sounds such as opening and closing valves. Our proposed method combines pre-trained OpenL3 embeddings with the reconstruction error of an interpolation autoencoder using a gaussian mixture model as the final predictor. The optimized model achieved 88.5% AUC and 76.8% pAUC on average over all machines and types provided with the development dataset, and outperformed the published baseline by 14.9% AUC and 17.2% pAUC.

3. CrowdRE

   Open CrowdRE Challenges in Software Ecosystems

   Johnson, D., Tizard, J., Damian, D., Blincoe, K., and Clear, T.

   In International Workshop on Crowd-Based Requirements Engineering (CrowdRE) 2020

   Abs

   CrowdRE has been argued to comprise four main activities: motivating Crowd members; eliciting feedback; analysing feedback and monitoring context and usage data. However, determining requirements for software ecosystems poses demands beyond those found by a single product owner. CrowdRE has particular challenges in handling the many, competing and heterogenous sources of RE relevant data in a software ecosystem. In this paper we pose these challenges against the background of Xero – a global accounting software company - as an ecosystem provider and Figured - a developer of a farming industry application - one of Xero’s ecosystem consumers.

4. InterNoise

   Compressed Air Leakage Detection Using Acoustic Emissions with Neural Networks

   Johnson, D., Kirner, J., Grollmisch, S., and Liebetrau, J.

   In Proceedings of the 49th Congress and Exposition on Noise Control Engineering (Inter-Noise) 2020

   Abs

   Compressed air is utilized in many branches of industry and represents one of the most expensive energy sources of industrial plants. The efficient detection of air pressure leaks goes hand-in-hand with cost savings and increased operational reliability. Some procedures of leakage detection for pressure lines are based upon the analysis of sound emissions. Such solutions use specific ultrasonic emission patterns to detect leakage; alternatively, personnel trained to hear leaks are deployed for detection. In this paper, we evaluate the potential of airborne sound analysis in the audible hearing range for the automated detection of compressed air leakage using artificial neural networks. Therefore, a novel dataset is developed and published. It contains recordings of adjustable leakage from a pneumatic contraption with different pressure levels from several microphones at different distances. Additionally, industrial background noises were applied to simulate real-world sound environments. Using this dataset, a convolutional neural network is trained for leakage detection. The results show that leakage detection by means of airborne sound in the audible range using machine learning techniques is possible and a promising contactless automated detection method.

5. SMSI

   Plastic Material Classification using Neural Network based Audio Signal Analysis

   Grollmisch, S., Johnson, D., Krüger, T., and Liebetrau, J.

   In Proceedings of Sensor and Measurement Science International (SMSI) 2020

   Abs HTML

   Analyzing the acoustic response of products being struck is a potential method to detect material deviations or faults for automated quality control. To evaluate this, we implement a material detection system by equipping an air hockey table with two microphones and plastic pucks 3D printed using different materials. Using this setup, a dataset of the acoustic response of impacts on plastic materials was developed and published. A convolutional neural network trained on this data, achieved high classification accuracy even under noisy conditions demonstrating the potential of this approach.

6. SMSI

   Visualizing Neural Network Decisions for Industrial Sound Analysis

   Grollmisch, S., Johnson, D., and Liebetrau, J.

   In Proceedings of Sensor and Measurement Science International (SMSI) 2020

   Abs HTML

   Recent research has shown acoustic quality control using audio signal processing and neural networks to be a viable solution for detecting product faults in noisy factory environments. For industrial partners, it is important to be able to explain the network’s decision making, however, there is limited research on this area in the field of industrial sound analysis (ISA). In this work, we visualize learned patterns of an existing network to gain insights about the decision making process. We show that unwanted biases can be discovered, and thus avoided, using this technique when validating acoustic quality control systems.

7. CMJ

   Detecting Hand Posture in Piano Playing Using Depth Data

   Johnson, D., Damian, D., and Tzanetakis, G.

   Computer Music Journal 2020

   Abs HTML

   We present research for automatic assessment of pianist hand posture that is intended to help beginning piano students improve their piano-playing technique during practice sessions. To automatically assess a student’s hand posture, we propose a system that is able to recognize three categories of postures from a single depth map containing a pianist’s hands during performance. This is achieved through a computer vision pipeline that uses machine learning on the depth maps for both hand segmentation and detection of hand posture. First, we segment the left and right hands from the scene captured in the depth map using per-pixel classification. To train the hand-segmentation models, we experiment with two feature descriptors, depth image features and depth context features, that describe the context of individual pixels’ neighborhoods. After the hands have been segmented from the depth map, a posture-detection model classifies each hand as one of three possible posture categories: correct posture, low wrists, or flat hands. Two methods are tested for extracting descriptors from the segmented hands, histograms of oriented gradients and histograms of normal vectors. To account for variation in hand size and practice space, detection models are individually built for each student using support vector machines with the extracted descriptors. We validate this approach using a data set that was collected by recording four beginning piano students while performing standard practice exercises. The results presented in this article show the effectiveness of this approach, with depth context features and histograms of normal vectors performing the best.

2019

1. VR

   Evaluating the effectiveness of mixed reality music instrument learning with the theremin

   Johnson, D., Damian, D., and Tzanetakis, G.

   Virtual Reality 2019

   Abs HTML

   Learning music is a challenging process that requires years of practice to master, either with lessons from a professional teacher or through self-teaching. While practicing, students are expected to self-evaluate their performance which may be difficult without timely feedback from a professional. Research into computer-assisted music instrument tutoring (CAMIT) attempts to address this through the use of emerging technologies. In this paper, we study CAMIT for mixed reality (MR) by developing MR:emin, an immersive MR music learning environment for the theremin, an electronic music instrument that is controlled without physical contact. MR:emin integrates a physical theremin with the immersive learning environment. To better understand the effectiveness of such environments, we perform a user study with MR:emin comparing traditional music learning with two virtual learning environments, an immersive one and a non-immersive one. In a between-groups study, 30 participants were trained to play a sequence of notes on the theremin using one of the three training environments. Results of our statistical analysis show that performance error during training is significantly smaller in the immersive MR environment. This does not necessarily lead to improved performance after training; analysis of post-training improvement indicates that immersive training results in the smallest amount of improvement. Participants, however, indicate that the MR:emin environment is more engaging and increases confidence during practice. We discuss potential factors leading to the decrease in learning and provide some environment guidelines to aid in the design of engaging immersive music learning environments.

2. PhD Thesis

   Johnson, D. (2019). MusE-XR: musical experiences in extended reality to enhance learning and performance. University of Victoria.

   Abs HTML

   Integrating state-of-the-art sensory and display technologies with 3D computer graphics, extended reality (XR) affords capabilities to create enhanced human experiences by merging virtual elements with the real world. To better understand how Sound and Music Computing (SMC) can benefit from the capabilities of XR, this thesis presents novel research on the de- sign of musical experiences in extended reality (MusE-XR). Integrating XR with research on computer assisted musical instrument tutoring (CAMIT) as well as New Interfaces for Musical Expression (NIME), I explore the MusE-XR design space to contribute to a better understanding of the capabilities of XR for SMC. The first area of focus in this thesis is the application of XR technologies to CAMIT enabling extended reality enhanced musical instrument learning (XREMIL). A common approach in CAMIT is the automatic assessment of musical performance. Generally, these systems focus on the aural quality of the performance, but emerging XR related sensory technologies afford the development of systems to assess playing technique. Employing these technologies, the first contribution in this thesis is a CAMIT system for the automatic assessment of pianist hand posture using depth data. Hand posture assessment is performed through an applied computer vision (CV) and machine learning (ML) pipeline to classify a pianist’s hands captured by a depth camera into one of three posture classes. Assessment results from the system are intended to be integrated into a CAMIT interface to deliver feedback to students regarding their hand posture. One method to present the feedback is through real-time visual feedback (RTVF) displayed on a standard 2D computer display, but this method is limited by a need for the student to constantly shift focus between the instrument and the display. XR affords new methods to potentially address this limitation through capabilities to directly augment a musical instrument with RTVF by overlaying 3D virtual objects on the instrument. Due to limited research evaluating effectiveness of this approach, it is unclear how the added cognitive demands of RTVF in virtual environments (VEs) affect the learning process. To fill this gap, the second major contribution of this thesis is the first known user study evaluating the effectiveness of XREMIL. Results of the study show that an XR environment with RTVF improves participant performance during training, but may lead to decreased improvement after the training. On the other hand,interviews with participants indicate that the XR environment increased their confidence leading them to feel more engaged during training. In addition to enhancing CAMIT, the second area of focus in this thesis is the application of XR to NIME enabling virtual environments for musical expression (VEME). Development of VEME requires a workflow that integrates XR development tools with existing sound design tools. This presents numerous technical challenges, especially to novice XR developers. To simplify this process and facilitate VEME development, the third major contribution of this thesis is an open source toolkit, called OSC-XR. OSC-XR makes VEME development more accessible by providing developers with readily available Open Sound Control (OSC) virtual controllers. I present three new VEMEs, developed with OSC-XR, to identify affordances and guidelines for VEME design. The insights gained through these studies exploring the application of XR to musical learning and performance, lead to new affordances and guidelines for the design of effective and engaging MusE-XR.

3. SMC

   DAW-Integrated Beat Tracking for Music Production

   Dalton, B., Johnson, D., and Tzanetakis, G.

   In 16th Sound & Music Computing Conference (SMC) 2019

   Abs HTML

   Rhythm analysis is a well researched area in music information retrieval that has many useful applications in music production. In particular, it can be used to synchronize the tempo of audio recordings with a digital audio workstation (DAW). Conventionally this is done by stretching recordings over time, however, this can introduce artifacts and alter the rhythmic characteristics of the audio. Instead, this research explores how rhythm analysis can be used to do the reverse by synchronizing a DAW’s tempo to a source recording. Drawing on research by Percival and Tzanetakis, a simple beat extraction algorithm was developed and integrated with the Renoise DAW. The results of this experiment show that, using user input from a DAW, even a simple algorithm can perform on par with popular packages for rhythm analysis such as BeatRoot, IBT, and aubio.

4. SMC

   OSC-XR: A Toolkit for Extended Reality Immersive Music Interfaces

   Johnson, D., Damian, D., and Tzanetakis, G.

   In 16th Sound & Music Computing Conference (SMC) 2019

   Abs HTML

   Currently, developing immersive music environments for extended reality (XR) can be a tedious process requiring designers to build 3D audio controllers from scratch. OSCXR is a toolkit for Unity intended to speed up this process through rapid prototyping, enabling research in this emerging field. Designed with multi-touch OSC controllers in mind, OSC-XR simplifies the process of designing immersive music environments by providing prebuilt OSC controllers and Unity scripts for designing custom ones. In this work, we describe the toolkit’s infrastructure and perform an evaluation of the controllers to validate the generated control data. In addition to OSC-XR, we present UnityOscLib, a simplified OSC library for Unity utilized by OSC-XR. We implemented three use cases, using OSCXR, to inform its design and demonstrate its capabilities. The Sonic Playground is an immersive environment for controlling audio patches. Hyperemin is an XR hyperinstrument environment in which we augment a physical theremin with OSC-XR controllers for real-time control of audio processing. Lastly, we add OSC-XR controllers to an immersive T-SNE visualization of music genre data for enhanced exploration and sonification of the data. Through these use cases, we explore and discuss the affordances of OSC-XR and immersive music interfaces.

2017

1. NIME

   VRMIN: Using Mixed Reality to Augment the Theremin for Musical Tutoring.

   Johnson, D., and Tzanetakis, G.

   In Proceedings of New Interfaces for Musical Expression (NIME) 2017

   Abs HTML

   The recent resurgence of Virtual Reality (VR) technologies provide new platforms for augmenting traditional music instruments. Instrument augmentation is a common approach for designing new interfaces for musical expression, as shown through hyperinstrument research. New visual affordances present in VR give designers new methods for augmenting instruments to extend not only their expressivity, but also their capabilities for computer assisted tutoring. In this work, we present VRMin, a mobile Mixed Reality (MR) application for augmenting a physical theremin, with an immersive virtual environment (VE), for real time computer assisted tutoring. We augment a physical theremin with 3D visual cues to indicate correct hand positioning for performing given notes and volumes. The physical theremin acts as a domain specific controller for the resulting MR environment. The initial effectiveness of this approach is measured by analyzing a performer’s hand position while training with and without the VRMin. We also evaluate the usability of the interface using heuristic evaluation based on a newly proposed set of guidelines designed for VR musical environments.

2016

1. ICMC

   Detecting pianist hand posture mistakes for virtual piano tutoring

   Johnson, D., Dufour, I., Damian, D., and Tzanetakis, G.

   In Proceedings of the International Computer Music Conference (ICMC) 2016

   Abs PDF

   Incorrect hand posture is known to cause fatigue and hand injuries in pianists of all levels. Our research is intended to reduce these problems through new methods of providing direct feedback to piano students during their daily practice. This paper presents an approach to detect hand posture in RGB-D recordings of pianists’ hands while practicing for use in a virtual music tutor. We do so through image processing and machine learning. To test this approach we collect data by recording the hands of two pianists during standard piano exercises. Preliminary results show the effectiveness of our methods.

2015

1. PacRim

   Guitar model recognition from single instrument audio recordings

   Johnson, D., and Tzanetakis, G.

   In Proceedings of the 2015 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM) 2015

   Abs HTML

   The main goal of this paper is to explore the recognition of particular guitar models from single instrument audio recordings. This is different than existing work in music instrument recognition that deals with identifying different instrument types. Through a set of experiments we evaluate different sets of audio features and classifiers for this purpose. To improve accuracy a composite classifier is implemented to first discriminate between electric and acoustic guitars. This affords flexibility in training different models for each guitar type. A data set consisting of audio recordings from 15 guitar models, each recorded with a set of different playing configurations, is used for training and testing. We have found that K Nearest Neighbors and Support Vector Machine (SVM) classifiers perform the best. Testing is done by leaving a specific playing configuration out of the training model. Specific test cases show satisfactory results, with one test case achieving over 70% accuracy and a second one over 50%; both using a composite SVM model.

2. ISEA

   Diving into Infinity: A Motion-Based, Immersive Interface for MC Escher’s Works

   Manaris, B., Johnson, D., and Rourk, M.

   In Proceedings of the 21st International Symposium on Electronic Art (ISEA 2015) 2015

   Abs PDF

   We describe a Kinect-based interface for navigating M.C. Escher’s works. Our interface is based on the Kuatro, a framework for developing motion-based interactive virtual environments. Kuatro utilizes the Model-View Controller (MVC) architecture and Open Sound Control (OSC) to provide an expandable environment for motion-sensor based installations for composers, artists, and interaction designers. We present a case study based on “Print Gallery”, an intriguing, self-similar work created by M.C. Escher in 1956. Our interaction design involves a Kinect sensor, a video projector, a Kuatro server, and a screen; it allows a user to zoom in and out, as well as rotate the image to reveal its self-similarity, by navigating prerecorded video material. This material is based on previous mathematical analyses of “Print Gallery” to reveal / explain the artist’s depiction of infinity. We discuss adapting this approach to other M.C. Escher works involving infinity.

2014

1. ICMC

   Kuatro: A Motion-Based Framework for Interactive Music Installations

   Johnson, D., Manaris, B. Z, Vassilandonakis, Y., and Stoudenmier, S.

   In Proceedings of the International Computer Music Conference (ICMC) 2014

   Abs HTML

   Kuatro is a development framework for motion-based interactive virtual environments. Using a Model-View-Controller (MVC) architecture and the Open Sound Control (OSC) protocol, the Kuatro offers composers, artists, and interaction designers an environment that makes it easy to develop installations utilizing motion sensors, such as the Kinect, Asus Xtion, Leap Motion, and smartphones. The framework allows tracking multiple users to help designers create collaborative (or competitive) interactive experiences. The main components of the framework are described, as well as the messaging protocol. Through OSC messaging, developers and artists are able to create any number visual and audio interfaces as well as add any number of motion sensors to the environment. The Kuatro framework is conceived as an extension of the jythonMusic audio-visual programming environment. Herein we present the Kuatro framework in the context of the interactive multimedia art installation, Time Jitters. We also discuss a follow-up music composition project, called Pythagorean Tetraktys.

2. HCI

   Harmonic Navigator: An Innovative, Gesture-Driven User Interface for Exploring Harmonic Spaces in Musical Corpora

   Johnson, D., Manaris, B., and Vassilandonakis, Y.

   In Human-Computer Interaction. Advanced Interaction Modalities and Techniques 2014

   Abs HTML

   We present Harmonic Navigator (HN), a system for navigating and exploring harmonic spaces extracted from large musical corpora, to be used in music composition and performance. A harmonic space is a set of harmonies (chords) and transitions between harmonies found in a music corpus. By navigating this space, the user can derive new harmonic progressions, which have correct voice leading. HN is controllable via a Kinect gesture interface. To aid the user, the system also incorporates stochastic and evolutionary techniques. HN offers for two primary modes of interaction: a harmonic transition selector, called harmonic palette, which utilizes a GUI to navigate harmonic transitions in a front-to-back manner; and a harmonic-flow scrubber, which presents a global overview of a harmonic flow and allows the user to perform common audio scrubbing and editing tasks. Both GUIs use colors to indicate harmonic density based on Legname’s density degree theory.

3. EvoMusArt

   A Novelty Search and Power-Law-Based Genetic Algorithm for Exploring Harmonic Spaces in J.S. Bach Chorales

   Manaris, B., Johnson, D., and Vassilandonakis, Y.

   In Proceedings of the International Conference on Evolutionary and Biologically Inspired Music and Art (EvoMusArt) 2014

   Abs HTML

   We present a novel, real-time system, called Harmonic Navigator, for exploring the harmonic space in J.S. Bach Chorales. This corpus-based environment explores trajectories through harmonic space. It supports visual exploration and navigation of harmonic transition probabilities through interactive gesture control. These probabilities are computed from musical corpora (in MIDI format). Herein we utilize the 371 J.S. Bach Chorales of the Riemenschneider edition. Our system utilizes a hybrid novelty search approach combined with power-law metrics for evaluating fitness of individuals, as a search termination criterion. We explore how novelty search can aid in the discovery of new harmonic progressions through this space as represented by a Markov model capturing probabilities of transitions between harmonies. Our results demonstrate that the 371 Bach Chorale harmonic space is rich with novel aesthetic possibilities, possibilities that the grand master himself never realized.

2013

1. AIIDE

   Harmonic Navigator: A Gesture-Driven, Corpus-Based Approach to Music Analysis, Composition, and Performance.

   Manaris, B., Johnson, D., and Vassilandonakis, Y.

   In Proceedings of the 9th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2013) 2013

   Abs PDF

   We present a novel, real-time system for exploring harmonic spaces of musical styles, to generate music in collaboration with human performers utilizing gesture devices (such as the Kinect) together with MIDI and OSC instruments/controllers. This corpus-based environment incorporates statistical and evolutionary components for exploring potential flows through harmonic spaces, utilizing power-law (Zipf-based) metrics for fitness evaluation. It supports visual exploration and navigation of harmonic transition probabilities through interactive gesture control. These probabilities are computed from musical corpora (in MIDI format). Herein we utilize the Classical Music Archives 14,000+ MIDI corpus, among others. The user interface supports real-time exploration of the balance between predictability and surprise for musical composition and performance, and may be used in a variety of musical contexts and applications.