Designing the future of human-AI collaboration through transparent and easy-to-use interfaces

Approach

I started the project by mapping out the space of AI research, evaluating available tools (apps, plugins, frameworks), their usability problems, and in general gathering insights about their designs. In addition, I worked with 6 composers throughout the design and ideation process to validate ideas and understand their needs when interacting with generative systems.

Problem

While machine learning works well in applications where a task can be easily generalized from training data, it fails at augmenting users in tasks where the goal is diverse, personal and not clearly defined. For example, current tools fail at: delivering novelty (new combinations not seen in the data) and offering granular control to users. This in turn, makes it difficult for musicians to find desired musical outcomes, overwhelming them with an infinite number of ideas, all very unrelated to the artist’s initial intention. In summary, the current design paradigm that machine learning applications are based on needs to be challenged if AI is to be a tool that can augment human capabilities beyond what either a human or machine can achieve in isolation.

Acheivements

Gradient introduces an alternative approach to designing with AI by focusing on making its algorithms transparent and easy to understand by its users. As a result, the interface allows composers to express their ideas by interacting with the algorithm to steer it towards novel outcomes. Instead of trying to create a system that composes for them, Gradient builds a way that musicians can compose with. By mixing simple components in a node-graph interface, new patterns can be reconfigured to create infinite new musical permutations. While Gradient focuses on music generation, the learnings acquired here can be used in different applications such as text generation or scientific discovery.

Evaluating existing tools

I began my research by evaluating existing tools through looking at their interaction models (how they work and what metaphors are they based on) and their information architecture. I used these tools myself to compose and explore musical ideas and see how they work. I followed a process similar to heuristic evaluation and task analysis, with the exception of not engaging a group of specialists. This step helped me understand how these applications differ, the spectrum of possibilities I could focus on, and what problems I was interested in solving moving forward.

My domain-specific knowledge in music composition and production allowed me to simulate the path a user might take from exploring websites, comparing value propositions, to installing and using tools.

I was interested in mapping out:

• Interaction models: how is the system designed and how does the user interact with it? What are the options and outcomes?
• Underlying technology: what machine learning technologies are they using (deep learning vs GANs)? Value proposition: how do they present themselves to artists? What features do they claim and what promises do they make?
• Value proposition: how do they present themselves to artists? What features do they claim and what promises do they make?
• User base: what user groups are they targeting?
• Musical output: what does the combination of interface and technology create? What kinds of musical output do certain systems prioritize?

Defining which users to focus on and why

The evaluation enabled me to understand the whole gamut of users different tools were targeting, and how certain user groups are currently being served by the tools available. By defining what each group needs and the tasks they are trying to solve, I was able to see if the solutions available could be improved upon. Specifically, I was able to uncover that composers were the least served by existing tools. Their needs are very specific and are the most challenging to design for. Composers need tools that are offering fine-grained controls, are deeply customizable, while also being easy to use. This in particular seems to be the focus in machine learning design over the next years: how do we make machine learning easily accessible not only to data scientists and programmers, but to everyone who wants to be augmented by what the technology can offer?

Another major insight I had was how the concept of augmentation was being used as a buzzword without considering what it might entail from an interaction design perspective. A lot of applications that claimed to augment composers in music creation tasks were in fact automating tasks for them.

User Research Overview

1. User interviews & contextual inquiry
2. Co-design exercises and sessions
3. Unfocused group panel discussion at CTM

User interviews

Once I defined my user group, I set out to understand how different composers approach the task of music composition. I visited them at their studios instead of meeting them remotely to get a better sense of their tools and their environment. I was also interested in understanding how they thought about music conceptually, and how concepts were translated into how they approached their tools (software, hardware, instrument).

I centered the interviews on a few open-ended questions to guide the discussion:

• User flow: How do they navigate from having an idea to finishing a composition? How do things begin and how things end?
• Entry points: What are the main entry points to their creative process? What are the structures that might guide the process?
• Conceptual models: What are the most prominent conceptual models that different composers have in common?
• Previous experience with ML: What are their frustrations, expectations and fascination with the technology?