XaiPient’s mission is to make AI transparent and easy to understand by offering ways to inspect the “whys” behind AI systems’ decisions. A big problem of machine models today is that they are good at predicting certain outcomes with a high degree of accuracy, but fail at explaining how they arrive at those outcomes (a problem known as black-box). As more aspects of the world are automated by AI, knowing how AI models operate is crucial to prevent accidents and biases when these systems are deployed in the wild.
We joined the founding team for four months to help them turn years of technical research into a tangible product. We helped the product team make sense of the fast-growing space of interpretability (XAI) by mapping out different competitors, investigating potential new technologies, and identifying relevant issues regarding the design of explanations in a wide range of applications, and for different users and domains. After gathering insights from users, we produced prototypes to validate hypotheses about the value proposition of certain features and aspects of the product.
The biggest design challenge we faced was: how do we build human-friendly explanations that let users understand what influenced a machine decision without relying on technical concepts and complex data visualizations? Additionally, how do we communicate decisions effectively for different levels of literacy and use cases? Since the prediction made by a machine learning system is probabilistic, how do we communicate uncertainty without influencing user behavior? And finally, how do we prevent explanations from being misunderstood, potentially putting at risk those affected by these very systems?
After validating different hypotheses and gaining insights about different use cases, we shifted our focus towards ideating on solutions. We prioritized thinking in components and design patterns rather than worrying about flows and secondary navigation aspects of the application. This decision enabled the team to make progress on the core components of the product before the strategy was fully defined. We worked with developers and designers to create a highly customizable pattern library comprising different explanation methods and visualizations that matched different levels of data literacy, goals and needs.
User Reserach (Andrea Kelemen ↗)
Data Visualization Design
Iterative Prototyping & Wireframing
User Testing & Market Fit
Product Stories & Storyboarding
A big problem of machine models today is that they are good at predicting outcomes with a high degree of accuracy, but fail at explaining how they arrive at those outcomes (a problem known as black-box).
Experts interview and literature review
Since AI effects so many aspects of society today, we knew it was essential to include different voices and perspectives beyond the ones found in the team. Thinking about the problem through a computational lens only solved part of the puzzle. It was necessary to go beyond initial assumptions if we were to design solutions that addressed the design challenges holistically. To gain such a perspective, we interviewed cognitive scientists, statisticians, HCI researchers, and scholars involved in algorithmic fairness to uncover insights about the challenges and limitations of "explaining AI”, and its implications for society at large.
Explainable AI today has a strong focus on mathematics and computer science to interpret “black box” models. While these are significant contributions, they tend to neglect the human side of the explanations and whether they are usable and practical in real-world situations.
Existing applications are focusing on offering complex data visualizations for data scientists and neglect a much larger user base comprising non-technical users (e.g. underwriters, regulators, and users working with machine learning algorithms in different applications to make predictions).
Recognizing this disparity, how can we develop visualizations and explanations that communicate the decision behind a prediction using natural language and non-technical components?
The need for interpretability stems from incompleteness in the problem formalization, creating a fundamental barrier to optimization and the system's evaluation.
Decision-makers, scientists, compliance and safety engineers, data scientists, and machine learning researchers all come with different background knowledge and communication styles.
We also developed personas based on the user interviews and literature review we conducted. We created three distinct categories that related to the product directly: auditors, underwriters, and business users. We also crafted product stories to bring the design and development team on board with how explanations might be used, and what are the real stories and questions users are trying to answer. We mapped the requirements for the different explanation components; including tone of voice, depth of explanation, level of technical literacy, and UX requirements.
To facilitate brainstorming sessions, we created a taxonomy that included different personas (first column), different high-level tasks, the different categories of explanations (UX), and the available methods used to explain methods (computer science). Our goal was to have participants starting on the left (users) to the far right (technology). This way In the process, designers and developers could evaluate if the explanations offered a good experience. In addition, we created a second category that included time constraints involving a particular task, and the expertise different users might have.
Our experience was that visualizing qualitative data helped the team to think creatively around certain constraints (user needs). By having different aspects of the product on the page (from user needs to the underlying technology) the team was able to realize the deficiencies of certain methods against different tasks and users, as well as the required technology to create an adequate user experience for different users.
Some of the concepts emerging from sessions were extremely technical, featuring heat maps and other highly dense visualization elements. Others were much simpler to understand, featuring counterfactual explanations ("why a certain prediction was made instead of another one"). Some of the most exciting ideas were based on logic programming and natural language processing and explored conversational AI models.
The research we conducted also helped to clarify the design of the SaaS architecture. Specifically, developing a solution that could work with already trained models instead of requiring proprietary technology became obvious to the team after talking to different users to understand their workflow.
Disclaimer: The solutions presented here are only illustrative and were not used in the final designs. Due to my NDA agreement, we're not able to disclose the final project outcome. If you have further questions or want to know more about a specific design solution, please get in touch.
Designing a pattern library of explainability components
Since aspects of the product strategy hadn't been defined, we suggested the team to focus on individual explanation components. Each component was focused on individual tasks and users, using different visualization techniques to communicate information. This way, components could be designed, validated, and tested without worrying about secondary aspects of the software architecture.
How might we communicate complex data-centric decisions to a non-technical audience? It was extremely important for us to think beyond regular data visualization components found in most data-centric applications (e.g. bar charts, plots, etc). We were interested in exploring the design of new components that were not intimidating, yet offering the right amount of information for the task and user literacy.
Designing the future of human-AI collaboration through transparent and easy-to-use interfaces
How can AI serve as a tool for cognitive augmentation? The project explores new ways humans can work with generative algorithms. I worked from UX research to designing and validating the data visualization system, and finally delivering a technical proof of concept.