Beyond Explainable AI

Home >

Beyond Explainable AI

Wojciech Samek and Klaus-Robert Mueller published new book on Explainable AI

To tap the full potential of artificial intelligence, not only do we need to understand the decisions it makes, these insights must also be made applicable. This is the aim of the new book “xxAI – Beyond Explainable AI”, edited by Wojciech Samek, head of the Artificial Intelligence department at the Fraunhofer Heinrich Hertz Institute (HHI) and BIFOLD researcher and Klaus-Robert Mueller, professor of machine learning at the Technical University of Berlin (TUB) and co-director at BIFOLD. The publication is based on a workshop held during the International Conference on Machine Learning in 2020. Co-editors also include AI experts Andreas Holzinger, Randy Goebel, Ruth Fong and Taesep Moon. It is already the second publication by Samek and Mueller.

Following the great resonance of the editors’ first book, “Explainable AI: Interpreting, Explaining and Visualizing Deep Learning” (2019), which presented an overview of methods and applications of Explainable AI (XAI) and racked up over 300,000 downloads worldwide, their new publication goes a step further. It provides an overview of current trends and developments in the field of XAI. In one chapter, for example, Samek and Mueller’s team shows that XAI concepts and methods developed for explaining classification problems can also be applied to other types of problems. When solving classification problems, the target variables sought are categorical, such as “What color is the traffic light right now, red, yellow, or green?”. XAI techniques for solving these problems can help explain problems in unsupervised learning, reinforcement learning, or generative models. Thus, the authors expand the horizons of previous XAI research and provide researchers and developers with a set of new tools that can be used to explain a whole new range of problem types and models.

The book is available free of charge.
C: Fraunhofer HHI

As the title “Beyond Explainable AI” suggests, the book also highlights solutions regarding the practical application of insights from methodological aspects to make models more robust and efficient. While previous research has focused on the process from AI as a “black box” to explaining its decisions, several chapters in the new book address the next step, toward an improved AI model. Furthermore, other authors reflect on their research not only in their own field of work, but also in the context of society as a whole. They cover a variety of areas that go far beyond classical XAI research. For example, they address the relationships between explainability and fairness, explainability and causality, and legal aspects of explainability.

The book is available free of charge here.

New professorship for Machine Learning and Communications

Home >

New professorship for Machine Learning and Communications

“I want to move beyond purely ‘explaining’ AI”

BIFOLD researcher Dr. Wojciech Samek has been appointed Professor of Machine Learning and Communications at TU Berlin with effect from 1 May 2022. Professor Samek heads the Department of Artificial Intelligence at the Fraunhofer Heinrich-Hertz-Institute.

Prof. Wojciech Samek receives his certificate of appointment from the President of TU Berlin, Prof. Geraldine Rauch.
C: private

Professor Samek – very warm congratulations on your appointment at TU Berlin. You have been involved in BIFOLD – Berlin’s AI competence center – since 2014. What connects you to BIFOLD?

“I have supported BIFOLD and its predecessor projects to the best of my abilities since the very beginning. My cooperation with BIFOLD is very important to me – as is strengthening Berlin’s role as a center of artificial intelligence (AI). I have worked very closely and successfully with BIFOLD Co-Director Professor Klaus-Robert Müller on the explainability of artificial intelligence. I have also enjoyed a successful cooperation with Professor Thomas Wiegand working at the interface between machine learning (ML) and compression. In addition to the 12 patent registrations resulting from these and other collaborative undertakings with researchers at TU Berlin, my collaboration with Thomas Wiegand has had a considerable influence on the new international MPEG-7 NNR standard on the compression of neural networks.”

What areas would you like to focus on in your future research at TU Berlin/BIFOLD?

“My goal is to further develop three areas: explainability and trustworthiness of artificial intelligence, the compression of neural networks, and so-called federated leaning. I aim to focus on the practical, methodological, and theoretical aspects of machine learning at the interface to other areas of application. The combination of machine learning and communications is something I find particularly interesting. In my research, I use explainability to improve ML models and to make them more robust. So, I am looking to move beyond purely “explaining.” My goal is to increase reliability and ultimately develop explanations based on checking mechanisms for ML models.”

How does this research complement the research work of BIFOLD and TU Berlin?

“These research areas are of great importance not only for BIFOLD with its focus on ’responsible AI’ and ‘communications’ as an area of application; they also offer a whole range of possibilities for collaboration with researchers at Faculty IV. Klaus-Robert Müller, Grégoire Montavon, and I are planning to transfer the explainability concepts we developed to other types of ML models and areas such as regression, segmentation, and clustering. Enhanced visualizations of explanations will enable us to develop more reliable and more explainable ML models for communication applications. My research will also focus on the development of enhanced compression methods for neural networks and federated learning. Examining the interaction between learning, compression, and communications is of great interest here. Overall, we can say that my research will strengthen the research profile of Faculty IV while at the same time offering very many new possibilities for cooperation at the interface between ML, information theory, signal compression, and communications.”

Do you already have an idea of what you would like to implement in teaching?

“I am really looking forward to the new challenges that teaching will bring. Within BIFOLD, I am planning a special course titled “Advanced Course on Machine Learning & Communications.” This would focus on federated learning, methods for the compression of neural networks, efficient neural architectures, and ML methods for communication applications. The lectures could also be supplemented by a lab. What I envisage is the implementation of federated learning on many networked devices. This would provide hands-on opportunities for students to learn directly about topics such as resource-saving ML, compression of neural networks, and federated learning. I would also be very interested in giving a course on explainable and trustworthy AI. A BIFOLD special course, a master’s/bachelor’s seminar, or a supplementary module to the bachelor’s course on cognitive algorithms would all provide suitable formats for this.”

Summer School Information Event

Home >

Summer School Information Event

Summer School Information Event

Date and Time: Monday, 25. April, 2022; 4:00 pm

Speaker: Andrea Hamm, Martin Schuessler, Dr Stefan Ullrich

Venue: virtual event

Participation: If you are interested in participating please contact:

Andrea Hamm and Martin Schuessler, supported by Dr Stefan Ullrich, will present the program of the BIFOLD Ethics in Residence’s Summer School, which will take place from 20.-24. June in a hotel around Berlin and at the Weizenbaum Institute. The Summer School complements and serves the technological research on artificial intelligence (AI) within the AI Competence Centres with aspects of ethics, explainability, and sustainability. It is organized within the Ethics in Residence program, which is realized between the Weizenbaum Institute for the Networked Society – the German Internet Institute – and the Network of the German AI Competence Centres.

The Summer School is fully funded by BIFOLD and open to all BIFOLD PhD students, and in addition those from the other centres of the German AI Competence Center Network (ML2R, MCML, TUE-AI, ScaDS, DFKI).

The program includes multiple hands-on workshops to advance the participants’ individual research projects, several high-profile international guest lectures and Q&A sessions with the guest speakers, a panel discussion, and participants’ presentation sessions for expert jury feedback. The international expert researchers and guest speakers joining have backgrounds in computing within limits, disaster research, and COVID-19 data research. In addition, the summer school offers two main tracks, one on explainable deep neural networks (XNN) and one on sustainable AI (SAI), for a more specialized training of the PhD students.

About the presenters & track leaders
Andrea Hamm
Copyright: private

Andrea Hamm is a doctoral researcher at the Weizenbaum Institute for the Networked Society and TU Berlin. In her dissertation, she investigates civic technologies for environmental monitoring in the context of making cities and communities more sustainable. She is particularly interested in the real-world effects of AI technologies, for instance to understand how AI-supported simulations contribute to reducing CO2 footprints and material consumption. She has published at international venues such as at ACM CHI Conference on Human Factors in Computing Systems, ACM ICT for Sustainability Conference, and the International Conference on Wirtschaftsinformatik WI. Her work focuses on the interdisciplinary transition from human-computer interaction (HCI) and design studies to communication studies. She is a member of the German-Japanese Society for Social Sciences and the AI Climate Change Center Berlin-Brandenburg. In 2019, she was a guest researcher at Aarhus University, Denmark, after previously studying at Freie Universität Berlin (Germany), Vrije Universiteit Brussel (Belgium), and Université Catholique de Lille (France).

Martin Schuessler is a tech-savvy, interdisciplinary human-computer interaction researcher with the belief that it is the purpose of technology to enhance people’s lives. As an undergrad, he followed this belief from a technical perspective by investigating the usability of new interaction paradigms such as tangible and spatial interfaces at the OvGU Magdeburg and Telekom Innovation Labs Berlin. As a PhD student at the interdisciplinary Weizenbaum Institute he adopted a broader often less technical perspective on the same belief (still involving a lot of programming). His dissertation work looks at ways to make computer vision systems more intelligible for users. Martin has been a visiting researcher at the University College of London Interaction Center and the Heidelberg Collaboratory for Image Processing. He has published articles at top international conference on Learning Representations (ICLR), Human Factors in Computing Systems (CHI), Computer-Supported Cooperative Work and Social Computing (CSCW) and Intelligent User Interfaces (IUI).

Martin Schüssler
Copyright: private

The Summer School as a whole and as a part of the BIFOLD Ethics in Residence Program is fostered by Dr. Stefan Ullrich.

BIFOLD Summer School

Home >

BIFOLD Summer School

Ethics in Machine Learning & Data Management

The BIFOLD summer school will take place from 20-24 June 2022 at a Hotel near Berlin. It will focus on the latest ethical considerations in machine learning and data management by offering lectures and workshops on two main tracks. The school is designed for doctoral students of the BMBF‘s network of AI competence centres and organized by the BIFOLD Graduate School in collaboration with the Ethics in Residence Program with researchers of the Weizenbaum Institute for the Networked Society – the German Internet Institute.

Sufficient time for hands-on workshops and individual feedback is included.
C: StockSnap/pixabay

The summer school complements technological research on artificial intelligence (AI) within the AI competence centres with ethical aspects of explainability and sustainability. It is part of the Ethics in Residence program. The program includes multiple hands-on workshops to advance individual research projects, several guest lectures including Q&A, a panel discussion, and Ph.D. student presentation sessions with expert jury feedback. The summer school offers two tracks on explainable deep neural networks (XNN) and sustainable AI (SAI) for more specialized training of the doctoral students. All of BIFOLD’s PhD students are invited to participate. In addition BIFOLD offers places for the PhD students of the German AI competence centre network (ML2R, MCML, TUE-AI, ScaDS, DFKI).

Invited international experts

International expert researchers with backgrounds in computing within limits, disaster research, and COVID-19 data research are joining the summer school as speakers and are reachable for individual feedback.

Daniel Pargman, Ph.D.
KTH Royal Institute of Technology, Stockholm, Sweden

Teresa Cerratto Pargman, Ph.D.
Stockholm University, Sweden

Yuya Shibuya, Ph.D.
The University of Tokyo, Japan

Raphael Sonabend, Ph.D.
Imperial College London, UK & University of Kaiserslautern, Germany

Alan Borning, Ph.D. & Lance Bannett, Ph.D.
University of Washington, Seattle, USA

Enrico Costanza, Ph.D.
University College London, UK

Focus tracks

Track XNN focuses on evaluating interpretable machine learning to provide students with the ability to empirically validate claims about interpretability:

  • Critical review of XAI methods: taxonomies of XAI approaches, review of explanation goals, user benefits and current results from user studies
  • Rigorous methods for validating explanation methods with users: interdisciplinary methodological training, suitable evaluation datasets, user tasks and study designs, participant recruitment, validity, and reproducibility considerations

Track SAI emphasizes on ecological and socio-political aspects of AI to understand how AI and data can contribute to action in the name of sustainability transition:

  • Sustainability in research and policies: What is sustainability? United Nations SDGs, COVID-19, critical thinking on AI, environmental monitoring, sustainable smart cities and communities
  • SAI approaches and methods: data feminism, digital civics, computing within limits, citizen science, social media data, and mixed methods.
Organisational details

Pandemic conditions permitting, the school will be held as an on-site event at a lakeside hotel near Berlin, allowing time and space for outside sports and relaxing. Participants are expected to attend the entire program, arrival from 19. June, departure 25. June 2022. There is no tuition fee.
Please get in touch with us in case you need child care.

Program venue: Weizenbaum Institute for the Networked Society
Accomodation: Hampton by Hilton Berlin City West, in walking distance to Weizenbaum & TU Berlin, in the heart of Berlin, with the nearby Tiergarten Park providing plenty of greenery  


Please send one pdf-file, including your CV, an abstract of your (preliminary) Ph.D. project, and a short motivation message describing why you would like to participate and what you would like to learn during the summer school, to
Application deadline is 30. April 2022.


Andrea Hamm
Doctoral Researcher, Research Group “Responsibility and the Internet of Things”, Weizenbaum Institute for the Networked Society & TU Berlin, Germany

Martin Schuessler
Doctoral Researcher, Research Group “Criticality of AI-based Systems”, Weizenbaum Institute for the Networked Society & TU Berlin, Germany

Dr. Stefan Ullrich
Weizenbaum Institute for the Networked Society, Berlin, Germany

Prof. Dr. Volker Markl
Co-Director BIFOLD

Prof. Dr. Klaus-Robert Müller
Co-Director BIFOLD

Dr. Tina Schwabe, Dr. Manon Grube
Coordinators of the BIFOLD Graduate School

Intelligent Machines Also Need Control

Home >

Intelligent Machines Also Need Control

Intelligent Machines Also Need Control

Dr. Marina Höhne, BIFOLD Junior Fellow, researches explainable artificial intelligence funded by the German Federal Ministry of Education and Research.

Happy to establish her own research group: Dr. Marina Höhne. (Copyright: Christian Kielmann)

For most people, the words mathematics, physics and programming in a single sentence would be reason enough to discreetly but swiftly change the subject. Not so for Dr. Marina Höhne, postdoctoral researcher at TU Berlin’s Machine Learning Group led by Professor Dr. Klaus-Robert Müller, as well as Junior Fellow at the Berlin Institute for the Foundations of Learning and Data (BIFOLD) and passionate mathematician. Since February 2020, the 34-year-old mother of a four-year-old son has been leading her own research group, Understandable Machine Intelligence (UMI Lab), funded by the Federal Ministry of Education and Research (BMBF).

In 2019, the BMBF published the call “Förderung von KI-Nachwuchswissenschaftlerinnen” which aims at increasing the number of qualified women in AI research in Germany and strengthening the influence of female researchers in this area long-term.
“The timing of the call was not ideal for me, as it came more or less right after one year of parental leave,” Höhne recalls. Nevertheless, she went ahead and submitted a detailed research proposal, which was approved. She was awarded two million euros funding over a period of four years, a sum comparable to a prestigious ERC Consolidator Grant. “For me, this came as an unexpected but wonderful opportunity to gain experience in organizing and leading research.”

A holistic understanding of AI models is needed

The topic of her research is explainable artificial intelligence (XAI). “My team focuses on different aspects of understanding AI models and their decisions. A good example of this is image recognition. Although it is now possible to identify the relevant areas in an image that contribute significantly to an AI system’s decision, i.e. whether the nose or the ear of a dog was influential in the model’s classification of the animal, there is still no single method that conclusively provides a holistic understanding of an AI model’s behavior. However, in order to be able to use AI models reliably in areas such as medicine or autonomous driving, where safety is important, we need transparent models. We need to know how the model behaves before we use it to minimize the risk of misbehavior,” says Marina Höhne outlining her research approach. Among other things, she and her research team developed explainable methods that use so-called Bayesian neural networks to obtain information about the uncertainties of decisions made by an AI system and then present this information in a way that is understandable for humans.

To achieve this, many different AI models are generated, each of which provides decisions based on slightly different parameterizations. All of these models are explained separately and subsequently pooled and displayed in a heat-map. Applied to image recognition, this means that the pixels of an image that contributed significantly to the decision of what it depicts, cat or dog, are strongly marked. The pixels that are only used by some models in reaching their decision, by contrast, are more faintly marked.

“Our findings could prove particularly useful in the area of diagnostics. For example, explanations with a high model certainty could help to identify tissue regions with the highest probability of cancer, speeding up diagnosis. Explanations with high model uncertainty, on the other hand, could be used for AI-based screening applications to reduce the risk of overlooking important information in a diagnostic process,” says Höhne.

Standup meeting: Each group member only has a few minutes to explain his or her scientific results.
(Copyright: Christian Kielmann)

Today, the team consists of three doctoral researchers and four student assistants. Marina Höhne, who in addition is associated professor at the University of Tromsø in Norway, explains that the hiring process of the team came with problems of a very particular nature: “My aim is to develop a diverse and heterogeneous team, partly to address the pronounced gender imbalance in machine learning. My job posting for the three PhD positions received twenty applications, all from men. At first, I was at a loss of what to do. Then I posted the jobs on Twitter to reach out to qualified women candidates. I’m still amazed at the response – around 70,000 people read this tweet and it was retweeted many times, so that in the end I had a diverse and qualified pool of applicants to choose from,” Höhne recalls. She finally appointed two women and one man. Höhne knows all about how difficult it can still be for women to combine career and family. At the time of her doctoral defense, she was nine-months pregnant and recalls: “I had been wrestling for some time with the decision to either take a break or complete my doctorate. In the end, I decided on the latter.” Her decision proved a good one as she completed her doctorate with “summa cum laude” while also increasing her awareness of the issue of gender parity in academia.

Understandable AI combined with exciting applications

Höhne already knew which path she wanted to pursue at the start of her master’s program in Technomathematics. “I was immediately won over by Klaus-Robert Müller’s lecture on machine learning,” she recalls. She began working in the group as a student assistant during her master’s program, making a seamless transition to her doctorate. “I did my doctorate through an industry cooperation with the Otto Bock company, working first in Vienna for two years and then at TU Berlin. One of the areas I focused on was developing an algorithm to make it easier for prosthesis users to adjust quickly and effectively to motion sequences after each new fitting,” says Höhne.  After the enriching experience of working directly with patients, she returned to more foundational research on machine learning at TU Berlin. “Understandable artificial intelligence, combined with exciting applications such as medical diagnostics and climate research – that is my passion. When I am seated in front of my programs and formulas, then it’s like I am in a tunnel – I don’t see or hear anything else.”

Marina Höhne has a passion for math. (Copyright: Christian Kielmann)
More information:

Dr. Marina Höhne
Understandable Machine Intelligence Lab (UMI)

Making the role of AI in Medicine explainable

Home >

Making the role of AI in Medicine explainable

Making the role of AI in Medicine explainable

Analysis system for the diagnosis of breast cancer

Researchers at TU Berlin and Charité – Universitätsmedizin Berlin as well as the University of Oslo have developed a new tissue-section analysis system for diagnosing breast cancer based on artificial intelligence (AI). Two further developments make this system unique: For the first time, morphological, molecular and histological data are integrated in a single analysis. Secondly, the system provides a clarification of the AI decision process in the form of heatmaps. Pixel by pixel, these heatmaps show which visual information influenced the AI decision process and to what extent, thus enabling doctors to understand and assess the plausibility of the results of the AI analysis. This represents a decisive and essential step forward for the future regular use of AI systems in hospitals. The results of this research have now been published in Nature Machine Intelligence.

Cancer treatment is increasingly concerned with the molecular characterization of tumor tissue samples. Studies are conducted to determine whether and/or how the DNA has changed in the tumor tissue as well as the gene and protein expression in the tissue sample. At the same time, researchers are becoming increasingly aware that cancer progression is closely related to intercellular cross-talk and the interaction of neoplastic cells with the surrounding tissue – including the immune system.

Image data provide high spatial detail

Although microscopic techniques enable biological processes to be studied with high spatial detail, they only permit a limited measurement of molecular markers. These are rather determined using proteins or DNA taken from tissue. As a result, spatial detail is not possible and the relationship between these markers and the microscopic structures is typically unclear. “We know that in the case of breast cancer, the number of immigrated immune cells, known as lymphocytes, in tumor tissue has an influence on the patient’s prognosis. There are also discussions as to whether this number has a predictive value – in other words if it enables us to say how effective a particular therapy is,” says Professor Dr. Frederick Klauschen from the Institute of Pathology at the Charité.

“The problem we have is the following: We have good and reliable molecular data and we have good histological data with high spatial detail. What we don’t have as yet is the decisive link between imaging data and high-dimensional molecular data,” adds Professor Dr. Klaus-Robert Müller, professor of machine learning at TU Berlin. Both researchers have been working together for a number of years now at the national AI center of excellence the Berlin Institute for the Foundations of Learning and Data (BIFOLD) located at TU Berlin.

Missing link between molecular and histological data

Determination of tumor-infiltrating lymphocytes (TiLs) using Explainable AI technology. Histological preparation of a breast carcinoma.
(© Frederick Klauschen)

Result of the AI process shows a so-called heatmap, which highlights the TiLs in red. Other tissue/cells: blue and green.
(© Frederick Klauschen)

It is precisely this symbiosis which the newly published approach makes possible. “Our system facilitates the detection of pathological alterations in microscopic images. Parallel to this, we are able to provide precise heatmap visualizations showing which pixel in the microscopic image contributed to the diagnostic algorithm and to what extent,” explains Müller. The research team has also succeeded in significantly further developing this process: “Our analysis system has been trained using machine learning processes so that it can also predict various molecular characteristics, including the condition of the DNA, the gene expression as well as the protein expression in specific areas of the tissue, on the basis of the histological images.

Next on the agenda are certification and further clinical validations – including tests in tumor routine diagnostics. However, Frederick Klauschen is already convinced of the value of the research: “The methods we have developed will make it possible in the future to make histopathological tumor diagnostics more precise, more standardized and qualitatively better.”


Morphological and molecular breast cancer profiling through explainable machine learning, Nature Machine Intelligence

Further information can be obtained from:

Prof. Dr. Klaus-Robert Müller
TU Berlin
Maschinelles Lernen
Tel.: 030 314 78621

Prof. Dr. Frederick Klauschen
Charité – Universitätsmedizin Berlin
Institut für Pathologie
Tel.: 030 450 536 053

BIFOLD Fellow Dr. Wojciech Samek heads newly established AI research department at Fraunhofer HHI

Home >

BIFOLD Fellow Dr. Wojciech Samek heads newly established AI research department at Fraunhofer HHI

BIFOLD Fellow Dr. Wojciech Samek heads newly established AI research department at Fraunhofer HHI

Dr. Samek (l.) and Prof. Müller in front of an XAI demonstrator at Fraunhofer HHI. (Copyright: TU Berlin/Christian Kielmann)

The Fraunhofer Heinrich Hertz Institute (HHI) has established a new research department dedicated to “Artificial Intelligence”. The AI expert and BIFOLD Fellow Dr. Wojciech Samek, previously leading the research group “Machine Learning” at Fraunhofer HHI, will head the new department. With this move Fraunhofer HHI aims at expanding the transfer of its AI research on topics such as Explainable AI and neural network compression to the industry.

Dr. Wojciech Samek: “The mission of our newly founded department is to make today’s AI truly trustable and in all aspects practicable. To achieve this, we will very closely collaborate with BIFOLD in order to overcome the limitations of current deep learning models regarding explainability, reliability and efficiency.“

“Congratulations, I look forward to a continued successful teamwork with BIFOLD fellow Wojciech Samek, who is a true AI hot shot.”

BIFOLD Director Prof. Dr. Klaus-Robert Müller

The new department further strengthens the already existing close connection between basic AI research at BIFOLD and applied research at Fraunhofer HHI and is a valuable addition to the dynamic AI ecosystem in Berlin.

“The large Berlin innovation network centered around BIFOLD is unique in Germany. This ensures that the latest research results will find their way into business, science and society.”

BIFOLD Director Prof. Dr. Volker Markl