About intelligent machines and smart people

At least since DeepMind’s 2016 victory over Lee Sedol in the strategy game Go, the notion that nothing can come close to human intelligence, creativity and intuition has been deeply shaken. AI (artificial intelligence) systems can perceive, adapt to, and learn from the environment without human intervention. Will robots and AI steal our jobs in the future? Or can human limitations and inadequacies be overcome with their help? Are we in for a paradigmatic shift enabled by the new cognitive capabilities of AI systems and, in particular, their ability to learn? 

These questions are not easy to answer and, above all, cannot be answered once and for all. The closest answer is likely to be “yes, but”. In this chapter, I outline my “yes, but” answer for algorithmic decision-making systems in which decisions are partially or fully automated in the hope of making faster, better, and more objective decisions.

The idea of having smart machines to help us make decisions is not new. People have been using data and model-based systems to support complex and difficult decision-making for about 50 years. However, with data-driven AI methods, the field of application has expanded to include even simple and common everyday decisions, where they either assist us by pre-selecting suitable alternatives or they make and execute decisions entirely for us. In many applications, we are not even aware that algorithmic decision-making systems (ADS) are operating in the background, blurring the boundaries between automated decision-making and decision support. The paradigmatic shift triggered by technological progress is based, on the one hand, on the increasing autonomy and the resulting capabilities of such systems and, on the other hand, on their widespread but often non-transparent use. 

Possible applications of algorithmic decision-making systems are manifold and diverse, and so are the reasons for them and their effects. For example, recommender systems (e.g. search engines) help to pre-select decision alternatives, pattern recognition systems (e.g. in medical diagnostics) reduce complexity, predictive analytics systems (e.g. prediction of creditworthiness) reduce uncertainty and risk, and assistance systems (e.g. automatic brake assistants) can be used to reduce human errors of judgment. 

Are the high expectations for these systems at the current state of AI technology even justified? So far, research has focused primarily on analyzing the impact of algorithmic decisions on affected individuals – and has already tempered the high expectations. It turns out that algorithmic decisions also have similar problems to human decisions due to biased or incomplete data, inadequate modeling, and problematic objectives. After all, AI systems are “only” developed by those selfsame error-prone humans that we want to replace – an irony of automation. In this chapter, however, I would like to shed light on another aspect that has received little public attention: What is the impact of ADS in the context of work when humans and algorithms (have to) solve tasks together? How does this affect their own self-image and perception of their role, and who ultimately bears responsibility for decisions?

Let’s start from the beginning. Indeed, under laboratory conditions, the combination of the complementary capabilities of humans and ADS can be shown to improve decision quality. While humans are needed to select and develop decision models, set objectives, and interpret the decision context, algorithms can analyze incredibly large amounts of data in a very short time and identify relationships and patterns. This requires a well-designed interface between humans and ADS.

When ADS are used in the context of work, sociotechnical systems are created in which people and machines perform tasks together. In this context, ADS are given the authority to act, which comes with a certain degree of autonomy. In addition, people make assumptions about the capabilities and competencies of ADS and develop certain expectations of them. For people, this not only changes their perception of their own role, but also their perception of their own abilities (compared to ADS) and self-efficacy. Finally, there is also a change in the attribution of responsibility for the work process and for the result.

The British TV comedy show Little Britain satirizes these changes: In this sketch, a mother comes to the hospital with her five-year-old daughter to make an appointment for tonsil surgery. After the receptionist enters the daughter’s information, she says the child is scheduled for bilateral hip surgery. Despite the mother’s objections, which the receptionist initially types into her computer, she keeps answering: “The computer says no!” Regardless of how reasonable the mother’s objections are and how wrong the computer’s statements are, the machine’s proposed decision ultimately prevails. The satire highlights how supposedly “intelligent systems” can absurdly shift the roles and responsibilities of humans and machines. Here we can see, that we are obviously placing too much trust into algorithmic machines. This weakens our autonomy and self-efficacy and eventually a diffusion of responsibility occurs. The critical questions are: Why does the receptionist rely on the faulty system without critically questioning it, and how could we avoid such situations in the context of work?

Mediation instead of decision

Practical experience and scientific studies confirm that this satire is unfortunately all too realistic and draws attention to a very critical aspect. An example of this is a case study in a Swedish public authority, in which the shifts in the role structure are clearly visible¹. This public authority uses an algorithmic decision-making system to assess the eligibility of individuals applying for government assistance benefits. Whereas employees used to assess and decide this themselves, they increasingly perceive themselves only as mediators between the system and the people making the requests. They “just keep the system running” even though they are formally responsible for the final decision. Officials express in interviews that the system proposes a decision based on all the information entered and therefore there can be no room for doubt. They are convinced that the ADS cannot be wrong. They consider the system to be highly competent, while equally the perception of their own competencies as well as the ability to act (in comparison) is perceived as lower.

Overconfidence in technology

Precisely because ADS are so complex and opaque, their decision-making processes cannot be fully understood by their developers, let alone their users. And yet, people generally have great faith in technology, even if they do not understand exactly how it works.

Overconfidence can go so far as to make people question their self-image: The result must be correct, because the analysis was made by a computer and computers draw better conclusions than humans. With arguments like these, people even justify obviously wrong classifications of themselves by the algorithmic decision-making system². However, with overconfidence, skills and competencies are lost because they are not regularly used and practiced. Decision-making processes whose rules and interrelationships are not understood also offer no learning opportunities for new experiences or insights for people. This increases our dependence on AI systems, which becomes especially problematic when they fail or do not function properly. Although it is expected in so-called out-of-loop scenarios that people can step in and take over the tasks of the systems again, they then lack the experience and skills to do so. Another irony of automation.

Appearance of neutrality

Autonomy and self-determined action require a certain degree of (independent) responsibility. Every decision an ADS makes is not made by humans and ultimately limits our power to act. This can certainly be a relief, but ultimately it also reduces our perceived control and sense of responsibility. The very fact that a decision is not made by a human being but is the result of an automated process lends it a certain neutrality and legitimacy – and thus weakens the users’ sense of responsibility. This is especially problematic because laypeople in particular place a great deal of confidence in ADS and prefer to rely on algorithms rather than the subject expertise of humans. But who bears the ultimate responsibility when something bad happens? When the ADS is wrong or when people are discriminated against by ADS?

Responsibility in a socio-technical decision-making system

Especially when using ADS that have an impact on the rights, life and health of people, the question of responsibility is highly relevant. For example, how should physicians be held responsible for diagnoses made based on misjudgments of an ADS provided to them by their employer? Whether responsibility for potential consequences of algorithmic decisions can be unambiguously assigned depends to a large extent on the transparency and explainability of the ADS used. Only those who understand how algorithmic decisions are arrived at, and can thus check and assess their plausibility, can take responsibility and be held accountable. However, given the high complexity of ADS, this overview is usually not available: Too many system components (i.e. systems within systems) interact. The responsibility is shared among many and not held by anyone in particular. Moreover, ADS cannot yet meet the requirements of moral action and responsibility. It is therefore essential to close the information asymmetry between the ADS and their users. It is not enough to have access to the data, objectives and decision model; users must also be able to comprehend and understand the decision-making process.

ADS are not objective or value-free, but AES sind nicht objektiv oder wertfrei, sondern the result of many value judgments das Ergebnis vieler Werturteile, wrapped in die in mathematische Codes und Rechenschritte mathematical codes and computational steps verpackt sind.

Algorithms sometimes make life easier, but ...

Even though ADS can make life easier, they do not always make it better. ADS are not objective or value-free, but the result of many value judgments wrapped in mathematical codes and computational steps. ADS impact on how we see ourselves by pushing us from the active role of decision maker to the passive role of facilitator. At worst, these systems limit our autonomy and self-determined lives. Because we do not understand them, they prevent us from learning from the experiences we have with them. Worse than that: We unlearn what we used to be able to do because we no longer apply our skills. Yet it is so simple: It is not the question “Where can machines replace us?” that should guide us in the development of new technologies, but rather “Where do we want to use machines?” In a complementary approach, with AI systems supporting us where work is dangerous or unhealthy, where we have weaknesses and our cognitive capacities are insufficient, they extend our strengths in creativity and innovation, our responsible actions, leadership, communication, and problem-solving, as well as our ethical decision-making.