Raffaella Aghemo, Lawyer
A paper by researchers from the University of North Carolina — specifically Gaurav Samdani, Ganesh Viswanathan and Abirami Dasu Jegadeesh — explores the tension between the human factor and artificial intelligence as a symbiosis of two effective approaches to solving complex and realistic tasks. Building on the principles of human-AI collaboration, it identifies how combined structures can improve processes such as decision-making, scalability and flexibility in sectors including healthcare, the automotive and transport industries, and education.
This paper, entitled ‘HUMAN-AI collaboration: balancing Agentic AI and autonomy in hybrid systems’, explains the rationale behind this analysis in the Introduction: ‘ In recent years, there has been a shift towards the use of human-AI hybrid systems. AI teams rely on artificial intelligence to perform complex cognitive tasks, as these hybrid systems are often considered more effective and efficient. At the heart of these hybrid systems lies a continuous and constantly reconfiguring interaction between agentic AI and autonomous AI.”
The diverse yet complementary capabilities of AI and humans must integrate effectively; however, this poses a significant challenge for collaboration between human operators and current and future AI tools, as it requires a balance between centralised decision-making within complex multi-agent systems and decentralised control, with each capable of acting independently as required within its specific environment. However, the text states: ‘Overall, experiences in this complex and highly interdisciplinary field demonstrate the need for a detailed understanding of how engineering decisions lead to shifts in the distribution of AI power between human and AI-based contributors. Furthermore, should these hybrid systems persist, the long-term scalability of the system through enduring power-sharing arrangements between humans and AI remains to be seen.’
The paper raises practical and ethical questions: «If a robot is considered a legal person and is granted legal rights, then who is to blame for any damage or loss caused by an independent decision made by the robot? According to the authors, in hybrid systems, humans remain agents endowed with free will, whilst artificial intelligence offers opportunities for learning, decision-making and proactive action to achieve its own objectives. It is clear that the operational dynamics between humanity and autonomy draw an ambiguous distinction, thus requiring a balance to achieve greater harmony. This can lead to a lack of trust, disappointment and wasted time in achieving organisational objectives. Maintaining this balance involves key factors such as:
– System transparency: Clear decision-making processes based on artificial intelligence.
– Environmental adaptability: Managing uncertainty in dynamic contexts.
– Human trust: Strengthening reliability and trust in AI.
– Control dynamics: Harmonisation between decision-making structures and the definition of human intervention.
– Time and context: The relationship between the level of AI decision-making autonomy and the urgency and complexity of decisions.»
This is known as the theory of socio-technical systems, which examines the interaction between social and technical components within systems. This theory maintains that the social and technical dimensions must constantly interact with one another to ensure the system functions efficiently. According to this theory, systems rely on human beings, as well as on processes and technological tools, to be managed and operated, and must therefore be designed and discussed simultaneously. This mutual adaptability is described as stability. Stability is achieved when human actors adapt, that is, modify their intentions, actions and cognitions in ways compatible with the predefined technology.
‘Although theoretical or conceptual representations of both active, agentic AI and autonomous action are valid, in practice they often appear to be mutually exclusive. This raises a number of ethical issues. From a privacy perspective, the greater the control an AI system exercises over its own decisions, the greater the amount of personal and agentive AI information to which it has access. Similarly, the ability to make decisions autonomously makes AI systems accountable for those decisions. Ensuring that AI systems are not aware of people’s sensitive characteristics is crucial to eliminating bias.”
The question that naturally arises is: to what extent should an AI system make its decision-making processes transparent to human staff? In practical terms, it is extremely difficult to predetermine the distribution of responsibilities. The problem of conflicting values, therefore, seems to arise from competition rather than cooperation. The ability to produce ethical outcomes can be compromised or enhanced at two levels: firstly, by the system’s operational policy, its ‘morality’; secondly, by the fundamental objectives or guidelines imparted to the system during its development.
The legal framework must address the question of who should bear responsibility in cases where a decision made by the artificial intelligence system underpinning the AI may have adverse consequences, particularly if such adverse consequences result from a division of labour between humans and AI. Where data is stored and processed, the legal framework must comply with data protection regulations; similarly, knowledge that has been learnt or inferred must be managed appropriately within the European context, in accordance with fundamental human rights, in particular the right to privacy.
In light of these considerations, the authors emphasise how crucial it is, when designing human-AI collaboration, that development is guided by users’ needs and the specific tasks they intend to perform. General design principles should reflect the purpose of the hybrid system, whether it be to expand the role and decision-making power of professionals, reduce documentation, work more efficiently, or improve the performance of a process or system. Most of these best practices are geared towards open-loop AI systems, and further research will be needed to understand how to balance these principles when the user is part of the control loop and the task is shared between humans and AI.
Training is essential: ‘Continuous training for developers and end-users of AI systems is vital for coordination and cooperation with human-AI hybrid systems. This includes training integrated into career pathways and the continuous professional development of practitioners.’
1) User-centred design: systems in which humans form part of an AI control loop require specific user-centred front-end design to ensure that users are able to moderate the system’s decisions.
2) Training on human-AI hybrid systems: ongoing training initiatives with groups of diverse professionals focus on the combined human-AI approach to resolving complex real-world scenarios and dilemmas, and enable trainers to critically examine how new AI technology is used to inform and/or make decisions.
3) Framework for end-user and public engagement: there is an established model for meaningfully engaging professionals and others in the co-design of AI-based, data-driven advisory systems.
4) Best practices for technology organizations that involve users in the development of best practices
5) Human-like AI assistance systems: ‘Human-in-the-Loop’ AI systems
In conclusion, the principles underpinning the effective design of hybrid systems must include:
– human-centred design or user-oriented design, involving user research, an understanding of users’ capabilities and limitations, iterative user testing, and the incorporation of user experience and broader ethical considerations into the design process
– Iterative testing with users, often mentioned in the design of these systems, in order to define clear objectives and design systems with functionalities that match the skills and knowledge of the user seeking to achieve those objectives
– Transparency, usefulness (hybrids must meet users’ needs and desires), inclusivity and diversity in design.
Certainly, as Father Paolo Benanti has pointed out, the future of these systems cannot be left to the ‘initiative’ of these autonomous systems; rather, measures must be put in place to ensure governance and oversight, as well as the ability for immediate human intervention, in order to prevent ‘unchecked AI-initiatives’ that would reduce humans to mere users rather than guides of the innovation taking place!