FLUENTLY: The essence of human-robot interaction

The FLUENTLY project aims to develop a new AI-based device that, once worn by workers, will enable industrial robots to interpret human language, gestures and emotions via a range of sensors and then convert them into executable instructions to improve workers’ well-being. Project coordinator Oliver Avram and the project team are redefining the way humans and robots interact in the workplace, paving the way for a future of seamless collaboration and enhanced productivity.

Avram, with his passion for robotics and a keen eye for innovation, is a huge driving force behind the FLUENTLY Project. His vision is clear: to bridge the gap between humans and robots, enabling them to work together harmoniously in various industries. “We’re targeting not only roboticists or AI developers but people with different backgrounds who want to work with robots. The goal is to redefine training for human-robot collaboration.”

The FLUENTLY project recognises the diverse challenges faced by humans in today’s industrial landscape, from risky operations to repetitive tasks. Avram explains, “There are various industrial fields where the physical and cognitive capabilities of humans are really pushed. We need more support from robots in these tasks. To really make this work efficiently though, we need mutual training. Humans and robots need to learn together, inspired by human teamwork. It’s about developing empathy and adaptability in robotic assistance. AI is hugely helpful for this, enhancing the robot’s cognitive abilities and enabling it to recognise operational context and suggest alternatives.”

Collaborative training in the Robo-Gym

Central to FLUENTLY is the concept of the Robo-Gym, a physical infrastructure based in SUPSI where humans and robots undergo joint training on specific applications. Avram envisions this as a transformative space where, with a helping hand from AI, humans and robots learn to bond and build a personalised collaborative process. “The idea is to improve the well-being of humans without sacrificing productivity. Human and robot must complete tasks together, adapting to each other’s needs.

The Robo-Gym will be the first and largest European hub dedicated to human-robot interaction for industrial applications. It serves two complementary purposes: on one side it is a training centre for FLUENTLY adopters, while on the other it is a training facility for the machine learning models at the core of FLUENTLY’s personalised, ever-evolving collaborative intelligence. The Robo-Gym activities are performed by trainers assisting users during the setup of equipment and with initial interactions with FLUENTLY, like effective teaching of handling movements and configuration of process parameters. A back-office prepares software setups, chooses appropriate AI methods to be associated with new tasks, dynamically updates training schedules based on results, and performs continuous improvements of the overall FLUENTLY platform.

FLUENTLY differentiate itself from traditional approaches to human-robot collaboration through its focus on personalised interactions and AI integration. “We propose five days of training in the Robo-Gym, with gradually increasing task complexity,” explains Avram. “The first two days are more about building trust in the human, increasing their readiness to work with the robot by showing them how they work and behave. This is followed by a second phase in the remaining three days in which the human and robot develop a working relationship made up of made up of individualised interactions crafted to address specific preferences, behaviours and needs.”

These sessions in the Robo-Gym are also hugely important for the AI to adapt to the specific industrial setting it will be working in. “We want the users to be able to speak freely to the robot in the same way they would to a human co-worker, but for this to work, the jargon of that specific industry must be taught to the AI, or the natural language recognition models will struggle,” explains Avram.

Improving battery disassembly

Alongside the development of the Robo-Gym, the project has targeted three large scale industrial value chains as use cases for its new technologies and ways of working. One of the use cases takes place in the context of disassembling e-bike battery packs. With the rise of electromobility, there’s an impending need to address the vast number of batteries reaching their end of life. Current disassembly methods are somewhat inefficient, involving energy-intensive shredding or labour-intensive manual processes.

The FLUENTLY project offers a sustainable solution by enabling targeted recovery of materials from batteries without mixing or damaging them, promoting circular economy principles. By automating routine tasks like unscrewing and integrating human expertise for troubleshooting and adaptable decision-making, the project aims to enhance safety, efficiency, and flexibility in the disassembly process.

The project’s innovative approach emphasises personalised interactions between humans and robots, facilitated by AI-driven models. “An important dimension here is that the human doesn’t always necessarily have to explicitly ask for help,” says Avram. “The AI models can detect that the human might be stressed or fatigued, and can suggest that they take a break or replan the workload. These kinds of interactions mean that when the person involved starts working at a sub-optimal level, perhaps towards the end of the day, the system can detect it and help to pick up the slack, or even play some music to boost the person’s mood.”

Streamlining additive manufacturing

A second use case in the project expands its scope to additive manufacturing, specifically focusing on the repair of high-value parts such as impellers and turbine blades. These parts, crafted from expensive materials like nickel alloys, endure harsh operating conditions, making their repair economically and technically challenging. FLUENTLY addresses this challenge by streamlining the preparatory steps before actual repair, involving intricate reverse engineering and defect analysis. Equipped with a 3D scanner and specialised software, FLUENTLY’s robotic station scans damaged parts, reconstructs their geometry, and generates repair strategies, laying the groundwork for subsequent additive manufacturing processes. By automating these laborious and precise tasks, FLUENTLY aims to enhance efficiency and consistency in the additive manufacturing workflow, ultimately reducing the time and effort required for repair preparation.

FLUENTLY leverages collaborative robots equipped with scanning capabilities and voice-guided programming to streamline the preparatory operations for additive manufacturing repairs. The project seeks to minimise manual intervention and enhance user-friendliness by replacing traditional teach pendants with voice commands and kinesthetic teaching methods. By integrating voice-controlled interactions with scanning software and PC interfaces,  FLUENTLY aims to revolutionise the programming and operation of robotic systems, improving operational efficiency but also democratising access to advanced manufacturing technologies by simplifying user interactions and reducing the learning curve associated with complex repair processes.

Enhancing aerospace assembly

The third use case within the FLUENTLY project targets the aerospace industry, specifically focusing on the assembly of engine nacelles, particularly the inlet cowl, which forms the top part of an aircraft engine. This assembly process involves handling large parts, some up to 1.6 meters and beyond in diameter, and requires meticulous attention to detail due to the iterative and repetitive nature of the tasks involved. For example, fixing over 200 fasteners onto the lip section of each nacelle is a time-consuming and labour-intensive process. FLUENTLY aims to alleviate these challenges by introducing robotic support to assist human workers in routine tasks such as fastener insertion and collaborative transportation of components during assembly. Through flexible workflow definition and real-time adaptation to human movements using speech, gesture and activity recognition FLUENTLY optimises productivity and ensures a seamless human-robot collaboration in the assembly process.

FLUENTLY’s approach emphasises enhancing efficiency and ergonomics in aerospace assembly tasks by integrating robotic assistance with human expertise. By distributing tasks between robots and humans based on productivity constraints and adapting work pace accordingly, FLUENTLY fosters a symbiotic relationship between automation and human labour.

Moreover, FLUENTLY enables smooth interactions between humans and robots through voice commands and collaborative handling of parts, allowing for agile and intuitive assembly processes tailored to the specific needs of each task. This collaborative framework not only improves productivity but also enhances safety and ergonomics, ultimately advancing the efficiency and quality of aerospace assembly operations.

A new era of human-robot collaboration

Avram believes the Robo-Gym will be central to unlocking the potential of the robot-human collaboration the project it is developing, not only in its role in developing the specific working relationships via AI, but also as a demonstration location where people can find out first-hand what is made possible by this new way of working. “We have had a lot of students interested in robotics come and learn in the Robo-Gym, and they have relished their chance to interact with the robots in this way. You can see that it sparks their enthusiasm and is, in a way, quite inspirational for them.

“On the other hand, I think this structure is also open to high-level management from companies, providing an unparalleled opportunity for them to see what can be achieved through this technology. It demonstrates very quickly that what we have done bridges the gap between robots and humans in a way that now makes this kind of technology much more approachable from a business perspective, and I think that is hugely valuable for the field moving forwards.”

Indeed, the FLUENTLY project holds immense potential in the context of Industry 4.0, where automation and robotics are reshaping traditional workflows. By fostering a culture of collaboration and innovation, it paves the way for a future where humans and robots work hand in hand, driving productivity and efficiency to new heights.