Researchers from the ORCA (Offshore Robotics for the Certification of Assets) Hub, a consortium of five universities led by Heriot-Watt and Edinburgh universities, have unveiled a new method of communication that allows machines and humans to speak the same language and understand each other’s actions in real time.

‘Building appropriate trust in offshore autonomous robots is essential’ says scientists

Named MIRIAM (Multimodal Intelligent inteRactIon for Autonomous systeMs), the new method of communication is a significant breakthrough that enables people to work with autonomous robots. It allows users to ask robots questions, understand their actions and receive mission status through a digital twin¹. MIRIAM uses natural language, allowing users to speak or text queries and receive clear explanations about the robot’s behaviour in an intuitive way.

By improving how robots communicate, MIRIAM will help to build the confidence of those who will be using them and overcome any adoption barriers in the workplace.

MIRIAM will be used by a new collaborative team that sees Heriot-Watt University, through the ORCA Hub, working and integrating its research with the expertise of Phusion, an engineering software firm and Merkle Aquila, a data science firm. They are the latest additions to global energy partner Total’s multimillion-pound research consortium that is working to develop autonomous capabilities with multiple robotic systems for work on offshore energy installations. MIRIAM will initially work with a tracked robot that will be deployed at Total’s Shetland Gas Plant.

Kris Kydd, Head of Robotics at Total E&P UK said:

“Total believes that robots have a huge amount to offer our industry. We are pioneering their use on oil and gas sites and this collaboration is helping us to make significant progress. This announcement is the first important step towards making the actions of autonomous robots clear and understandable, and this will allow them to gain the confidence of the wider workforce.  Robots offer immediate advantages such as increased safety and efficiency.  In the long-term, they offer us new ways of working and are limited only by our imagination.”

Helen Hastie is a professor of computer science at Heriot-Watt University and an ORCA work package lead. She explains:

“When robots have an autonomous element, it means that they sense the environment and can make certain decisions by themselves. However, there is currently a communication barrier between robots and their human supervisors in relation to the reasoning behind system actions. This is particularly problematic in remote, highly challenging, and hazardous environments such as offshore, which can involve multiple vehicles and platforms.

“This lack of transparency can lead to a reduction in situation awareness and therefore user-confidence in the robots, hindering the human-robotic team and resulting in unnecessary aborts or laborious manual manipulation of the robot. It is therefore essential that a supervisor understands what the robot is doing and why.

“MIRIAM’s technology is designed to provide invaluable transparency in this process. Our team of researchers have taken a user-centred design approach, which supports Total’s forward-thinking efforts to making operations safer offshore. We have also modified MIRIAM to the specifics of Total’s robotics programme.”

Phusion IM is supporting the project with the implementation of a Master Data Model that will enable the framework for a digital twin of Total’s facilities. This will allow the creation of autonomous routines, enabling Total’s robot to undertake maintenance checks and feedback results in a consistent manner to the digital twin – creating a single source of information for the operating asset.

MIRIAM will allow humans to communicate with this digital twin, and the data held in it, including the robot, via text or speech. This means supervisors can ask where the robot is going, what it plans to do next and why, and query data it has gathered during its missions.

Steve Pearson, Chairman of Phusion IM, said:

“This ground-breaking project demonstrates the importance of high-quality, accessible engineering data as an enabler of digital transformation.  Through our Master Data Model [digital twin] developed for Total, the robots are able to develop capabilities through machine learning and deliver the ultimate benefits of efficiency and operational improvements alongside workforce safety.”

Merkle Aquila is the industrial data science, data engineering and agile delivery collaborative partner. The company has been working with the team to develop solutions using AI to extract maximum value from robotic image data and demonstrate the automation of manual human inspection using computer vision algorithms.

Sean Robertson, Total Client Director from Merkle Aquila, said:

“We have been able to develop working solutions quickly using agile delivery methods, algorithms developed on the Microsoft cloud, tactical data collection and close team collaboration. We have demonstrated leading innovation in tackling these barriers to develop a robust and adaptable solution.”

The project will be used by the original OGRIP (Offshore Ground Robotics Industrial Pilot) robot developed by Total E&P UK, Austrian tech firm Taurob and the Aberdeen-based Oil and Gas Technology Centre (OGTC). OGRIP is designed to support exploration and production operations in increasingly harsh and challenging conditions, including extreme cold, arid climates and isolated locations.