Evolution of Industrial Robots from the Perspective of the Metaverse: Integration of Virtual and Physical Realities and Human–Robot Collaboration (2024)

2.1. Development Trajectory

With continuous technological advancements, the application of industrial robots is also evolving to meet increasingly complex manufacturing demands and technical standards. The evolution of virtual–physical integration in industrial robot applications can be divided into four stages.

• Early Automation and Programming:

From the 1960s to the 1970s, with the emergence and use of the first industrial robots, the focus was on basic automation technology and programming. During this period, all operational steps were performed in the real world, with the primary goal of enabling operators to understand and master the operation of the equipment. By the late 1970s to the 1980s, applications began to include microprocessor control and more flexible programming languages, such as VAL (Variable Assembly Language). At this stage, all operational steps were also completed in the real world.

2.

Integrated Systems and Enhanced Functions:

In the 1990s, applications started focusing on the use of robots in integrated systems, involving enhanced functions such as vision recognition and intelligent sensing. During this development stage, the application of industrial robots began using offline data to recreate the actual working scenarios of robots in virtual environments [11]. Although the operation of robots in reality and virtual simulation was not synchronized in time, this method promoted the diversification and parameterized planning of applications [12]. Meanwhile, these virtual scenarios could be connected to real control systems, allowing for semi-physical simulation and debugging of the actual control systems, enhancing the interactivity and practicality of operations [13].

3.

Intelligent and Collaborative Robots:

From the early 21st century to the present, the focus has shifted to the intelligence and collaborative capabilities of industrial robots, emphasizing the integration of robot systems with AI and IoT technologies. At this stage, to overcome the inconsistency in time scales, the application of industrial robots has realized information-driven virtual scenarios that are synchronized with real systems in real time. Currently, Adriana Florescu et al. use Tecnomatix Plant Simulation(Tecnomatix/Siemens Product Lifecycle Management Software Inc.) to create a simulation model of a robotic arm system [14]; F. Gabriele Pratticò et al. use VRTS(HTC Vive Pro kit and Unity game engine) for virtual reality training with robots [15]; Fuwen Hu et al. use SOLIDWORKS for dual-arm modeling and simulation [16]; and Xi Wang, Luis Pérez, and Pengxiang Xia use Unity v2020.1 for the simulation of industrial robot working environments [17,18]. Based on real-time, accurate virtual scenarios, parallel application systems synchronized with real-time status are constructed to simulate future planning and path selection [19]. During the simulation process of parallel systems, the optimal solution is selected and connected to the actual control system through information flow, and converted into control commands to guide the physical robots to execute tasks according to the optimal strategy.

4.

Industrial Metaverse Evolution:

In the future, the industrial metaverse will be a new stage in the application of industrial robots. In this entirely virtual industrial environment, broader global collaboration and communication can be achieved, creating boundless operational and training scenarios, making production and operation geographically unrestricted, and greatly enhancing industrial efficiency and effectiveness [20]. Alberto Martínez-Gutiérrez et al. discussed enabling technologies (e.g., VR, DT, IIoT, etc.) to design a connection framework to achieve the purpose of the industrial metaverse [21]. The industrial metaverse will provide an immersive and interactive operational platform where users can perform real-time operational training and solve complex engineering problems through virtual collaboration, further promoting innovation and the application of industrial robot technology [22].

2.2. Analysis of Virtual–Physical Integration in Various Stages

The levels of intelligence and virtual–physical integration have progressively increased across the four development stages described above. Figure 1 illustrates the system characteristics, integrated technological concepts, and human–machine collaboration methods exhibited at each stage, revealing the connotations of virtual–physical integration at each stage.

The system characteristics exhibited at each stage, categorized by the degree of intelligence, are as follows:

During the Integrated Systems and Enhanced Functions stage, this system characteristic was mainly achieved through 3D modeling and virtual reality technologies. For example, using CFD (Computational Fluid Dynamics) offline modeling technology, it is possible to effectively monitor the state of smart factories and provide operational guidance [23]. By the stage of Intelligent and Collaborative Robots, with the development of augmented reality (AR) and digital twin technologies, virtual information can be superimposed on the actual physical environment, allowing operators to intuitively see the equipment status and operational guidance in the real environment, thereby improving operational efficiency and accuracy.

2.

Immersive Training and Simulation:

Initially, this system characteristic was achieved through virtual reality (VR) technology. With the help of head-mounted devices and controllers, operators could enter a fully simulated industrial environment to perform equipment operation and maintenance training. VR training has significant advantages in improving operator skills and proficiency [24]. Similarly, AR, digital twins, parallel systems, and cyber-physical systems (CPS) technologies can also realize this system characteristic, endowing it with new connotations.

3.

Real-time Simulation and Monitoring:

This system characteristic is realized through AR technology, which allows real-time monitoring of equipment operational status. Operators can test and optimize equipment behavior in a virtual environment, where the simulation environment can reflect the reactions of real equipment in real time, achieving efficient virtual–physical integration. In CPS factory modeling and simulation, real-time monitoring and feedback are used to optimize production processes [25].

4.

Production Process Simulation and Validation:

This system characteristic combines the advantages of virtual simulation and actual operations, using digital twin technology to simulate and validate production processes. Operators perform operations in a virtual environment to ensure the optimization of production processes and the effectiveness of actual operations. The application of digital twin technology in multi-robot systems enhances production line optimization and operational efficiency [26].

5.

Reverse Control:

This system characteristic is achieved through the combination of real-time data feedback and advanced algorithms, enabling reverse control of equipment. Behaviors and adjustments learned in the virtual environment can be directly applied to actual operations, enhancing adaptive and efficient operations. Real-time monitoring servers allow for real-time data feedback and reverse control of equipment, improving the system’s adaptability and efficiency [27].

6.

Human–Machine Integration and Collaboration Platform:

This system characteristic creates an immersive, fully virtual workspace where operators and team members can interact in real time with colleagues around the globe. The virtual collaboration and innovation platform allows for work and learning in an environment without geographical and physical limitations, realizing global collaborative innovation in the industrial metaverse. In industrial environments, intuitive adaptive control technologies that enhance human–machine interaction can significantly improve team collaboration efficiency and innovation capabilities [28].

The proportion of information technology in the above system characteristics continues to increase, deepening the transition from physical to virtual. The integrated technological concepts evolve from 3D modeling, VR, and AR to digital twins, parallel systems, and CPS, progressively enhancing the degree of virtual integration and supporting the industrial system through trial and error, deduction, simulation, and computation enabled by information technology.

Three-dimensional modeling is the foundational technology for realizing VR, which, along with VR and AR technologies, collectively realizes digital twins [29]. Digital twins can extend into parallel systems, and when combined with control, they form programmable CPS. The higher the degree of virtual–physical integration, the more intelligent the system characteristics supported. For instance, digital twin technology can support offline simulation and display, immersive training and simulation, real-time simulation and monitoring, and production process simulation and validation. Parallel systems further enable reverse control, and the highest degree of virtual–physical integration in CPS can provide a complete human–machine integration and collaboration platform.

As the degree of virtual–physical integration increases, user participation evolves from on-site operations in the real world to on-site operations via virtual interfaces, and ultimately to virtual operations by virtual humans in a virtual world.

3.1. Concept of Implementing Industrial Metaverse Human–Machine Collaboration via WEB

By integrating WEB, ROS, and WEBXR technologies, an industrial metaverse platform is constructed, enabling users to collaborate with industrial equipment in virtual reality, as shown in Figure 2.

The real-time motion data of the robotic arm are transmitted to the ROS through a socket connection. The ROS (Robot Operating System), serving as the core backend, is responsible for processing and managing these data to achieve real-time interactive operation of the robotic arm [34].

The ROS converts the processed data into model data and subscription messages, which are transmitted to the front-end Three.js application via WebSocket [35]. Three.js uses these data to render a 3D model of the robotic arm and display it on the webpage, allowing users to view the arm’s actions and status in real time within a browser.

Three.js integrates the WEBXR API to transmit the rendered 3D scene to VR devices. When users wear VR headsets, they can interact immersively with the 3D model in virtual reality. Operations performed by users in VR are transmitted back to Three.js through the WEBXR API, which then sends the operation data back to the ROS, achieving real-time virtual and physical interaction.

Users operate within the virtual scene using VR devices, and these operation data are fed back to the ROS in real time. The ROS adjusts the robotic arm’s actions based on the user’s operations and updates the latest status in the virtual scene via Three.js, forming a closed-loop control system.

This system has extensive application prospects in various fields, such as industrial manufacturing, remote training, and virtual experiments, providing robust technical support for the transition from Industry 4.0 to Industry 5.0.

3.2. Design Architecture of Multi-Industrial Robot Connection WEB Simulation System

This system is divided into three parts: industrial robots, the ROS, and the WEB platform, as shown in Figure 3.

• Controller and Body Part: The core of the robotic system includes the underlying hardware and controllers, consisting of physical robots, drive motors, sensors, processors, and controllers that directly control these hardware components. This framework includes processors that run control algorithms and handle robotic computation tasks, motor drivers that receive control commands to drive robot motors and execute actions, sensors that collect robotic state information such as position, speed, and environmental data, communication interfaces for data transmission between hardware components, control algorithms for robotic motion control such as path planning and posture adjustments, and interfaces linking sensors, actuators, communication modules, and other hardware components.

• ROS: The ROS is a flexible framework for robotic applications, providing service design, hardware abstraction, low-level device control, common functionality implementation, and message passing. This framework includes topics for asynchronous message passing via a publish–subscribe mechanism, services offering request–response style interactions, ActionLib for handling asynchronous server communication such as long-duration movements, a parameter server for storing and managing ROS parameters used for configuration, IO management related to hardware devices, and nodes in the ROS for information publishing, subscribing, and parameter reading [36].

• WEB Platform: As a cross-regional, cross-platform user interface, the WEB platform allows users to interact with the robotic system over the Internet. This platform addresses three key issues: the WEB interface accessible via browsers providing a graphical user interface, data transmission and communication through protocols like WebSocket for data exchange with the ROS [37], and backend processing for logical operations and data exchange, such as importing URDF models and exchanging robotic joint data. This framework includes login authentication to ensure system security and control access permissions, a real-time data display showcasing robot status such as position, speed, and fault information, a user interface for sending commands like start, stop, and path planning, and motion detection for monitoring robotic movements and providing real-time feedback.

These three parts work together to provide a complete process from user commands to robot actions. Users issue commands via the WEB platform, the ROS acts as middleware to process these commands and transmit them to the controller, and the controller executes the corresponding physical operations, controlling the robot to complete tasks. This framework allows for the rapid development and deployment of robot applications and can be flexibly adapted to different hardware and software requirements.

The system requires two-step communication to support the interaction between industrial robots and the WEB platform. The physical industrial robot connects to the ROS, and the ROS connects to the WEB platform, with the ROS used as middleware to achieve real-time monitoring and control of the physical industrial robot’s posture remotely.

3.3. Function Design and Implementation of the WEB Platform

The WEB platform can display the real-time posture of robots and provide features such as coordinate axis switching, state tracing, and multi-industrial robot loading. The combination of these functions ensures that the multi-industrial robot platform can offer an efficient, interactive, and comprehensive operating environment, thereby enhancing the skill levels of operators and increasing production efficiency.

4.1. Communication between Physical Robot and ROS

There is bidirectional communication between the physical robot and the ROS, as shown in Figure 7.

ROS nodes in the ROS parameter server receive key information about the robot, such as the state of the robotic arm, joint positions, motion trajectories, and timestamps, by subscribing to ROS topics. These nodes can include controller nodes of the physical robot, which use protocols like Controller Bridge and Sensor Driver to publish the control data and sensor data of the robot to the corresponding ROS topics [38].

After these data are published through the topics, other ROS nodes can subscribe to these topics to receive the data. For example, a processing node may load the robot’s URDF model and obtain model information, such as joint configurations, default states, and collision detection information, and coordinate transformations based on the received data. Additionally, this processing node can create a virtual environment and use technologies like ROS Bridge to customize topic information, generating a virtual robot model and user simulation interface to achieve mapping between the physical robot and the virtual model.

Users interact with the virtual model through the simulation interface, allowing them to monitor and control the physical robot mapped to the virtual model. The ROS uses algorithms like Cartesian path planning and forward/inverse kinematics based on topic information to plan the motion trajectories of the robot’s joints, ensuring that the trajectories conform to the preset thresholds of the robot model, thereby controlling the physical robot to perform actions according to the planned movements.

Through this method, real-time bidirectional communication is achieved between the virtual model in the ROS and the physical robot, effectively connecting the virtual and physical worlds and enhancing the flexibility and application range of the robot system.

One of the obstacles to implementing this solution is that some robot companies do not provide corresponding URDF models for their robots, requiring users to convert CAD models into URDF models. For example, KUKA only provides CAD models.

4.2. Communication between ROS and WEB Platform

The connection between the ROS and the WEB platform is established as shown in Figure 8.

In the ROS parameter server, the URDF model is converted into a 3D model usable by Three.js, and relevant configuration data such as robot arm parameters, rotation axis parameters, and collision information are transmitted. The system uses ROSBridge to build a communication bridge between the ROS and the WEB application, converting ROS messages into a format suitable for the WEB [39]. Then, utilizing Three.js, a JavaScript library specifically used for rendering 3D models and environments on web pages, detailed and interactive robots and their operating environments are created.

With the help of rosjs, the system can achieve basic interactions with the ROS, while ros3djs is used for 3D visualization of robots and environments [40], providing a real-time, dynamic, and interactive robot control and simulation environment on the WEB platform.

After completing these two communication steps, information exchange between the WEB page and the physical robot is realized. Through this integration, we can create an interactive, multi-user industrial robot simulation platform in the metaverse, which is not only suitable for education and training but also provides strong support for the research and application of industrial robots.

This integration method demonstrates the great potential of combining advanced WEB technology with industrial robot simulation, offering new possibilities for virtual laboratories and the planning and testing of complex tasks.

5.1. URDF Model Classes

Three.js does not directly support loading URDF models, so it requires parsing the URDF model for loading. The system introduces modules such as STLLoader and ColladaLoader in Three.js, which are used to load different types of 3D model files. STLLoader is used to load STL format files, a file format commonly used in 3D printing and computer-aided design, representing 3D models through triangular facets. ColladaLoader is used to load Collada format files, which contain information about the model’s geometry, texture, lighting, and even animations and physical effects. Additionally, the system includes custom URDF model classes such as URDFBase, URDFCollider, URDFVisual, URDFLink, URDFJoint, and URDFMimicJoint to handle different components of the model. Their respective functions are shown in Table 1.

Each class inherits from Object3D, which means they can all be operated and transformed as 3D objects in the Three.js environment.

The URDFBase class provides common properties and methods for all other URDF-related classes. The functions of URDFCollider, URDFVisual, and URDFLink classes are mainly implemented by inheriting the methods of Object3D. The URDFJoint and URDFMimicJoint classes extend the functionalities of the base class, adding properties and methods to control joint behaviors. The URDFRobot class acts as a top-level container, managing all links, joints, and other components, and provides methods for obtaining and setting joint values.

5.2. URDF Model Conversion

The process of converting a URDF model into the URDFRobot’s visual and collision model framework is shown in Figure 9. The system parses the URDF file, which defines the robot’s structure and properties in XML format. Next, the system loads the STL or Collada models referenced by the URDF file, which describe the geometry of each robot component in detail. Based on these geometric data, the system creates Visual objects and Collider objects, used for rendering the robot’s appearance and performing collision detection in physical simulations, respectively.

Then, the system constructs the URDFRobot object, which contains all the robot’s parts and their relationships. To simulate the connections and movements between the robot’s parts, the system creates URDFLink and URDFJoint objects, which reflect the physical properties of the links and the motion constraints of the joints. The system also processes and generates the data needed for the interactive interface and analyzes the motion range of the robot joints to ensure accurate reflection of the robot’s motion limits in the simulation.

Finally, the entire robot model is converted and exported in a Three.js-compatible format, allowing the robot model to be rendered on the WEB platform using the Three.js library, and thereby providing users with a visual operation and interaction experience.

5.3. WEB Interaction of Robot Model

After converting the URDF model to the Three.js format, the next step is to add the instance of the converted URDFRobot class to the Three.js 3D scene. Then, using the Three.js rendering engine, the robot model is rendered into the ‘<canvas>’ element on the webpage, providing users with a visual interactive interface.

Additionally, by using the ‘setJointValue’ method of the URDFJoint class, the system can receive and apply the position data of the robotic arm from the ROS, updating these data in real time to the joints of the robot model. This allows the dynamic state of the industrial robot to be reflected on the webpage in real time, enabling users to intuitively observe every movement of the robot during task execution.

6.1. Industrial Robot Simulation

This system utilizes the ABB IRB2400 robot(ABB Ltd.:) in RobotStudio to simulate a physical industrial robot. RobotStudio, developed by ABB Robotics, can fully simulate all the performances of a physical industrial robot. After loading the ABB robot action code using the teach pendant, the robotic arm starts executing the actions corresponding to the code. The overall effect is shown in Figure 10.

6.2. Industrial Robot Connection to ROS

In the Ubuntu 18.04 operating system, enter the following command line: roslaunch abb_irb2400_moveit_config moveit_planning_execution.launch sim: = false robot_ip: = xxx.xxx.xxx.xxx, where ‘xxx.xxx.xxx.xxx’ is the IP address of the specific robot to be connected. After a successful connection, a connection success message will be displayed on the RobotStudio teach pendant.

In this step, the ROS provides a communication interface for data exchange between the robot and the computer. By invoking drivers, the ROS sends control commands to the robot’s motors and actuators and receives the robot’s status information.

6.3. ROS Connection to WEB

Load the ROSlib library in the WEB. ROSlib is a JavaScript client library that enables communication with the ROS in the browser. By using ROSlib, a new ‘ROS’ object can be created, which will connect to the server running the ROS via the WebSocket protocol:

var ros = new ROSLIB.Ros({url: ‘ws://<your-ros-bridge-server>:<port>’});

Enter the IP address of the ROS to connect to the robot already in communication with the ROS. Next, create a new ‘Topic’ object for the robot’s status information. It subscribes to the corresponding topic in the ROS as a listener to receive real-time data from the robot, as shown in Figure 11.

var listener = new ROSLIB.Topic({

 ros: ros,

 name: ‘/robot_joint_states’,

 messageType: ‘sensor_msgs/JointState’

});

listener.subscribe(function(message) {

});

The ROS uploads the robot’s joint positions, speeds, torques, temperatures, status data, target positions, path planning, motion commands, and timestamps for synchronizing robot status and motion trajectory data. This information allows the model in the WEB to simulate the industrial robot’s status in real time.

The collaboration of the entire system relies on these seamless information flows from the robot to the ROS, and then to the WEB application, and the information is finally displayed in 3D form on the user interface through Three.js. This lays the foundation for information flow synchronization for the future industrial metaverse. One of the obstacles encountered during this part of the research is the need to first examine the packets sent by the physical robot to determine which packet contains the arm position parameters and which one contains the movement trajectory of the arm. If the manufacturer’s development documentation does not provide this information, it will be necessary to identify it through manual comparison.

7.1. Model Loading Time Analysis

Figure 12 below is a comparison of model loading using the system with KUKA Sim Pro (version 1.1) and RobotStudio (version 6.08), the robot simulation communication software from KUKA Robotics GmbH and ABB Ltd., respectively.

Under the condition of loading the same model, each platform loaded the model 25 times, and the loading times were recorded by the testing tool. The test results are shown in Figure 13 and Figure 14.

According to the test results, our system’s average loading time for the KUKA KR150 model is 2.953 s, compared with 5.426 s for KUKA Sim Pro, resulting in a loading efficiency improvement of approximately 45.58%. The average loading time for the ABB IRB2400 model is 4.9224 s, compared with 6.5384 s for RobotStudio, resulting in a loading efficiency improvement of about 24.72%.

The significant improvement in model loading efficiency can be attributed to our system’s smaller codebase, which excludes functionalities such as offline programming, signal analyzers, and sales tools, focusing solely on model simulation. This specialization allows for greater efficiency in the loading of the same models compared with traditional-brand robot simulation software.

7.2. Model Drive Response Time

For the response time test, each model’s actions were divided into three groups, driving axes 1, 2, and 3, respectively. The model’s three-axis markings and driving directions are shown in Figure 15.

Each group of actions was tested 25 times. In the same environment, the control command issuance for the models was tested. The response time test results for our system’s model driving are shown in Figure 16 and Figure 17.

According to the test results, the average time for driving KUKA KR150 axes 1, 2, and 3 with our system is 43.72 ms, 46.48 ms, and 54.08 ms, compared with KUKA Sim Pro’s 76.92 ms, 79.64 ms, and 90.08 ms, resulting in loading efficiency improvements of approximately 43.16%, 41.64%, and 39.96%, respectively. The overall average response time improvement is about 41.50%.

For the ABB IRB2400, the average time for driving axes 1, 2, and 3 with our system is 62.76 ms, 64.12 ms, and 57.12 ms, compared with RobotStudio’s 85.32 ms, 89.48 ms, and 83.44 ms, resulting in loading efficiency improvements of approximately 26.44%, 28.34%, and 31.54%, respectively. The overall average response time improvement is about 28.75%.

The improvement in response times can be attributed to the limitations of traditional simulation software’s teach pendants, where commands must be transmitted to the model through protocols like TCP/IP, EtherNet/IP, CAN bus, and RAPID. In contrast, our system’s commands are directly transmitted via WebSocket, resulting in enhanced model driving response times.