Design and implementation of a three-dimensional simulation system for a humanoid table tennis robot

This paper designs and implements a simulation system that can verify the planning algorithms such as table tennis trajectory planning and robot leg movement planning. The simulation system and the humanoid robot’s control system are connected based on the RTnet real-time Ethernet protocol. The table tennis game of the humanoid robot is implemented using the simulation of the forward and inverse kinematics of the robot arm, the hitting planning, and the forward and inverse kinematics of the robot leg.


Introduction
In recent years, research in the field of robotics has become very popular and has attracted the attention of many researchers. [1][2][3][4] They have successively developed robots for various fields. Humanoid robots have the characteristics of complex structures, high manufacturing costs, and high risks in walking and other operations. Humanoid robots can be imitated by human form or behaviors, [5][6][7][8] have higher levels of movement and intelligence, [9][10][11][12] and reflect more the level of mechanical manufacturing and automatic control, so are widely studied. With the advancement of research on humanoid robots, many scientists have carried out research on humanoid robot table tennis. Knight and Lowery 13 developed a humanoid robot SARCOS which can play a number of rounds of table tennis with a humanoid shape and is an important progress of the study of humanoid table tennis robots. Sun et al. 14 completed the research on the trajectory prediction of table tennis. Some others developed the arm of table tennis 15,16 with 6 or 7 degrees of freedom. In Sun et al., 17 a humanoid robot is designed for table tennis game and the table tennis player. The strategy and trajectory planning of the game are studied. 18,19 Therefore, it is necessary to utilize the simulation technology for studying their principles. As for the mechanical design of the humanoid robot, it is necessary to optimize the mechanism design of the humanoid robot in order to avoid losses caused by the designer's mistakes. When it comes to the work of a humanoid robot such as walking, it is important to first simulate its work to avoid the danger of dumping or being interfered by the surrounding environment during the work. Table tennis is a high-speed, high-definition task for humans, even more for robots, so it is very necessary to use the simulation system. Simulation technology 20 can be an efficient and safe tool for the research of humanoid robots, and it is widely used in the development of the humanoid robot field.
With the development of the computer simulation technology, the robot simulation technology has improved. Microsoft's Robotics Developer Studio 21 is an integrated environment for robot control and simulation research under the Windows operating system. It provides a large number of simulation models for industrial robots and mobile robots with friendly and realistic three-dimensional (3D) simulation scenarios. Users can control simulation machine models for robotics simulation through Microsoft's virtual programming language. MathWork's Simulink is tightly integrated with MATLAB and is an integrated environment for dynamic modeling, simulation, and analysis of systems. Camelot's ROPSIM 22 is an offline programming and dynamic simulation software widely used in the industry, and its offline programming system is especially praised by the industry. However, these simulation systems are too versatile, and it is difficult to meet the needs of functional customization such as humanoid table tennis robot, especially in the realtime performance.
During the process of project design, the simulation system can optimize the mechanism design of the humanoid robot. The ball trajectory collected by high-speed stereo vision is analyzed to establish the air flight model of the ball and the collision model between the ball and the racket. 23 After the design process of the robot mechanism, a realistic 3D humanoid robot model is built in the simulation system for the prediction of the ball trajectory and the motion planning of the robot arm. [24][25][26] The virtual table tennis experiment is carried out to verify the trajectory prediction algorithm of the ball, the trajectory planning algorithm of the humanoid robot arm and the leg, and so on. After the design and algorithm verification, the simulation system can be seamlessly connected to the control system of the humanoid robot to control the real humanoid robot for table tennis game.
The remainder of this paper is as follows. Section ''Humanoid robot table tennis game system'' presents the structure of the humanoid robot table tennis game system. Section ''Forward and inverse kinematics'' introduces the proposed forward and inverse kinematics. In section ''Seamless connection between the simulation system and the real system'', seamless connection between the simulation system and the real system is proposed. Experimental results and analysis are presented in section ''Experimental results''. Section ''Conclusion'' draws the conclusion of this paper.

Humanoid robot table tennis game system
The humanoid robot system for table tennis game is very complicated as is shown in Figure 1. There are more than 30 separately controllable joints in the whole body, which is named Huitong by the Beijing Institute of Technology. It has different sensors, including a sixdimensional force/torque sensor, a joint position sensor, a stereo vision, and so on.
It is necessary to maintain a dynamic balance for the humanoid robot's walking and hitting. The system structure of the humanoid robot is shown in Figure 2. Since real-time control of the robot joints is necessary for the control system of the humanoid robot, the realtime Linux system based on the Xenomai real-time framework has been chosen. The control of the distributed joint controllers is implemented via RTnet-based real-time Ethernet. The feedback sensing data from the six-dimensional force/torque sensors and distributed joint sensors are transmitted to the host computer via the RTnet-based real-time Ethernet. The prediction result of the vision system is transmitted to the host computer through Wi-Fi, and the user can interact with the host computer through the mouse and the keyboard. In addition, the external control interface through the RTnet real-time Ethernet is preserved in the control system of the humanoid robot, which is the interface between our simulation system and the real control system.

Forward kinematics
There are two methods commonly used in the study of robotic forward kinematics. Denavit and Hartenberg proposed the D-H parameter method, which first establishes a coordinate system for each link of the robot through the homogeneous transformation between the coordinate systems, which finally obtains the representation of the coordinate system executed at the end in the base coordinate system.
Another method is based on the spin theory, which expresses the equation of motion as the product of the exponents of the spins. In this paper, we will use the D-H parameter method. The coordinate system established according to the above-mentioned D-H parameter method is shown in Figure 3.  According to the definition of the D-H connecting rod parameters, the D-H parameters of the humanoid robot arm are presented in Table 1.
The homogeneous transformation matrix of the coordinate system fig relative to coordinate system fi À 1g is a function of four link parameters After obtaining the homogeneous transformation matrix of each link, the matrix is multiplied to obtain the final kinematic equation of the humanoid table tennis robot arm 0

Inverse kinematics
The inverse joint kinematics calculates the joint data of the robot to be sent to the joint articulation in each joint motor. It is also one of the most critical parts of our simulation. A 7-degree-of-freedom humanoid arm has an infinite number of arm configurations corresponding to the position of the end effector in the workspace and without the limitation of the range of articulation. That is, there are infinite sets of inverse kinematic solutions.
In particular, when the third joint angle is 90°, the Zaxis of the coordinate system of the first joint is parallel to the Z-axis of the coordinate system of the fourth joint, and given the pose of an end effector, a group can be uniquely determined. The joint configuration corresponds to it. That is, the solution of inverse kinematics is unique. We define the plane consisting of the origin S of the first, second, and third joint f1g, f2g, f3g coordinate systems, the origin E of the coordinate system of the fourth joint, and the origin W of the fifth, sixth, and seventh joint f5g, f6g, f7g coordinate systems as the reference plane. For any joint configuration, we define u as the angle between the plane of SEW and the reference plane, as shown in Figure 4. In this way, we can solve the seven joint angles with the unknown number u.
The humanoid robot arm end-effector coordinate system is known, and its homogeneous matrix in the base coordinate system f7g is 0 7 T = nx ox ax px ny oy ay py nz oz az pz For the fourth joint angle, independent of u, according to the cosine theorem Figure 3. Humanoid robot arm coordinate system.  where L 1 and L 2 are the lengths of the upper and lower arms. When the third joint angle is 90°, let 0 3 R 0 be the pose matrix of the third joint coordinate system f3g in the base coordinate system, and the angles of the front three joints of the arm are 0 u 1 , 0 u 2 , 0 u 3 , respectively. When the angle between the arm and the reference plane is u, the pose matrix of the third joint coordinate system f3g in the base coordinate system is 0 3 R u , and the angles of the first three joints are u u 1 , u u 2 , u u 3 , respectively. Then, we have the geometric relationship Substituting 0 u 3 = 908 and equation (2)  cos 0 u 2 = L 1 pz sin 0 u 1 + L 2 pz sin 0 u 1 cos 0 u 4 + L 2 py sin 0 u 4 sin 0 u 1 (px 2 + py 2 + pz 2 ) By homogeneous transformation, equation (1) is available Let the unit vector of the SW ! vector fpx, py, pzg be fkx, ky, kzg, then R SW !(u) is kxkxvu + cu kxkyvu À kzsu kxkzvu + kysu kxkyvu + kzsu kykyvu + cu kykzvu À kxsu kxkzvu À kysu kykzvu + kxsu kzkzvu + cu where su = sin u, cu = cos u, and vu = 1 À cos u, which can be solved by equations (4)-(8) where A i , :::, F i (i = 1, 2, 3) are constant coefficients. In the same way, we obtain tan u u 5 , cos u u 6 , and tan u u 7 as cos u u 6 = A 6 sin u + B 6 cos u + C 6 ð13Þ where A i , :::, F i (i = 1, 2, 3) are constant coefficients too. Therefore, we get the expression of u 1 , . . . , u 7 with u as the parameter. For a given u, from equations (2), (9)- (14), combined with the range of motion of the joint, a unique set of seven joint angles can be obtained.

Seamless connection between the simulation system and the real system
The external interface of the distributed joint controller based on RTnet real-time Ethernet is preserved in the control system of humanoid table tennis robot. We can directly transmit the simulation control signal of the table tennis robot to the actual control system of the humanoid robot through this interface, making the simulation system capable of controlling the real humanoid robot for table tennis game. The system structure of the seamless connection between the simulation system and the real system of the humanoid robot is shown in Figure 5.
The simulation system also uses the Linux system based on Xenomai's real-time framework. The trajectory prediction of the ball, the leg trajectory planning, and the arm trajectory planning are all implemented within this real-time system. The 3D graphics simulation runs on the non-real-time Linux operating system. The real-time system communicates with the non-realtime system to change the simulation scenario. The Figure 5. Communication structure of the simulation system and humanoid robot system. real-time system of the simulation system communicates with the preserved interface in the humanoid robot control system through RTnet-based real-time Ethernet.

Xenomai-based real-time system
Xenomai is a real-time framework based on the Linux kernel, which can provide user-space-based programs with comprehensive interface-independent real-time and seamless integration into the GNU/Linux environment. The structure of the adopted Xenomai is shown in Figure 6.
The bottom layer of Xenomai is Adeos, which directly interacts with the hardware. Adeos can abstract the upper kernel software as domains. The real-time kernel is the Xenomai domain, and the non-real-time Linux part is the Linux domain. Adeos provides a unified abstract interface to the domain by hardware abstraction layer. The Xenomai domain provides different real-time interfaces to the user space with different real-time system specifications through different skins. The simulation system of the humanoid table tennis robot is based on the Xenomai real-time framework. The simulation system is divided into real-time and non-real-time parts. The real-time part is executed in the Xenomai domain, mainly for online table tennis trajectory prediction. If the humanoid robot enters the range within which it can hit, the trajectory planning modules of the arm and the leg can calculate the joint motion of the humanoid robot, respectively. Then, the signals are sent to the control system of the humanoid robot by real-time Ethernet cycle. The non-real-time part in the Linux domain is mainly responsible for the 3D scene rendering and implementing the table tennis simulation. The real-time Xenomai domain and the non-real-time Linux domain exchange data through shared memory and pipeline. The real-time and nonreal-time processes are shown in Figure 7.

Real-time communication based on RTnet
RTnet is an open-source real-time Ethernet communication protocol stack in Xenomai, which utilizes standard Ethernet cards and provides driver support for the most common Ethernet cards. RTnet implements User Datagram Protocol (UDP)/Internet Protocol (IP), Transmission Control Protocol (TCP)/IP, Internet Control Message Protocol (ICMP), and Address Resolution Protocol (ARP) and provides POSIX standard Socket function interfaces for user space and kernel space.
To ensure the delay of network transmission, RTnet needs to use a dedicated Ethernet segment. The devices on the dedicated Ethernet segment control the access of the communication channel through RTmac. Both the humanoid robot control system and simulation system in this paper use the RTnet protocol stack and connect with each other directly through twisted pair. The connection structure is shown in Figure 8.
It can be seen from Figure 8 that RTnet implements real-time communication by replacing its traditional Ethernet Media Access Control (MAC) layer protocol based on collision detection and collision avoidance, and it has its own MAC layer protocol RTmac. It can  Twisted pair Figure 8. Connection between the humanoid robot control system and the simulation system.
provide interfaces for configuring and analyzing realtime communication through RTcfg and RTcap.
The simulation system and the humanoid robot control system can communicate according to the UDP protocol. The control system of the humanoid robot is the server and the simulation system is the client. The format definition of a UDP data packet is shown in Figure 9. The UDP data contain the UDP packet header part and the joint data. The joint data part contains the number of joints, the check information of the joint data, and the data of each joint. The data of each joint include joint number, joint angle, and so on.
The communication process of the simulation system and the humanoid robot is shown in Figure 10.
First, the humanoid robot waits and receives the data at the designated port. After getting the data sent by the simulation system, the legality of the data will be checked. If the data are correct, then the humanoid robot control system sends the data to the distributed driver and informs the simulation system that the data have been correctly received. If the received data are incorrect, the simulation system will be notified to resend it. Although a reliable transmission strategy for UDP transmission is added to ensure the reliability of real-time Ethernet transmission, it may affect the realtime performance of the network. However, in our experiments with hundreds of thousands of data transmission and reception, the packet loss or packet error never occurs, and the transmission delay of the 1000byte packet is below 0.1 ms.

Layout network
First, in the 3D simulation system based on Coin3d, the game table of humanoid robot table tennis is constructed as is shown in Figure 11.
It can verify the forward and backward kinematics of the arm, the stroke planning, and the motion planning. Then, through the Linux-based Xenomai realtime system framework and RTnet real-time Ethernet communication protocol network architecture, the seamless connection between the simulation system and the humanoid robot control system is realized, and the simulation system drives the real humanoid robot to play the table tennis game.
All experiments have been recorded with the arm hitting the table tennis ball. In Figure 12, there are 20 sets of data of robot deviation in the xyz position. If the robot fails to hit the ball, or the hitting effect is obviously incorrect, then there is no record.

Data acquisition and analysis
It can be found from the experimental data that the deviation between the predicted point and the observed hitting point under the control of the simulation system is consistent.   There is a clear consistency in the hitting time offset, that is, the hitting point is basically earlier than the predicted time. In the X direction, the real velocity is basically larger than the predicted value ( Table 2).
By analyzing 20 sets of data, the deviation between the hitting time and the predicted time is 10 ms on average, and the maximum is 30 ms. At present, the error between the simulation system and the actual control system is in the allowable range and is related to the calibration error of the vision system and the parameter error of the ball model.

Conclusion
In this paper, we have proposed the implementation of the seamless connection between the simulation system and the humanoid robot control system. It allows the simulation system to drive two real humanoid robots to play the multi-round table tennis game. First, this paper briefly introduces the real-time system adopted by the humanoid robot control system. Then, according to the demands of the humanoid robot control interface, Xenomai is chosen as the real-time part of the simulation system. RTnet is used for the simulation system and humanoid robot control. The system includes the interface of real-time Ethernet communication and introduces the definition of specific implementation and communication format. Finally, our simulation system can drive the real humanoid robot to play multi-round table tennis game, and the proposed method is verified by our experiments.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research was funded by the National 863 Program of China (2012AA041403) and the National Natural Science Foundation of China (61305107).