Distributed Intelligent Assistance Robotic System with Sensor Networks Based on Robot Technology Middleware

This paper proposes a distributed intelligent assistant robotic system in order to improve the quality of the elderly people's life in the population aging society. The system is composed of embedded distributed sensor networks and multirobot intelligent platform. In the proposed system, we use SP2ATM (simultaneous path planning and topological mapping) with RBPF (Rao-Blackwellized particle filter) for path planning and localization of mobile robots. In order to perform service tasks, an accurate and reliable 3D environment map is reconstructed by using an effective 3D reconstruction technique from multiview stereo. Besides, the system provides user human tracking services based on multifeature mean shift under the double-layer locating mechanism. To improve the feasibility and reliability of the system, this proposed system is developed based on distributed control technology RTM (robot technology middleware). This paper presents the architecture of the proposed system and the experimental results verify effectiveness of the approaches.


Introduction
Assistance robotic system is particularly important for elderly people, with the increasingly serious aging problem. In recent years a lot of research results have been achieved in this field. Chung et al. [1] developed an intelligent service robot to help humans transport the object which they want to move. Rampinelli et al. [2] designed a system capable of obtaining the pose of robotic wheelchairs by the cameras in fixed positions to support the aged and disabled. Both Song et al. [3] and Tseng et al. [4] proposed a surveillance robot to monitor the home environment for security. van Osch et al. [5] developed a teleoperated robot which was controlled by a human operator to perform tasks normally done by humans in an environment. Alvarez-Santos et al. [6] presented a tourguide robotic system that is able to communicate with users by recognizing hand gestures and providing voice feedback. Lu et al. [7] built an intelligent home space for service robot based on multipattern information model and wireless sensor networks. In order to simplify the design process of assistance robotic system and increase reusability of robotic modules, distributed technology was introduced by [8][9][10].
Although service robot has been widely researched, some key technologies such as path planning, 3D reconstruction, and human tracking are bottlenecks for developing intelligent assistant robotic system. To address this problem, researchers have done many works. Oriolo [11] adopted SRT (sensor-based random tree) exploration method and randomly selected from the boundary the next exploration target. However, the positioning error restricted mobile robot to realize autonomous exploration in the case of high position accuracy. There exist many approaches to 3D reconstruction from pairs of images [12,13]; these approaches were typically too computationally intense for real-time applications. In order to track targeted person, many researchers have accomplished some achievements. Cipolla and Masanobu [14] primarily considered foreground segmentation based on disparity image. Other approaches [15,16] assumed that face detection could be used in the following system. However, the real-time tracking performance of these algorithms needs to be improved. The algorithm [17] perfectly eliminates the background interference to target model and track, but the tracking based on color cannot accurately locate target in the case that there are similar color objects in environment. In this paper, we focus on modular design theory and the key technologies for assistance robotic system. We developed a distributed intelligent assistant robotic system based on RTM. The indoor assistant robotic system is composed of embedded distributed sensor networks and multirobots platform. Distributed sensor networks, which transplant open RTM-aist on the embedded platform under Cortex-A8 framework, are adopted to perceive indoor environment. And robot can accurately deliver the objects which the user wants to the place where the aged or disabled is by robot path planning and positioning systems. We developed SP 2 ATM algorithm with RBPF to realize path planning in an unknown environment. This algorithm effectively improves the location precision of mobile robots and lowers computation complex to ( ⋅log 2 ). In order to perceive home environment, we propose an effective 3D reconstruction approach, which embeds optical flow-scene flow framework into stereo visionbased 3D scene reconstruction for an environment-learning assistance mobile robot. Compared with the other methods, the proposed approach has the advantage of being robust to illumination change and the accuracy of the deformed surface of the environment can be improved. Besides, aiming at realizing a human following task in the system, a human tracking method based on multifeature mean shift under the double-layer locating mechanism is proposed to solve the problem of distinguishing target, occlusion, and quick turning. In order to enable the system to be extended and integrated easier for a new system or new applications, distributed control technology RTM is introduced into the developed assistance robotic system. RTM enables assistance system to be language independence and operating system independence. RTM aims to modularize robotic functional elements as RTC (RT component). RTCs of the system can be implemented by different programming languages, run in different operating system, or connected in different networks to interoperate. This paper details the architecture of the proposed system and gives some experimental results to verify the effectiveness of the approaches.
The remainder of this paper is organized as follows: Section 2 describes the overview of system. Section 3 presents the developed key technologies for the assistant robotic system. Section 4 details the implementation of RTC. The experimental results are given in Section 5 and Section 6 concludes the paper.

System Overview
The proposed system is shown in Figure 1. Compared with the main methods for robot to get environment information by equipped camera and many expensive sensors on the classical robotic platform, the approach of distributed sensor networks proposed in this paper has many advantages such as parallelism, flexibility, and fault tolerance, which can share the information from sensors, overcome the uncertainty of the sensors, and improve the efficiency of performing complex tasks. The robotic system is implemented by Open RTMaist based on distributed technology. Embedded cameras in distributed sensor networks (implemented in Cortex-A8) are installed in the rooms and corridors to monitor the indoor environment. The robot (xPartner robot AS-R) which is mounted with URG scanning LRF (laser range finder) and Flea2 digital camera uses the improved SP 2 ATM with RBPF-SLAM algorithm to help user transmit object. Meanwhile, the 3D map can be reconstructed by the stereo vision system which is installed on xPartner robot, when the xPartner robot automatically moves in the room. In addition, aiming at realizing a human following task in a cluttered environment, the robot (an American Mobile Robot Inc. Pioneer 3-DX) which is mounted with a stereo camera and a RFID uses multifeatures and various kinds of sensory information to provide user human tracking service.

Key Technologies
Three key technologies (path planning, 3D reconstruction, and human tracking) are developed in the distributed intelligent assistant robotic system. The system can not only provide object delivery service based on SP 2 ATM with RBPF-SLAM algorithm but also provide users with monitor tracking service by multifeature mean-shift algorithm. Meanwhile, the position of the robots can be displayed on the 3D reconstruction environment maps in real time.

Path Planning.
Path planning is the key technology to provide object delivery service. Under the grid-mixed topology map model, SP 2 ATM algorithm is easily implemented to path planning and does not need more storage space. In addition, this algorithm can effectively reduce the space complexity and computational complexity of the largescale environment grid map. Besides, based on the structure of HTM (hierarchical topology map), node update rules provide a global optimal path planning strategy to improve exploration efficiency further. On this basis, RBPF-SLAM algorithm based on the grid map is introduced into the robot autonomous exploration mission. The node is constantly updated by using particle filter during the position estimation of the mobile robot. This algorithm effectively improves the location precision of the topological node, thus preventing robot operation failure caused by accumulated position error.
The improved SP 2 ATM with RBPF-SLAM algorithm is mainly adopted in the function module to achieve object delivery tasks. In this part, SP 2 ATM algorithm [19,20] is used for environment exploration and the RBPF-SLAM is adopted for robot localization. AST [21] (admissible space tree) is a set of nodes and graphs demonstration, which is addressed to describe the admissible space in the environment and facilitate map building for mobile robot. After that, HTM [21] is maintained with two-layered structure which describes, respectively, the possible nodes generated by AST and an undirected graph of topology nodes.
Sometimes, some nodes which are closed to the established topology or possible nodes may not belong to a former topology's AST. In order to avoid revisiting and guarantee the global convergence of instant goal node, node updating rules [21] are used to append, rebind, or get rid of those nodes.
When the robot fails to find any possible nodes in current AST, it is necessary to backtrack to a nearest topology node   which is determined by Bellman-Ford algorithm adopted in [21].
In order to improve the location precision of the topological node, RBPF-SLAM is used for path planning. The complexity of the SLAM problem arises from the very high-dimensional state space which consists of the variables describing the environment state and the robot pose. PF (particle filter) is an effective Monte-Carlo approach for SLAM. Murphy presented a RBPF-based mobile robot SLAM. The general idea of this approach is the decoupling of map estimation and robot pose determination based upon the global map estimation. Compared with EKF, RBPF requires less demanding; its computation complex is reduced to ( ⋅ log 2 ). In our scheme, for saving memory, each particle carries a sample pose of robot individually and all particles share a common environment map.
The SP 2 ATM with RBPF-SLAM algorithm is detailed in Algorithm 1.
As Shown in Figure 2, the service robot can generate topological maps to help users deliver objects using Algorithm 1.

Scene Flow-Based Environment 3D Reconstruction.
When the robot automatically moves in the room, users can check the current robot position on the 3D reconstruction environment map. An effective reconstruction approach, which embeds optical flow-scene flow framework into stereo vision-based 3D scene reconstruction for an environmentlearning mobile robot, is proposed. This part is inspired by the idea of casting the surface adjustment problem as the nonrigid motion recovery issue. As optical flow vector field estimation outperforms traditional dense disparity for its inherent advantage of being robust to illumination change and being optimized and smoothed in global sense, the deformed surface is improved in quality dramatically. Block diagram of reconstruction scheme is shown in Figure 3.
For the proposed 3D reconstruction technology in this paper, SIFT (scale-invariant feature transform) corner detector is employed for interesting feature points extraction.
The multiview camera setup parameters, including focal, principal point, and baseline length, must be carefully calibrated based on the digitized points derived by triangulation measurement.
To improve position accuracy of the calculated point set in practical implementation, bundle adjustment technique is employed to refine the scattered points reconstruction to produce jointly optimal 3D structure and viewing parameter (camera pose and/or calibration) estimates concurrently [22], since both of which are corrupted by noise. Now, building a raw deformable surface mesh based on the acquired discrete points efficiently is considered. For an implicit surface building mission, energy minimization can be solved using RBF (radial basis function), a well-established mathematical tool for solving scattered data interpolation problems [23], which accounts for its popularity in surface recovery applications.
Implicit surface tessellation is needed for surface digital presentation in computer and for human visualization, and usually implicit surface is converted into triangulated models. During this process, two steps are mainly involved. Firstly, 3D space is divided into adjacent cubic cells whose corners are evaluated in the implicit surface function: negative values are considered inside the surface, positive values outside. Then, within each cell, the intersections of cell edges with the implicit surface are connected to form one or more polygons.
The adjustment problem is cast as a nonrigid motion recovery issue. For this purpose, the reference frame image is back-projected onto the base mesh model surface and the textured model is projected into the comparison camera frame to synthesize an image prediction in that camera [24]. The synthesized image is in fact a feedback to the grabbed comparison image frame, depending upon the assumption that the reconstructed polygonal mesh surface is the true description of real scene structure.
The main objective of optical flow calculation is to derive a flow field which estimates the motion of pixels in two consecutive image frames [25] and, in turn, to estimate the relative motion between the camera and the objects in the scene. Consider As demonstrated in Figure 4, the point denotes the projected point on primitive mesh model along a reference camera ray and is the back projected image point on the th comparison camera corresponding to intersection , while denotes real observation. The raw estimation is refreshed into an updated position: The direction and the corresponding unit vector along the ray specified by the projection center of reference camera and the current image point is easy to be determined; therefore, where signifies magnitude of this vector. Equation (3) multiplies by jacobian matrix and yields where is the 2 × 1 inner product between each Jacobian matrix with its associated reference ray vector Δ ⃗ the estimated optic flow vector in the th image frame, which is determined by the above stated TV-L1 method [25]: So, the adjusted raw polygonal mesh based on scene flow is applied to obtain accurate 3D reconstruction model of real environment to enable users to understand the current position of robot.

Person Tracking.
The assistance robot can also provide the human tracking service. A method which fuses multifeatures and couples various kinds of sensory information is proposed in this paper. Because the RF signals have an excellent range in indoor environments and the tag ID can distinguish other passersby and followers, the RFID detection system is embedded in our system ( Figure 5). In order to deal with the unstable signals of radio frequency [26], the approach applies the Bayesian rule to determine the location of the object. In order to improve the precision of localization, stereo cameras are adopted to track the target by multifeature. The image preprocessing of stereo vision benefits from the coarse location estimated by RFID. Then, prelocation based on headshoulder is utilized to determine the candidate targets.
However, the single feature cannot reliably distinguish target between other people. Therefore, a multifeature mean shift to track the target further is utilized as shown below.
In the mean-shift tracking algorithm, the desired object is firstly selected with the position given by prelocation based on head-shoulder. The pixel locations inside the rectangle are denoted by { }, = 1, 2, 3, . . . , . The selected region is considered the object model, where its color histogram is calculated bŷ International Journal of Distributed Sensor Networks Here, hs ( ) is the binary image of the silhouette. new (‖( − 0 )/ℎ‖ 2 ) is the improved convex kernel function based on the silhouette of target, which is utilized to overcome color or texture clutter in background, various illuminations, and other related limitations. As a consequence, the value of new kernel in the background is 0. While in the foreground, the value of new kernel is the same with Epanechnikov function. ( ) is the bin number (1, 2, . . . , ) associated with the color at the pixel of normalized location , ( ) is the delta function, and is the constant normalization.
The pixel locations of the candidate area, centered at in the current frame, are represented by { }, = 1, 2, . . . , ℎ . The probability of the color in the candidate model is given bŷ where ℎ is the constant normalization.
Local binary pattern (local binary patterns, LBP) is an effective texture description operator, and we implement LBP to describe the texture of person's clothes. Similarly, the texture histogram of the target is defined aŝ where V is the constant normalization and ( ) is the bin number (1, 2, . . . , ).
The probability of the texture V in the candidate model is given bŷ where V is the constant normalization. In this system, both color and texture features are integrated into the mean-shift tracking framework. In order to combine the target color and texture information, a colortexture similarity function based on Bhattacharyya coefficient is introduced to seek the position of maximum likeness between the object and candidate models: wherê( ) is color-texture similarity function,̂is color Bhattacharyya coefficient, and̂V is texture Bhattacharyya coefficient. The goal in each frame is to estimate the object that maximizes (10). It should be noted that 0 is the estimated object location in the previous frame. The Bhattacharyya coefficient is approximated in the current frame by its firstorder Taylor expansion where is weight for (11). The search for the new object location of maximum likeness 1 in the current frame can be achieved by using the gradient based mean-shift optimization. During this procedure, the target position is iteratively moved from the current position̂0 to the new position 1 according to where ( ) = − new ( ).

Implementation of RTCs
The RT system is constructed by connecting the ports of multiple components as an aggregation of each RTC function. The OpenRTM-aist is an open source implementation of the RT-Middleware specification based on CORBA technology. It is developed by the National Institute of Advanced Industrial Science and Technology Intelligent Systems Research Institute Task Intelligence Research Group. In this paper, to improve the flexibility, reliability, and real time performance, distributed intelligent assistant robotic system is designed by using RT middleware. Underlying functional components of the system are partitioned according to the following fundamental components.
To obtain module partition projects with different granularities, fuzzy tree graph clustering algorithm and AHP (analytical hierarchy process) are utilized to analyze correlation between modules and classify samples from the view of function and structure. Furthermore, the system is evaluated from four aspects such as integration complexity, module  cohesion, and coupling between modules and system load balancing degree. These four aspects provide reasonable and effective solutions for robot module partition granularity problem.
According to the requirements of the system to provide the services, the partition functional components are divided into LRF (laser range finder) data acquisition, vision image obtaining, mobile robot control, feature extraction, map generation based on LRF, and so forth. The components structure chart of this system for indoor assistant robots is shown in Figure 6.

RTCs Implementation.
All functions proposed in this paper, which can be operated with each other to implement complex function, are packaged into RTCs. There are four classes of function modules implemented in this system, SP 2 ATM with RBPF-SLAM modules, person tracking modules, 3D reconstruction modules, and distributed sensor modules. We will take SP 2 ATM with RBPF-SLAM modules as example to describe function modules implementation. The implemented system is detailed as below. (ii) Outport<TimedDoubleSeq>: output the possible nodes, topology nodes, and LRF data; (iii) Outport<TimedVectorSeq>: output the control command of mobile robot.
When a system is designed with above RTCs, users just need to know that the laser data can be acquired from InPort and the local and global maps can be attained from Out-Port without understanding the details of the encapsulated SP 2 ATM.

Experimental Results
In this part, to verify the effectiveness of this proposed intelligent assistant robotic system based on distributed technology RTM, the experiments are performed in the laboratory. All RTCs of services are distributed in different locations and make up the intelligent assistant robotic system. During the experiment process, laser data reading module of SP 2 ATM is operated on the robot platform, and possible nodes searching module is operated in the user's computer. The interoperation between these modules is implemented by GIOP (general inter-ORB protocol) under RTM framework.

Hardware Platform.
The hardware setup mainly consists of two mobile robots. The robot for performing person following task is composed of an American Mobile Robot Inc. Pioneer 3-DX mounted with a stereo camera and RFID. Two FLEA2 FL2-08S2M cameras produced by point grey company are fixed nearly parallel in optical axis direction and look forward to constitute a stereo camera. Two color images and a depth image can be captured from the stereo camera and fed to a computer by an IEEE 1394B cable. Their focal length is about 3.5 mm. The baseline is 140 mm and the horizontal FOV (field of view) is about 45 ∘ . Resolution of both the images is 800 × 600. The RFID system is composed of a RFID tag, two antennas, and a card reader. The ID tag is attached to a given person. The antennas can still detect the given person. Then, the card reader receives the detecting data and transmits it to the computer for data processing. The horizontal field of view is 60 ∘ , which is wider than that of the stereo camera.
The other robot, for helping user transmit object, is the xPartner robot AS-R with URG scanning laser range finder and Flea2 digital camera. The URG scanning LRF offers both serial (RS-232) and USB interfaces to provide laser scans. The field of view for this detector is 240 degrees and the angular resolution is 0.36 degrees. Distances are reported from 20 mm to 4 meters, and the resolution is 1 mm. Besides, Flea2 digital camera is used as vision system with 3.5 mm length and 70 ∘ field of view. Digital images are acquired into laptop computer through 1394 bus.

Results
. Some experiments were carried out. Figures 8  and 9 show the results of different mean-shift algorithms. As shown in Figure 8(a), the mean shift based on texture cannot precisely track given person due to similar texture. Although the algorithm [17] perfectly eliminates the background variation by the modified kernel function, the tracking algorithm based on color cannot accurately locate target because of the similar background color. In comparison with the above mean-shift algorithms, the proposed method and algorithm [18] are more accurate, which is attributed to combining the color and texture. However, the iteration times of the proposed approach is less than the algorithm [18] due to the improved convex kernel function as shown in Figure 10, prelocation based on head-shoulder and RFID, which increasingly improve the real-time performance. Figures 11(a) [17] (c) Results of algorithm based on [18] (d) Results of our algorithm Base on [18] Our algorithm The robot autonomous path planning results for object delivery are shown in Figures 11(i)-11(n). In the experiment, the robot started from the room which user A stayed in and passed corridor to the room where user B stayed. The movement process is shown in Figures 11(j) and 11(k). Then, the robot obtained a book from user B and came back with the same path to user A, as shown in Figures 11(k)-11(n). During the experiment process, users can not only observe the robot position on 3D reconstruction environment maps but also observe each room and corridor situations by distributed sensors. RTM framework enables assistance system to be effectively independent of operating systems and programming  languages, increases the reusable value of function modules, and simplifies the development processes. The experiment results verify the effectiveness of three key technologies of this proposed system, and distributed control technology makes the system structure flexible.

Conclusion
In this paper, we have developed a distributed intelligent assistance robotic system and proposed three key technologies for providing some basic services to support the Object delivery and 3D reconstruction map aged or disabled. In the developed system, the SP 2 ATM algorithm was improved with RBPF-SLAM to realize the accurate location of mobile robots. On this basis, the 3D reconstruction approach, which has the advantage of being robust to illumination change and the high accuracy of the deformed surface compared with the other methods, was implemented to reconstruct home environment. Besides, human tracking accuracy was improved by fusing multifeatures and coupling various kinds of sensory information. In addition, all developed technologies were integrated by using distributed control technology RTM; therefore, the flexibility and scalability of the system were improved. The experiments verified the effectiveness of the proposed intelligent assistant robotic system and advancement of the three key technologies. For the future work, we will focus on extension and optimization of system function.