International Journal of Advanced Robotic Systems Robotic Platform for Automated Search and Rescue Missions of Humans Regular Paper

which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract We present a novel type of model incorporating a special remote life signals sensing optical system on top of a controllable robotic platform. The remote sensing system consists of a laser and a camera. By properly adapting our optics and by applying a proper image processing algorithm we can sense within the field of view, illuminated by the laser and imaged by the camera, the heartbeats and the blood pulse pressure of subjects (even several simultaneously). The task is to use the developed robotic system for search and rescue mission such as saving survivals from a fire.


Introduction
The proposed device consists of a combination between a photonic remote vital life signal sensing device and a robotic platform. The photonic device can be used for continuous and accurate monitoring of various biomedical parameters, and especially, the sensing of blood pulse pressure and heartbeat, as part of the remote detection of vital life signals in smoky conditions.
The system includes a camera and a laser. The laser illuminates a given field of view, while the camera takes images at a high sampling rate. A close range demo of the proposed system was constructed before, to monitor various biomedical parameters such as heartbeat [1,2], blood pulse pressure [3] and glucose concentration [4], as well as to track the 3D movement of spontaneous contractions of rat cardiac muscle cells [5]. High precision inspection of the 3D movement of red blood cells allowed early detection of malaria [6]. Note that in our photonic device we inspect the secondary reflected speckle pattern. The speckles are self-interference random patterns [7] and have the remarkable quality that each individual speckle serves as a reference point from which one may track the changes in the phase of the light that is being scattered from the surface [7]. Because of that, speckle techniques, such as electronic speckle-pattern interferometry (ESPI), have been widely used for displacement measuring and vibration analysis (amplitudes, slopes and modes of vibration), as well as characterization of deformations [8][9][10][11][12][13][14]. Usage of speckles was also applied for improving the resolving capabilities of imaging sensors [15], as well as ranging and 3D estimation [16]. However, we do not use regular ESPI configuration, but rather defocus the optics of our camera to allow the reflected secondary speckle pattern to shift, rather than to change due to tilting of the reflecting surface. By tracking the shift of those patterns, one may accurately estimate the movement of the reflecting surface and from that to be able to estimate various biomedical parameters such as heartbeats.
The developed robotic system is applicable for search and rescue missions such as saving survivors from a fire. When a fire starts in an apartment or an office, it takes time until the fire fighting forces come to take care of it. At that time the apartment/office is already full of smoke and this makes it difficult for the fire fighters to see the apartment structure and observe survivors.
The system we present here can help in decreasing the time for finding casualties by first reporting the finding of any casualties and second, the location of the casualties in the apartment. A map of the apartment is built as the search progresses. The navigation procedure is a classic SLAM (Simultaneous Localization and Mapping) problem [17]. Using a camera or infrared (IR) sensors for mapping or localization as appears in the literature [18][19][20], or a combination of sensors like USB cameras, microphones, pyro-electric sensors and infrared cameras as appears in [21 and 22], are good solutions for detecting victims in many environments but are less effective in fire and heavy smoke. Therefore, the SLAM process will be based upon ultrasonic sensors, particle filters and navigation algorithms. The robot will start searching an apartment, while the above-mentioned laser system scans the area for human heartbeats. When a room with human survivors is found, the robot will pass the doorway and approach the human survivors, evaluate their vital life signs and transmit the data regarding their medical state, as well as their current location, to a control station via a wireless communication channel. There is no need for absolute accuracy on the exact location of a survivor within the room.
The control station supervisor can then make a decision whether there is enough information to take action. The robot, if a human was found, will signal immediately to the rescue team, while continue its search in the other areas of the apartment. If no human was found, the robot will send a special signal to the rescue team, avoiding potential damage and risk to the fire fighting team members.
The main benefit in the configuration we propose in this paper is related to the fact that there is an automated robotic platform that is capable of the remote detection of life signals and performing rescue operations without endangering human rescue crew members. Currently there is no such platform available. The technology for remote life signals detection is novel as well [1].
In this paper we first describe the robotic platform and then perform experiments with the photonic remote life signals sensing device. In the experiments we try to detect the heartbeat of a fireman that was standing in a room filled with cold smoke.

Robotic platform
The robot hardware configuration is presented in Fig. 1 and includes a control unit based on a Spartan 3 FPGA board, four DC motors with shaft encoders, four omniwheels, 16 ultrasonic distance sensors, a compass, a voltage system including a battery, a main voltage distributer and voltage regulators, a four motor controller and a communication device. On top of it the photonic life signals detecting device is assembled.  Figure 2 presents the robot wheel configuration. Next we will discuss the kinematics and the speed control mechanism of the robotic platform. The common robot kinematics can be written as:

Robot Kinematics
where,  is the angle between wheel i and the robot x- is the angular velocity of wheel i,   is the robot angular velocity about the z-axis of the global reference frame and d is the distance between wheels and the robot centre. Desired robot velocity ( desired  ) is used to generate the robotʹs wheel angular velocity vector ( r  ).
This wheel angular vector is the control signal, which is sent from the FPGA processing board. The output angular velocity ( sensed  ) of each wheel is sensed by each motor encoder.
The output velocity contains information about disturbances. The disturbance error can be found by comparing the desired velocity and the sensed velocity. The output velocity can be defined as in Eq. (2), where it is assumed that the disturbance is constant for a specific surface. Two kinds of disturbance are modelled in Eq.
where †  is the pseudo inverse of the kinematic equation,  is the disturbance gain matrix due to the robot coupling velocity friction and  is the disturbance vector due to the surface friction.
The correction of the speed of each motor is done by a PID algorithm for each motor speed.
The four motor control adds additional PID controllers for the robot translation and rotational vectors. The implemented control system is shown in Fig. 3. It uses wheel encoders as feedback sensors to close the control loop. Each wheel PID controller is responsible for making it rotate at the commanded speed. Interleaved with them are two PID controllers responsible for the robot motion. The robot drives faster when no object is sensed by the distance sensors or when the distance to the front wall or object is above a predefined threshold.

Localization and mapping using particle filter
In particle filters, the samples of a posterior distribution are called particles [23] and are denoted as: is a concrete instantiation of the state at time t, i.e., a hypothesis as to what the true world state may be at time t. Here M denotes the number of particles in the particle set t X . In practice, the number of particles M is often a large number. In some implementations M is a function of t or of other quantities related to the belief ( ) t bel x .
The particle filter [23], which is nonparametric filter, relies on Bayes filter posterior: is the weight of a particle.
Here the robot localization is carried out using particle filtering, or the sequential Monte Carlo method as just described. The inputs we use are objects detected by the distance sensors, motors tracking and the compass data. By using the filter, we were able to make an educated guess of the robotʹs location and direction, relative to the fixed objects in the field (corners, doorways or obstacles).  5(a)). Given the sensorsʹ data, particles are graded by the angle the object is seen at and by distance. Noise is added to each input ( Fig. 5(b)). After a few iterations, the particles converge to the robotʹs real location (Fig. 5(c)).
After some more iterations, the particles are very close to the robotʹs real location. Note that some randomly scattered particles (10%) are added to the particle sets in each iteration (Fig. 5(d)), in order to be able to handle failures such as robot kidnapping or global localization failures.

Navigation
We will interpret an apartment as a maze and use mazesolving methods for navigating through the apartment. The maze is built with a main course, with sideways (rooms) breaching from the main course.
In order to decide upon the navigation procedure we used QFD (quality function deployment) [24], the Pugh concept selection method [25] and FMEA (Failure Mode and Effects Analysis) [26] design tools. The layout of the QFD diagram is shown in Fig. 6. It is often called the house of quality because of this configuration. Each room in the house has a name that implies its function. Table 1 describes what is found in each of the "rooms" of this house. Another popular conceptual design tool is the Pugh concept selection method. This tool compares each concept relative to a reference or datum concept and for each criterion determines whether the concept in question is better than (+), poorer than (-), or about the same as the reference concept (S). Thus, it is a relative comparison technique. Studies show that a small group is better than an individual at selecting ideas.
One of the design tools for preventing failures is Failure Mode and Effects Analysis (FMEA). This is a proactive design tool technique and quality method that enables the identification and prevention of process or product errors before they occur. It assists in identifying the different failure modes of a system and might be used for reliability evaluation. What is found in each of the "rooms" of the house of quality Customer requirements (whats). Can be a problem statement etc.
Competitive assessment shows how the top two or three competitive products, ideas or algorithms etc. rank with respect to customer requirements. This section starts with ranking each customer requirement on a scale of 1 to 5 and then by considering the planned improvement and any requirements that are planned for special attention and then builds up to an importance rating. Importance rating: Here the importance of each requirement is evaluated based on customer importance rate, the existing product and the planned product.
Engineering characteristics (hows) that enable satisfying the customer requirements are listed in columns.
The correlation matrix shows the degree of interdependence among the engineering characteristics in the "roof of the house". Often this "roof" is not used because either the EC correlation is irrelevant or the supply information is useless. In cases where trade-offs should be made, the "roof" data is important.
The relationship matrix determines the correlation between the engineering characteristics (EC) and the customer requirements (CR). To do this, each CR is taken and for each EC, in turn, it is asked whether it will significantly, moderately, slightly, or not at all impact the customer need. Generally a nonlinear 9/3/1/0 scale (sometimes five is added) is used to weight disproportionately those ECs that strongly affect CR.
To determine the absolute importance, first multiply the numerical value in each of the cells of the relationship matrix (6) by the importance rating (3). Then, sum the numbers in the cells of each column. These totals show the absolute importance of each EC in meeting the CR. Relative importance is the absolute importance, normalized on a scale from 1 to 100, by taking each absolute importance value, dividing it by the sum of absolute importance values and multiplying it by 100. Those ECs with the highest rating should be given special attention, for these are the ones that have the greatest effect upon customer satisfaction. The technical competitive assessment benchmarks your company performance against two or three top competitors for each of the engineering characteristics. Generally a scale of 1-5 (five being the best) is used. Often this information is obtained by acquiring examples of the competitorʹs product and testing them. Technical difficulty indicates the ease with which each of the engineering characteristics can be achieved.
Setting target values. By knowing which the most important ECs are, understanding the technical competition and having a feel for the technical difficulty, the team is in a good position to set the targets for each engineering characteristic. Table 1. Description of what is found in each of the "rooms" of the house of quality.
Using the FMEA, Pugh and QFD analyses we decided upon the following left wall navigation, presented in Fig. 7. The robot uses a special memory location for keeping track of its movements. Each forward movement adds a positive distance, according to the wheel encoder counting number, to a longitudinal memory variable. Each backward movement adds a negative distance to the longitudinal memory variable. Each left movement adds a negative distance to a width wise memory variable. Each right movement adds a positive distance to the width wise memory variable. The scan is complete when both the longitudinal and the width wise memory variables contain zero with a small tolerance.

Scanning for life signals
The remote photonic life signal sensing device is installed on top of the robot and is capable of rotating ±90º relative to the robot driving direction and rotates faster than the robot speed. The scanning system will be continuously active. When a life signal is detected the location of the robot, the robot's angle and the angle between the robot and the device are recorded. Then the scan device returns to be aligned with the robotʹs forward direction. Then the robot rotates to the position of the scan device where life signals were just detected and the system verifies that indeed the life signal exists, in order to double check. The location is then transmitted to the control station.
Each angle is recorded in the memory using First In First Out (FIFO) data structure. In the case of an additional need for the robot to guide the fire fighters in the apartment, the robot uses the following method. With no loss of generality we assume that all turns are right angles. When the robot starts navigating from the entrance, its direction is regarded as a reference to all angles. Four directions relative to the reference direction are defined: A -Same as reference direction B -90º from reference direction C -180º from reference direction D -270º from reference direction These are different from the common angle notation presented in Fig. 8. The robot can navigate in the apartment corridor, stopping at locations of victims or by doorways with victims. Sequences of DABCB or BCDAD can be omitted from the course as they imply a room search. Then, multiple consecutive occurrences of the same direction can be diminished to one.
In the case of the example of an apartment presented in Fig. 7, the sequence of angles recorded is: ADABCBADABCBABCBCDADCBCDADCD, where underlined sequences indicate in-room searches that can be omitted. The navigation with fire fighters will be AAABCCCD, decreased to ABCD. During performing this course the robot stops near each room with casualties in it and near each casualty in the corridor. When a victim is found in a room the robot returns using Last In First Out (LIFO) data that is extracted from the stack. The robot navigates the shortest path by deleting routes that are redundant, using the logic described above.
However, in real rooms furniture and other obstacles might be attached to walls. For this reason, we change the method somewhat. When the robot encounters a doorway it assigns the letter "O" before navigating in the room. When it goes out of the room it assigns another "O". Then for the shortened navigation it deletes all sequences between the two "O"s ("O"s inclusive) and resumes as before.

Control Station
The control station is used to build a map of the apartment with locations of casualties. The user interface allows for on-line and off-line presentations of the mapping process and location display (see Fig. 9).
The control station "listens" constantly to the robot. When a message is received from the robot, it goes to a specific memory address. Then the message is decoded and a map component is built. If the message reports a casualty location, a special sign is placed over the map.

Operation principle of the photonic device
The photonic device is based on tracking the temporal change in the secondary speckle pattern reflected from the inspected subject. The reflected secondary speckle pattern is actually localized self interfering patterns (and therefore the detection module does not require an external interferometer). In the case of movement of the reflecting surface, those self-interference random patterns are randomly changed. We have previously demonstrated [1,2] that properly defocusing the imaging lens of the camera yields the movement of the reflecting surface and its tilting is mainly expressed not in a random change of those patterns, but rather in their movement. Thus, the change of the tilting of the reflecting surface in time causes the movement of the random pattern in time. Capturing a set of images versus time and correlating each sequential image with another can yield the relative movement of the patterns. This relative movement is obtained simply by allocating the time varying position of the correlation peak for each one of the correlation images (simply by allocating the position of the maximum in the correlation image). This temporal movement of the position of the correlation peak is plotted versus time after proper high pass filtering to remove possible noises. The obtained curve is related to the temporal movement of the surface of the skin and as this movement is connected to the mechanics of the beating of the heart, the desired sensing is extracted.
As previously stated in our setup we measured a relative value of the spatial displacement of correlation peaks between two adjacent frames taken by a fast digital camera. Relative displacement under proper defocusing is equal to: where  is a relative spatial shift of the speckle pattern due to object displacement,  is the tilting angle of the object and  is an optical wavelength. Assuming that the change in the angle is small enough, we obtain a linear proportion between the relative shift and the actual relative radial movement of the inspected surface: where d is the axial displacement of the vibrating object. The measurement of this displacement is done with nanometre accuracy because the shift of the speckle pattern is a function of a fraction of the optical wavelength (which in our experiment was 532nm).

Smoke and fire experiments with the photonic device
The goal of the cold smoke experiment was to obtain a good pulse reading in thick smoke conditions using the developed photonic sensing device. In the experiment we wanted to check the capability of the photonic system to find heartbeats of a subject who was located in a shed wrapped in smoke. As previously mentioned, the final aim is rescuing people during a fire involving difficulties such as bad vision conditions, mainly because of the smoke.
The experimental setup included a computer (the processing was done with Matlab software), a laser that projected light at wavelength of 532nm (SUWTECH, DPGL -2100, CW @ 532nm), high-resolution camera (Pixelink BL-771) and a power source. The light beam was projected by the laser source onto the fireman's chest, positioned within a room with cold and dense smoke.
With the camera we can observe the modulations and the software finds out if there is a heartbeat. The schematic sketch of the experimental setup is seen in Fig. 10(a), where the square resembles the shed, which was filled with the dense, cold smoke. Without smoke With heavy smoke positioned about 8 metres away from the laser-based system. We took a reference and measured his pulse. Afterwards we filled the room with smoke, until the visibility was zero. We measured the man's pulse, while we kept filling the room with smoke as the smoke began to scatter. It is important to mention that when the shed was full of smoke one could not see anything but the reflections coming back from the laser beam.
The experiment was run about 20 times. An example of the obtained results is seen in Fig. 10(b), where one may see the remote sensing of the heartbeat without and with the presence of the heavy, cold smoke. The pulse shapes are easily seen in the graphs. Each black arrow points at a pulse. Each pulse represents one heartbeat. The measured patterns presented in Fig.  10(b) were very repeatable. The deviations in the heart rate were less than 5%. Thus, in the experiment we proved that the remote life detecting system is capable of measuring heartbeats at a distance of several metres, even in heavy smoke conditions. Note that although the provided results were applied on a single fireman due to security and safety reasons, they were collected over a long period of recording, including 20 separate experiments in different smoke conditions. Another important point to mention is related to the fact that in bad visual conditions, as in a smoky environment, one may have difficulty distinguishing between humans and pets, for example. However, as the heartbeat signature of humans is very different from animals in the rate of the beating, as well as by the shape of each beat, such discrimination is actually quite a simple task.
In the second experiment we validated the capability of the optical remote sensing platform to sense the heart beating through fire. In this experiment we sent a subject to a distance of 100 metres away from the sensing optics (the white rectangle mark in Fig. 11(a) designates the subject as it was seen with our optical infra-red sensor) and placed a strong fire (we used a barbecue device) in front of him ( Fig. 11(b)). This time we used remote sensing optics that work in infrared, rather than at 532nm. In Fig. 11(c) we show the experimentally obtained heartbeat of the subject while the sensing was done and while illuminating through the fire with our laser. One can see the periodic beating repeating itself every second (approximately) and a repeated beating pattern very well. This experiment was run 25 times. The measured patterns presented in Fig. 11 were very repeatable. The deviations in the heart rate were less than 3%.
For a better understanding of the principles described in this paper, in Fig. 12 we show one of the prototypes we worked with for the photonic remote sensing device (left) as well as the robotic platform (on the right).  One period of the remote photonic extraction of the phonocardiogram Note that the developed sensor can measure, with nanometric accuracy, the movement of any inspected surfaces. As the precision is so high (nanometric) various biomedical parameters can be extracted. Due to blood pulsation and the heartbeat, the surface of our skin moves. This is a very small movement but as said before it can be detected on a nanometric scale. Obviously closer to the blood vessel the movement is larger and if the measurement is done with the surface being remote from the vessel then the movement is proportionally smaller. Nevertheless, one does not need to be located exactly on the blood vessel. Moreover the subject in our experiment was wearing a full set of fireman's equipment that isolated his skin from the illuminating beam. Nevertheless due to the high accuracy of the proposed sensing approach the beating of his heart could be sensed. Thus, not only is where the measurement is done not critical (which is good as in heavy smoke conditions the visibility is very low), but also the subject is not required to expose the relevant skin for performing the measurement.

Conclusions
In this paper we have presented a novel robotic platform, on top of which a new photonic remote life-sensing device is installed. The application discussed in this paper is related to the usage of the new constructed tool to assist fire fighting teams in rescuing people caught in fires. Experiments were conducted in a room with cold smoke, in which the heartbeats of a fireman were successfully detected in heavy smoke conditions. In another experiment we were able to sense the heartbeats of a subject through fire and from a distance of about 100 metres.