Analysis of human–robot interaction at the DARPA Robotics Challenge Finals
Abstract
In June 2015, the Defense Advanced Research Projects Agency (DARPA) Robotics Challenge (DRC) Finals were held in Pomona, California. The DRC Finals served as the third phase of the program designed to test the capabilities of semi-autonomous, remote humanoid robots to perform disaster response tasks with degraded communications. All competition teams were responsible for developing their own interaction method to control their robot. Of the 23 teams in the competition, 20 consented to participate in this study of human–robot interaction (HRI). The evaluation team observed the consenting teams during task execution in their control rooms (with the operators), and all 23 teams were observed on the field during the public event (with the robot). A variety of data were collected both before the competition and on-site. Each participating team’s interaction methods were distilled into a set of characteristics pertaining to the robot, operator strategies, control methods, and sensor fusion. Each task was decomposed into subtasks that were classified according to the complexity of the mobility and/or manipulation actions being performed. Performance metrics were calculated regarding the number of task attempts, performance time, and critical incidents, which were then correlated to each team’s interaction methods. The results of this analysis suggest that a combination of HRI characteristics, including balancing the capabilities of the operator with those of the robot and multiple sensor fusion instances with variable reference frames, positively impacted task performance. A set of guidelines for designing HRI with remote, semi-autonomous humanoid robots is proposed based on these results.
Get full access to this article
View all access and purchase options for this article.
References
Amigoni F, Bastianelli E, Berghofer J, et al. (2015) Competitions for benchmarking: Task and functionality scoring complete performance assessment. Robotics & Automation Magazine 22(3): 53–61.
Bullock IM, Ma RR, Dollar AM (2013) A hand-centric classification of human and robot dexterous manipulation. IEEE Transactions on Haptics 6(2): 129–144.
Burke JL (2004) Moonlight in Miami: A field study of human-robot interaction in the context of an urban search and rescue disaster response training exercise. Master’s Thesis, University of South Florida, USA. Available at: http://scholarcommons.usf.edu/etd/973.
Carlson J, Murphy RR (2005) How UGVs physically fail in the field. IEEE Transactions on Robotics 21(3): 423–437.
DARPA (2014) DRC trials score analysis. Available at: http://www.theroboticschallenge.org/dashboard/scoreboard/DRCTrialsScoresAnalysisAnonymousv11DISTAR22423.pdf (accessed March 2014).
DARPA (2015) DRC finals rule book. Available at: http://archive.darpa.mil/roboticschallenge/sites/default/files/docs/2015_04_09_DRC_Finals_Rule_Book_DISTAR_24388.pdf.
DeDonato M, Dimitrov V, Du R, et al. (2015) Human-in-the-loop control of a humanoid robot for disaster response: A report from the DARPA Robotics Challenge Trials. Journal of Field Robotics 32(2): 275–292.
Endsley MR (1995) Toward a theory of situation awareness in dynamic systems. In: Parasuraman R, Mouloua M (eds) Human Factors: The Journal of the Human Factors and Ergonomics Society. vol. 37, pp. 32–64. Mahwah, NJ: Lawrence Erlbaum.
Endsley MR (1996) Automation and situation awareness. In: Automation and Human Performance: Theory and Applications pp. 163–181.
Fallon M, Kuindersma S, Karumanchi S, et al. (2015) An architecture for online affordance based perception and whole body planning. Journal of Field Robotics 32(2): 229–254.
Fleiss JL (1981) Statistical Methods for Rates and Proportions. New York: John Wiley & Sons.
Huang H-M, Messina E, Albus J (2003) Autonomy level specification for intelligent autonomous vehicles: Interim progress report. In: Proceedings of the Performance Metrics for Intelligent Systems (PerMIS) Workshop, Gaithersburg, MD, 16–18 September 2003.
Johnson M, Bradshaw JM, Feltovich PJ, et al. (2014) Coactive design: Designing support for interdependence in joint activity. Journal of Human-Robot Interaction 3(1): 43–69.
Johnson M, Shrewsbury B, Bertrand S, et al. (2015) Team IHMC’s lessons learned from the DARPA Robotics Challenge Trials. Journal of Field Robotics 32(2): 192–208.
Kohlbrecher S, Romay A, Stumpf A, et al. (2015) Human-robot teaming for rescue missions: Team ViGIR’s approach to the 2013 DARPA Robotics Challenge Trials. Journal of Field Robotics 32(3): 352–377.
Kitano H (2000) RoboCup Rescue: A grand challenge for multi-agent systems. In: Proceedings of the International Conference on Multi-Agent Systems (ICMAS-2000), Boston, MA, 10–12 July 2000, pp.5–12. Washington, DC: IEEE Computer Society.
Landis J, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics (33): 159–174.
Micire MJ (2002) Analysis of robotic platforms used at the World Trade Center disaster. MS Thesis, University of South Florida, FL.
Murphy RR, Burke JL (2005) Up from the rubble: Lessons learned about HRI from search and rescue. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 49(3): 437–441.
Murphy RR, Schreckenghost D (2013) Survey of metrics for human-robot interaction. In: 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) 3–6 March 2013, pp. 197–198. IEEE.
Nagatani K, Kiribayashi S, Okada Y, et al. (2013) Emergency response to the nuclear accident at the Fukushima Daiichi nuclear power plants using mobile rescue robots. Journal of Field Robotics 30(1): 44–63.
Nielsen CW, Goodrich M, Ricks RW (2007) Ecological interfaces for improving mobile robot teleoperation. IEEE Transactions on Robotics 23(5): 927–941.
ROS (2016) Powering the world’s robots. Available at: http://moveit.ros.org/; http://wiki.ros.org/graspit (accessed July 2016).
Okura F, Ueda Y, Sato T, et al. (2013) Teleoperation of mobile robots by generating augmented free-viewpoint images. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3–7 November 2013, pp. 665–671. IEEE.
Olsen DR, Goodrich MA (2003) Metrics for evaluating human-robot interactions. In: Proceedings of PERMIS 2003. 16–18 September 2003, pp.4–11.
Pellenz J, Jacoff A, Kimura T, et al. RoboCup rescue robot league. In: Robot Soccer World Cup 2014 15 July, pp.673–685. Springer International Publishing.
Radford NA, Strawser P, Hambuchen K, et al. (2015) Valkyrie: NASA’s first bipedal humanoid robot. Journal of Field Robotics 32(3): 397–419.
Schreckenghost D, Milam T, Fong T. Measuring performance in real time during remote human-robot operations with adjustable autonomy. IEEE Intelligent Systems 25(5): 36–45.
Settimi A, Pavan C, Varricchio V, et al. (2014) A modular approach for remote operation of humanoid robots in search and rescue scenarios. In: Proceedings of Modeling and Simulation for Autonomous Systems: First International Workshop, MESAS 2014, Rome, Italy, 5–6 May 2014, pp.192–205. Springer International Publishing.
Steinfeld A, Fong T, Kaber D, et al. (2006) Common metrics for human-robot interaction. In: Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction, Salt Lake City, Utah, 2–3 March 2006, pp.33–40. ACM.
Stentz A, Herman H, Kelly A, et al. (2015) Chimp, the CMU Highly Intelligent Mobile Platform. Journal of Field Robotics 32(2): 209–228.
Torricelli D, Gonzalez-Vargas J, Veneman JF, et al. (2015) Benchmarking bipedal locomotion: A unified scheme for humanoids, wearable robots, and humans. Robotics & Automation Magazine 22(3): 103–115.
Yanco HA, Drury JL (2007) Rescuing interfaces: A multi-year study of human-robot interaction at the AAAI Robot Rescue Competition. Autonomous Robots 22(4): 333–352.
Yanco HA, Norton A, Ober W, et al. (2015) Analysis of human-robot interaction at the DARPA Robotics Challenge Trials. Journal of Field Robotics 32(3): 420–444.
Zucker M, Joo S, Grey MX, et al. (2015) A general-purpose system for teleoperation of the DRC-HUBO humanoid robot. Journal of Field Robotics 32(3): 336–351.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: February 9, 2017
Issue published: June 2017
Keywords
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in The International Journal of Robotics Research.
View All Journal MetricsArticle usage*
Total views and downloads: 1337
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 28 view articles Opens in new tab
Crossref: 39
- Design method for a hexapod curling robot with multiple tasks, multiple mechanism topologies, and multiple functions capabilities.
- Human-Robot Team Performance Compared to Full Robot Autonomy in 16 Real-World Search and Rescue Missions: Adaptation of the DARPA Subterranean Challenge
- A survey on legged robots: Advances, technologies and applications
- Visual augmentation of live-streaming images in virtual reality to enhance teleoperation of unmanned ground vehicles
- An efficient grasping shared control architecture for unpredictable and unspecified tasks
- Experimental Assessment of Human–Robot Teaming for Multi-step Remote Manipulation with Expert Operators
- 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN)
- Autonomous Ships: A Thematic Review
- Global footstep planning with greedy and heuristic optimization guided by velocity for biped robot
- 2023 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)
- View More
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
IOM3 members can access this journal content using society membership credentials.
IOM3 members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
