Using the Delphi Technique to Identify Key Elements for Effective and Sustainable Visitor Use Planning Frameworks

Protected areas around the world receive nearly 800 billion visits/year, with international tourism continuing to increase. While protected areas provide necessary benefits to communities and visitors, the increased visitation may negatively impact the resource and the recreational experience, hence the need to manage visitor use in protected areas around the world. This research focused on obtaining information from experts to document their experiences utilizing one visitor use planning framework: Visitor Experience and Resource Protection (VERP). Using the Delphi Technique, 31 experts from seven regions around the world were asked to identify elements necessary for effective visitor management, as well as elements that facilitated or limited success when using VERP. Elements were categorized and rated in terms of importance. Scoring of the final categories was analyzed using Wilcoxon and Median non-parametric statistical tests. Results suggest that planning challenges stem from limitations in organizational capacity to support a long-term, adaptive management process, inferring that VERP may be sufficiently developed, but implementation capacity may not. The results can be used to refine existing frameworks, and to aid in the development of new recreation frameworks.


Introduction
This study focused on identifying the elements of an effective and sustainable visitor management framework to manage visitor use and recreation in protected areas. We explored the perceptions of visitor management experts around the globe, drawing on their knowledge based on past experiences. Outdoor recreation has both positive and negative impacts on protected areas worldwide. Recreation is vital to our society as a whole, as it benefits physical and mental health, family relationships, economic sustainability, and so on (Collins & Brown, 2007). On the other hand, recreation and tourism development is one of the factors leading to the decline of threatened species (Czech, 2000;Scherfose, 2009), and overuse may also cause undesirable conditions for visitors, threatening the quality of the visitor experience (Eagles & McCool, 2002). Various visitor use frameworks have been developed to provide management professionals with tools for (a) defining management objectives, (b) collecting and utilizing resource and visitor data, and (c) monitoring use for long-term and sustainable management of the world's protected areas (Borrie, McCool, & Stankey, 1999; contributes to the expanding body of knowledge surrounding visitation management on a global scale.

Visitor Impact Management (VIM) in Protected Areas on a Global Scale
Protected areas around the world offer countless benefits to nature and society; they offer a diversity of recreation opportunities to visitors, open spaces in a world of intense development, protection of flora and fauna habitat, conservation of cultural and historic resources, and economic benefits to surrounding communities, among many others (Lockwood, Worboys, & Kothari, 2006). With increasing visitation and a growing diversity of recreational opportunities, the dual task of maintaining a high-quality visitor experience and at the same time protecting vital natural and cultural resources has become increasingly complex (Manning & Anderson, 2012;Ruschkowski et al., 2013).
Global protected areas together receive roughly 800 billion visits/year (Balmford et al., 2015), and international travel for tourism has shown virtually uninterrupted growth, reaching 1,087 million international arrivals in 2013 (United Nations World Tourism Organization [UNWTO], 2014). International arrivals are still on the rise, with a projected increase of 3.3% each year from -2030(UNWTO, 2014. Along with the growth in the tourism industry is a change in visitor motivations and behaviors, increasingly so with new innovations in outdoor recreation technologies, capabilities, and changing visitor demographics (Hvenegaard et al., 2012;Lockwood et al., 2006). The increase and diversity of visitors to protected areas may result in resource degradation such as loss of biodiversity, along with unacceptable social impacts that degrade both the visitor experience and the health of ecosystems (Ruschkowski et al., 2013); thus, it is no surprise that managing conservation threats from tourism have been one of the leading responsibilities of protected area managers in recent years (Buckley, 2012).
The ways in which tourism impacts conservation efforts differ markedly across political and social domains (Buckley, 2012), hence the necessity for international collaboration to inform visitor management planning efforts in various settings around the world. The benefits of enhancing international collaboration to discuss visitor and recreation planning include (a) facilitating shared learning that addresses the diversity of approaches to similar challenges, such as developing an effective and sustainable visitor management framework, and (b) encouraging practicing professionals to reflect upon their own experiences in such a way that they can share, learn, and adapt. This in turn may enhance lessons learned and global insights into common yet unique challenges. Providing the platform to share these collected experiences could potentially enhance learning via sharing best practices, management methodology, and implementation techniques.

Visitor Management Frameworks
Visitor management is a growing and increasingly important arena for protected area managers and recreation scientists dedicated to supporting effective management of conservation land to ensure high-quality recreation opportunities as well as natural resource conservation (McCool, 2006). It has been recognized that the utilization of a guiding framework aids managers in making informed decisions through each stage of the management process including data collection, policy, planning, implementation, and monitoring (McCool, Clark, & Stankey, 2007). Country-and agency-specific frameworks for visitor management have been developed, providing various sets of objectives, tools, and experiences (Ruschkowski et al., 2013). McCool et al. (2007) contend that a visitor management framework is . . . a process that involves a sequence of steps that leads managers and planners to explicate the particular issue. A "framework" in this sense does not necessarily lead to formulating "the" answer to an issue, but provides the conceptual basis through which the issue may be successfully resolved [and management strategies be identified and implemented]. (p. 25) Frameworks vary across agencies and nations, yet they all intend to maintain or enhance the integrity of the natural and cultural resources (Nilsen & Tayler, 1997). Some of the more common recreation management frameworks include Recreation Opportunity Spectrum (ROS), Limits of Acceptable Change (LAC), Benefits-Based Management (BBM), and VERP. They all share a set of basic principles, including (a) formulating management objectives with indicators and standards of quality, (b) monitoring of indicators of quality, and (c) identifying steps to implement management practices to maintain the set standards (Manning & Anderson, 2012). However, they each differ in terms of how they address and frame recreation management issues, including the diversity of conditions, values, and management objectives. While the frameworks share core similarities, differences occur in each as a reflection of limitations or dissatisfaction with prior approaches (McCool et al., 2007), hence our focus on just one of the frameworks for the study. Still, each framework is described briefly below.
ROS was one of the first recreation management frameworks developed, setting the stage for future frameworks beginning in 1980. Compared with frameworks proceeding ROS, it is more of a concept than a framework. McCool et al. (2007) describe the underlying concept of ROS. " . . . quality recreational experiences are best assured by providing a range or diversity of opportunities" (p. 50). ROS formalized this idea into a recreation planning framework, and has since been used widely around the world. Due to its longevity and popularity, its effectiveness has been assessed at great length (Dawson, 2001;Wearing & Archer, 2003), hence our decision to focus on a newer and less well-known recreation framework.
LAC was developed by the U.S. Forest Service in response to the increasing need to establish a recreation carrying capacity for wilderness areas in the late 1980s (McCool et al., 2007). The central question of LAC is "how much impact is acceptable and what strategies should be taken to avoid unacceptable impacts" (McCool et al., 2007, p. 71)? Other frameworks that address the same central question were developed around the same time, yet were tailored to fit the needs of different agencies. For instance, VERP has similar concepts to LAC, yet was developed for the management needs of the National Park Service, rather than the Forest Service. VERP is the focus of this study, as it has received less attention than LAC in terms of assessments of outcomes.
BBM is arguably the most recent recreation management framework to be developed (Lee & Driver, 1996). The BBM approach is a result of pressure for the government to be held accountable when addressing the question of what the general public gains from investing in recreation settings (McCool et al., 2007). In other words, BBM considers the benefits that will be achieved from a recreation experience, and manages toward the desired outcome. While BBM has received considerable attention in outdoor recreation research (Allen, 1996;Stein & Lee, 1995), there has not yet been a widely accepted document that describes the steps and processes for managers to follow, thus very few applications have been tried. For this reason, it would be difficult to assess the success of BBM.

Visitor Experience & Resource Protection (VERP)
This study addresses the VERP framework as an example to provide a basis for comparison among expert experiences. VERP is a visitor management framework developed by the U.S. National Park Service, thus has been applied most frequently in U.S. National Parks. However, principles of the framework have also been applied in protected areas internationally (Farrell & Marion, 2010). The VERP handbook (National Park Service [NPS], 1997) outlines the necessary steps for managers to use the framework: (a) assemble an interdisciplinary project team, (b) develop a public involvement strategy, (c) develop statements of park purpose, significance, and primary interpretive themes and identifying planning constraints, (d) analyze park resources and the existing visitor, (e) describe a potential range of visitor experiences and resource conditions (potential prescriptive zones), (f) allocate the potential zones to specific location in the park (prescriptive management zones), (g) select indicators and specify standards for each zone, developing a monitoring plan, (h) monitor resource and social indicators, and (i) take management actions.
VERP and similar decision-making frameworks have been used to inform the development or amendments to General Management Plans (GMPs) in U.S. national parks (McCool et al., 2007), establish a social carrying capacity in Arches National Park (Lawson, Manning, Valliere, & Wang, 2003), determine acceptable visitor behaviors and crowding levels for visitor satisfaction in Acadia National Park (Manning, 2010), assess transportation options and alternatives in Denali National Park , guide trail and campsite monitoring protocols in Yosemite National Park (Schreiner & Leung, 2013), address crowding in Galapagos National Parks (Wallace, 1994), inform the development of a new framework more suitable for Central and South American Protected Areas (Farrell & Marion, 2010), among others. Despite this diversity of applications, and efforts to adopt an integrated approach to visitor management, core problems persist and are still limiting management progress (Farrell & Marion, 2010;Ruschkowski et al., 2013). Hence, it is important to generate knowledge from experts who have used the VERP framework, and learn from various expert perceptions and lessons learned within diverse contexts across the globe.

Delphi Technique
The Delphi Technique is a method "designed to obtain consensus of opinions of a group of experts (via) a series of intensive questionnaires interspersed with controlled opinion feedback" (Dalkey, & Helmer, 1963, p. 458). Since its inception in the 1950s, the Delphi Technique has been used to facilitate dialogue and interactions among experts who cannot interact face-to-face (Gupta & Clarke, 1996;Rowe & Wright, 1999). The goal is often to reach a consensus through (a) generating ideas and/or make decisions regarding difficult or complex management objectives, (b) organizing and structuring communication within a group, and (c) combining individual judgments to address a lack of agreement or incomplete knowledge (Gupta & Clarke, 1996;Powell, 2003). The field of recreation planning is lacking in knowledge exchange among experts in dispersed regions on a global level (Haider, 2006). Because the Delphi Technique facilitates knowledge exchange and idea generation, we believed it would be an effective tool to facilitate dialogue and build consensus among experts regarding VERP.
The Delphi process includes at least two rounds of questionnaires, with the first round of questions being open ended, and each subsequent round being based on the responses of the previous (Ruschkowski et al., 2013). Each Delphi round consists of data collection and analysis, followed by development of new questionnaires and response formats to be shared with the expert panel in the next round (De Urioste-Stone, McLaughlin, & Sanyal, 2006). The first round, also referred to as the "explanation phase" (Ziglio, 1996, p. 9), includes broad/open-ended questions regarding problems, objectives, or solutions (Delbecq, Van de Ven, & Gustafon, 1975;Fletcher & Marchildon, 2014;Powell, 2003). The technique is flexible in nature; it has been used widely in fields such as nursing, business, education, industry, and more broadly in social science and natural resource fields (McKenna, 1994).
The Delphi Technique has been used in many studies to generate ideas and reach consensus. It is most commonly used when research is addressing a complex issue that requires involvement of existing experts from a specific discipline. For example, a 2015 study uses the Delphi technique to reach expert consensus regarding the biosecurity of livestock disease. The authors used 16 livestock disease experts to reach a consensus regarding the relevance of specific biosecurity measures. The Delphi technique was used in this study as a way of facilitating expert dialogue to reach conclusions (Kuster, Cousin, Jemmi, Schüpbach-Regula, & Magouras, 2015). The Delphi Technique has also been used widely in the social sciences of environmental management including land-use conflicts, developing decisions support conflicts, and predictions and adaptations to the impacts of tourism and climate change (Landeta, 2005). A review of studies that use the Delphi technique in the conservation management field identified 36 papers. The papers reviewed suggest that the Delphi technique is an efficient, practical, and systematic approach to addressing complex issues (Mukherjee et al., 2015). These studies provide examples of the various approaches to Delphi, as details and length may differ based on time limitations, reaching saturation, or expert responsiveness (Ruschkowski et al., 2013). The approach used for this study is considered the classic Delphi, where descriptive statistics and facilitated dialogue are used to encourage idea generation and assess movement toward consensus among experts using three separate rounds of questionnaires (Hsu & Sanford, 2007).
Strengths of the technique include (a) widen knowledge through multiple rounds, (b) encourage decision-making, and (c) achieve consensus on topics of uncertainty or little empirical evidence (Gupta & Clarke, 1996;McKenna, 1994;Powell, 2003). The technique has been shown to be an inexpensive way to combine knowledge from various experts who are spatially separated (McKenna, 1994;Mehnen, Mose, & Strijker, 2013;Powell, 2003). It has also proven to aid in facilitating group dialogue and the sharing of knowledge, while maintaining anonymity and minimizing group conflict (Gupta & Clarke, 1996;Stokes, 1997). One of the biggest strengths of the instrument is the ability to capitalize on expert knowledge about a particular topic (Mehnen et al., 2013). It is for these reasons that we employed the Delphi Technique in this study.
The Delphi Technique has certain limitations. Researchers have argued that the ethical consideration of anonymity may reduce expert accountability, resulting in rushed and therefore less valuable insights from the expert panel (Gupta & Clarke, 1996;Powell, 2003). However, the Delphi Technique is also seen as a reliable alternative to generating ideas using focus groups, as panel members can freely express opinions without the potential judgment of others in the group (Mukherjee et al., 2015;Ruschkowski et al., 2013). Others have found that the time commitment required of panel members may lead to members dropping out and/or losing interest (Mehnen et al., 2013;Williams & Webb, 1994). Panel members discontinuing participation has proven to be limiting in some cases due to the relatively small sample sizes of Delphi studies from the beginning, as researchers are often targeting very specific key informants (Ruschkowski et al., 2013). Similarly, the inclusion of strictly expert knowledge can result in loss of valuable ideas from other practicing professionals, harboring potential bias (McKenna, 1994). Last, the sheer complexity of some topics has impacted the ability of experts to move toward consensus (De Urioste-Stone et al., 2006).

Methodology
We chose to use the Delphi Technique as a means of facilitating shared knowledge generation among international protected area management experts regarding planning approaches that will guide effective management of visitation in light of increasing recreational pressures worldwide. The Delphi Technique made this important cooperation possible considering the limited funds and time available to the researchers, and the time investment needed from the expert participants. Using the knowledge of international expert professionals and academics, we aimed to capture important definitions about the topic area being studied, and determine if there was a consensus about essential visitor management planning topics.
This study consisted of three rounds with the expert panelists. For each Delphi round, data were generated and analyzed; materials were developed in a response format; and then the materials were shared with all panel members for their use in responding to the next round (Dalkey & Helmer, 1963;Delbecq et al., 1975;Landeta, 2005;Powell, 2003;Ruschkowski et al., 2013). All forms and letters were reviewed by the University of Maine's Institutional Review Board and pretested to ensure clarity prior to data collection by round (Keeney, Hasson, & McKenna, 2001).
This study utilized Qualtrics Survey Software as a means for contacting and distributing the first two rounds of questionnaires to the expert panelists via e-mail and Internet. Qualtrics is a sophisticated survey software that facilitated the allocation of expert surveys, and allowed for real-time analysis. Qualtrics allowed us to create an e-mail list of the selected experts for efficiently sending invitations, reminders, and thank you letters. Qualtrics made it possible to have real-time data collection and database creation, allowing for efficient data gathering and analysis of international expert opinions. The virtual design facilitated a fast analysis of responses, leading to efficient development of the next round of questionnaires, keeping the entire process to a reasonable time frame and maintaining the interest of the panel experts. The first round was distributed on June 2, 2014. The participants were initially assigned a deadline of 2 weeks for completion with two reminders being sent, and an extension of 2 more weeks. The second round was distributed on July 9, 2014. Due to the low response for Round 2, three extensions were given, along with reminders for each. The third and final round was distributed on September 2, 2014, and was followed by two reminders with one extension of an additional 2 weeks. Qualtrics also allowed selected panelist to opt out of the e-mail list if they chose not to participate. These automatic settings are meant to streamline organization and data management to save time for both the researcher and the respondent.
Qualtrics also allowed the team to maintain confidentiality of all responses; only the research team could see the names of respondents. All information was held on a secure, locked connection, with only the lead researcher having access to the login information.

Participant Selection and Response Rates
The Delphi Technique is based on the inclusion of experts in the topic of study while ameliorating group processes like peer pressure and bias (Plummer & Armitage, 2007). For this study, expert panel members were those who held current knowledge on VIM in protected areas, had international recognition and credibility based on their knowledge of the topic, and represented diverse perspectives to include a wide range of viewpoints. The expert participants for this study were selected using specific criteria selection and chain referral strategies (Emmel, 2013;Patton, 2002). Experts were identified based on their fit within predefined criteria including; (a) considered an expert in the field of visitor management, (b) held international experience, and (c) had professional experience with the VERP framework. Initial selection began by identifying level of involvement in the field of visitor management in protected areas through compiling a literature review to identify experts based on papers published, work accomplished, academic and/or administrative position, and so on. The second step included the referral of additional experts by selected study participants through the process of chain referral (Bennett, 2010;Creswell, 2013). This participant selection strategy of peer referral made it possible to identify a diverse group of experts from various backgrounds, which would have been limited if solely based on existing literature.
Of the 41 experts who were initially identified, 10 did not consider themselves experts of the VERP framework, thus concluding that they could not effectively inform the study. These individuals who no longer met the predefined criteria were removed from the list of possible experts. Therefore, 31 experts were asked to participate. Of those, 17 responded to Round 1 (Table 1), representing over half of the invited experts.
These response rate levels were expected (Keeney et al., 2001;Sharkey, 2001;Williams & Webb, 1994), with similar trends being reported in other Delphi studies (Kaynak, Bloom, & Leibold, 1994;Ruschkowski et al., 2013). However, the 100% response rate in the third and final round suggests that those nine experts who participated in the second and third rounds were highly committed to the study topic. While we recognize that nine is a low number from which to draw conclusions, the nine experts who participated throughout the study are highly knowledgeable and dedicated to their field; so much so that they took the time out of their busy schedules to ensure that this Delphi study can help protected area planning efforts in the future. Research shows that "the Delphi technique is particularly suitable for complex issues where the outcome is not dependent on the sample size of the respondent, but rather on the different perspectives and expertise of respondents and their indirect group interactions" (Mukherjee et al., 2015, p. 15). Thus, our reliance on the dialogue among nine experts is justified. We believe the participant dropout that we observed in the first two rounds occurred in this study for several reasons. First, the study began in June, which is a busy field season for most who were asked to participate, as the majority of identified experts live and work in North America, but conduct research or have experience working in other regions of the world. We predict that many experts were out in the field and/or unable to achieve consistent Internet access due to travels, which would limit their ability to participate. Furthermore, the time commitment required, which averaged to be 1 hr and 6 min per round, may have limited experts' ability and willingness to participate. One way to address this issue would be to offer a longer time period for participants to complete the questionnaire. Another method would be to limit the questions asked. This was a complex study covering a number of topics in Round 1, thus narrowing the study to one focus area would have likely increased participation.

Delphi Round 1
The first round asked panel members to provide (a) a definition for an effective program, (b) a list of elements that should be included in the development of a visitor management framework in general, and (c) the key facilitating and limiting factors of VERP based on their experiences. Demographic information, including professional background and global regions of experience, was also collected for further analysis. An online questionnaire, with mostly open-ended questions, was used to allow the space for panel experts to use their own words, expand on important ideas, and provide rich data. The textual data, along with a demographic profile for each panel member, was transferred from Qualtrics to the NVivo10 database to code member attributes that were meant to aid in analysis. The database also included reflections, memos, and descriptions from the research team that informed the decisions made during coding, ensuring transparency (Miles, Huberman, & Saldana, 2014). The definitions of an effective program provided by the panel members were not coded, as unaltered definitions were compiled into one document to be sent back to the panel members. Qualitative content analysis was used to better understand the diversity of the definitions shared by the experts, helping the research team identify common and unique ideas (Mayring, 2007) that may inform further analysis in proceeding rounds.
To generate categories and descriptions of (a) important elements for a successful visitor management plan and (b) facilitating and limiting factors of VERP, we used a systematic classification process of coding, where we began with a researcher-generated "start list" of provisional codes (Hsieh & Shannon, 2005). These researcher-generated codes were developed through qualitative content analysis (Mayring, 2007), where we counted the number of occurrences that a potential category was mentioned by the panel as a whole. With this, we were able to start with a list of elements that would guide the second cycle of coding. This second cycle was a more inductive approach, using the open-ended questions to further describe the most frequently discussed elements (Miles et al., 2014). The descriptions of all categories were developed using ideas and words provided by the expert panelists, coded, condensed, and then summarized by the researchers.

Delphi Round 2
During the second Delphi round, the expert panel members' received three documents: (a) definitions of an effective program, (b) a list of categories and definitions of elements necessary for effective visitor management planning, and (c) categories developed from shared facilitating and limiting factors of using the VERP framework specifically. Providing the verbatim definitions of an effective program from each expert was meant to give each panel member an idea of the range of definitions provided by other experts in the study. Using the categorized and defined elements that are necessary for effective visitor management planning, the panel members were asked to rank the level of importance of each element by allocating 100 points, assigning the most points to the most important elements. Last, using the list of categories and descriptions of facilitating and limiting factors of VERP, the experts identified the two facilitating and two limiting factors they thought were the most important by assigning a "1" or a "2" next to the chosen factors. The purpose of this was to efficiently identify which factors of VERP were the most facilitating or most limiting according to the expert panelists.

Delphi Round 3
The purpose of this concluding Delphi round was to finalize the rankings of element categories necessary to develop an effective visitor use management framework, while moving toward consensus. The last round was meant to provide the opportunity for the expert panelists to reassess and rerate the categories based on the information provided, along with providing additional comments to explain any potential changes. The form included mean, median, standard deviation, and range for each category to inform members of how the entire panel rated the 15 element categories.
Of the nine response forms sent for Round 3, nine were received, marking 100% response rate for the first time throughout the study. In light of the final ranking, SPSS was used to generate descriptive statistics for this last round. Comparisons in the rankings of the topic categories across rounds were conducted using the non-parametric Wilcoxon test.

Panel Expert Profiles
The majority of the panel members held terminal academic degrees, with 88% of respondents in the first round holding a PhD level degree in a related field, the remaining 12% held a master's-level degree. The mean number of regions where participants of the first round have visitor management experience was 2.05, ranging anywhere from one region to seven regions, marking the high diversity of the panel in terms of geographic experience. Of the 17 panel members who participated in the first round, 59% had experience in one region of the world. Of the nine panel members in Rounds 2 and 3, 67% of respondents reported having experience in one region, but the specific regions varied. Those panel members who have worked in two or more regions represent prior international collaboration, thus contributing a diversity of knowledge and unique experiences. One of the panelists from the first round reported working in all seven regions that were represented in this study. Table 2 highlights the global knowledge of the panel of experts through each Delphi round based on the number of regions where they have worked.
Of the 15 participants who had experience in North America in Round 1, six also had experience working in other regions including Latin America, Europe, Asia, and Africa. The most common region where experts had experience for all Delphi rounds was North America, with Latin America also being well represented. While Africa was highly represented in Round 1, none of experts with experience in Africa continued to participate throughout Rounds 2 and 3. However, each other region remained represented by at least one expert, ensuring a diversity of perspectives throughout the study. The continued diversity of our panel experts throughout the study, as represented in Table 3, allowed us to remain confident that our sample of nine in the second and third rounds captured the body of expertise needed to have an adequate and insightful dialogue about the topic.
Background experiences represented a broad spectrum, ranging from heritage management research and recreation planning, all the way to training new rangers and conservation consulting. Due to the higher proportion of male professionals in the field of recreation management, the panel was comprised of 100% males, limiting expert gender diversity. Each Delphi round generated new ideas and contributed key findings that would guide analysis throughout. Therefore, what follows is a description of the results generated by each Delphi round. Table 4 includes a list of commonalities and uniqueness's identified from the definitions of an effective program provided by the 17 expert panelists who participated in Round 1. While differences emerged, most experts shared in their definition that an effective program should (a) include welldefined goals, (b) achieve positive outcomes, and (c) use resources efficiently. The unique ideas listed in Table 4 were those that were mentioned by only one panel member. Similarly, it was observed that most of the unique ideas emerged from experts who have worked in areas outside of the United States.

Delphi Round 1
Following are selected definitions of an effective program provided by the panel experts. These definitions reflect some of the common and unique ideas shared, and the diversity of views. For instance, Panelist #3 focused on program performance based on implementation characteristics. Meanwhile, Panelist #12 focused on ethics, sustainability, and efficiency, whereas Panelist #15 was more concerned with the program evaluation process. Although definitions differed, most mentioned the necessity to define objectives and standards, along with efficient use of resources and monitoring performance. The variety of definitions of an effective program is a result of such a diverse expert panel.
A program is effective when it is implemented and improved desired conditions. It is effective when it can learn and adapt and change the pathways of change as conditions change. It can alter its vision and desired outcomes. It can sustain itself over time beyond initial funding. Implementation success has six characteristics: (1) Achieves planned outcomes; (2) quality of program or plan design; (3) achieves unplanned positive outcomes; (4) Others model their interventions based on design of program; (5) achieves planned tasks; (6) degree to which stakeholders refer to program as a reference to guide them forward. This is called performance rather than conventional conformance (conformance between tasks planned and tasks completed). (June 8, 2014, Panelist #3, Latin America) Program effectiveness requires specification of goals and objectives, metrics to "operationalize" goals and objectives, long-term monitoring to assess progress towards goals, and adaptation based on monitoring outcomes. (June 9, 2014, Panelist #4, United States) Program effectiveness is the ability for a program to sustainably meet its stated objectives and goals. Ethics, efficiency, practicality, and repeatability must be considered also in conjunction with this program performance appraisal. (June 20, 2014, Panelist #12, United States and Kenya) Program effectiveness is an evaluation process that monitors effectiveness based upon a set of agreed to standards. The standards may be set at an international, national or agency level. The key is that standards are defined and regularly monitored and recorded in order to create a transparent record. Well-developed definitions of standards also need to exist in order to allow effective measurement. (June 26, 2014, Panelist #15, United States, Latin America, Asia, Africa) The first round generated a total 156 elements that experts perceived as important to be considered when using a guiding framework to achieve effective and sustainable visitor use planning. Of those 156 elements, 15 unique categories emerged. Table 5 lists the categorized elements, along with a description of each category that was developed by the research team using the words and ideas provided by the panel members. These categories represent the diversity of the 156 elements that were originally shared. Experts were generally consistent with themes, yet provided diverse descriptions for each, marking the importance of terminology for both naming the categories and describing them in a way that would represent the ideas shared by each panel member. For instance, many experts described social data in different ways; therefore, the description includes spatial analysis, visitor impacts, visitor concessions, budgets, and market segments to address a wide range of the observed social data interpretations. Category descriptions intentionally cover a wide range of possible meanings.
Last, Table 6 lists the aspects of VERP that facilitated or constrained success based on expert experiences working with the framework. We infer that experts perceive that the

Delphi Round 2
Of the 12 limiting and facilitating factors of VERP from Round 1, experts identified the two most important for each category in Round 2. Table 7 shows that experts perceive the most important facilitating factors of VERP to be the underlying motivations behind using such a framework. The purposes of establishing a visitor use framework are to (a) guide best management practices based on a systematic, datadriven process and (b) ensure that management strategies are able to efficiently respond to emerging management challenges. Experts ranked "defining clear objectives" as the third most facilitating factor of VERP, with "the development of measurable indicators and standards" ranked fourth. Not surprisingly, "strong political support" was not perceived to be a facilitating factor of VERP by any of the panel experts. Table 8 remains consistent with our interpretation that the most prevalent limiting factors of VERP are those that are outside the domain of the framework itself. It is not the principles behind the planning frameworks that hinder success, but rather the organizational structure wherein such frameworks operate. Through identifying the significance of each element based on expert rankings, we remain confident that VERP's success is based on the context in which it is implemented, thus establishing the necessity of having a strong organizational capacity to ensure program effectiveness. As shown in Table 8, the most prevalent limiting factors of VERP as reported by the panel of experts include "limited funding and resources" and "management assumptions and biases." Following close behind these as limiting factors of VERP are "agency turnover" and "failure to clearly define objectives." Last, none of the experts perceived that "methods for developing alternatives," "VERP planning leading to increased facility development," and "an outdated planning model," were limitations to VERP planning outcomes in their experiences.
Results from Round 2 also established the initial expert rankings of elements for a successful visitor management  Note. Panel members were asked to self-identify their regions of expertise in the first round.

Category Description
Assessment It is important for planners and managers to assess a situation before understanding how to better manage it. An assessment includes a description of existing conditions and experiences of visitors. Therefore, an effective visitor management plan should include an assessment of current conditions, issues, threats, visitor expectations, and visitor characteristics to better understand the conflicts and how to fix them. An assessment also includes identifying the most pressing resource protection and visitor experience issues. All assessment should be based off of empirical and anecdotal data on both public and expert/staff input. Carrying capacity A visitor management plan should include an assessment of the carrying capacity of the park for visitor use. This should include both resource and experiential dimensions, along with considerations of what should happen if carrying capacity is violated. Public participation An effective plan not only needs to plan how it will educate the public regarding its specific initiatives (interpretation), but it also must include public opinion in the planning process itself. Without public participation, planners and managers would not know the expectations of the citizens that the plan will likely impact. Citizen participation is not a one-time procedure. The planning process for a successful visitor management plan includes an ongoing process of public review in updating plan processes. This includes an information management and communication strategy for sharing the necessary information with the public effectively.

Context
Visitor management plans need to take into account the specific context in which it is being used. Context includes the ecological, social, and managerial atmosphere in which the plan will be implemented. Ecological: The specific nature of the visitor use issue. What sort of environment is being used and managed?
What is the time frame in which the conflict should be alleviated? Social: What is the nature of the conflict the plan is meant to solve? Is it a conflict between different user groups? A conflict between visitor and managers? A conflict between the users and the physical environment? What are the users' opinions and attitudes? Managerial: How does the specific managerial organization work? Is it an NGO, government organization, many groups working together? What assumptions and constraints are guiding their decisions? Defining future conditions An effective visitor management plan should include a prescription of desired conditions, or clearly defined outcomes. These prescriptions can serve as a basis for what the planners are trying to achieve based on both social and natural desired outcomes. One way in which managers prescribe future conditions is through creating zones of the varying desired outcomes for each. Desired future conditions may not be the same for every stakeholder, hence the importance of developing alternatives. Alternatives take into account the different visions or concepts for the future of the park, and act as a method for including all stakeholder opinions throughout the planning process.

Financial planning
Having the financial resources to successfully plan for, implement, and monitor a visitor management plan often depends on the managerial context in which the plan is conducted. Financial stresses can limit the resources allocated to visitor management, hence impacting the success of the program. Planning for financial sustainability may limit the possibility of depleting resources. A method used to understand the finances necessary to complete a project includes creating a calendar of activities that includes the necessary components of implementation and monitoring, thus creating a budget by which the managers can refer to. Governance and organizational strategy Make clear the roles of responsibilities of the stakeholders involved, whether it is on the level of an organization or an individual. This would include allocating responsibilities, creating a timeline to ensure that each actor stays on track, making a capacity building strategy for the future of the protected area, and the sustainability of the plan. Implementation strategic planning Planning how one will actually implement a visitor management plan means trying to understand all of the consequences of the plan, whether they are intended or not. Implementation includes creating specific actions and regulations to address cultural and natural resource concerns, and how those changes will be controlled. Successful implementation takes into account both the technical and social aspects necessary to enhance the success of the program. This plan should include both immediate actions, and those taken over the life of the plan, include material and personnel costs, and responsible parties. Implementation planning also demonstrates managerial awareness of how their actions will impact visitor experiences, diversity of visitor uses, and social relationships/conflicts in the area. Interpretation While the focus of visitor management plans are often in regards to a specific conflict, educating the public of the conflict and what they can do to minimize the conflict is just as important. Planning interpretive themes, methods of communication, and proposals for improved new products are all a part of planning for interpretation to the visitors.

Monitoring
Once a plan has been implemented, continued monitoring of the outcomes helps determine if desired conditions are being reached. Monitoring outcomes are used to determine if the standards of quality are being maintained and assess the effects of management changes. Monitoring takes long-term commitment and funding. (continued)

Category Description
Objectives, indicators, and standards Management objectives are often broad statements that describe the type of recreation conditions that will be provided, including the type and intensity of different management actions. Indicators, on the other hand, are more specific and measurable. They include quantifiable empirical measures of the management objectives. Standards are the minimum acceptable condition of those defined indicators. The process of identifying these objectives, indicators, and standards should be scientifically based.

Response planning
An effective visitor management plan should include a stepped or threshold response to conditions to violate or approach violating standards. Along with this, a list of potentially effective strategies for responding to visitor quality and resource conditions that are deteriorating should be included. This includes a strategy for evaluating any new or unforeseen visitor activities. An effective visitor management plan should include recommended actions ranging from indirect and unobtrusive actions, to direct and potentially obtrusive. Review of literature, past plans, and policies To move forward in the planning process, one must understand what has been offered and attempted in the past, and other policies taking place in the present. A clear understanding of any policies that may be necessary for the plan to be successfully implemented is also important. The plan should include a review of literature to identify relevant information about the specific issue, and to understand the influence of causal and influential factors. This should aid in the integration of other land uses, communities, plans, and efforts, both current and future.

Social data
An effective visitor management plan should include spatial knowledge of visitor distribution across space and time. It should also include spatial knowledge of visitor services and facilities improvement projects across space. Visitor impact measurements and a budget that directly connects investments in facilities and services with desired changes based on visitor satisfaction. Basic data on present visitor markets along with additional markets that managers hope to target. Zones Many visitor management plans include zoning mapping to determine future conditions, objectives, and indicators and standards of quality. Zones are often prescriptive, meaning that they are used to determine future conditions, rather than assessing current conditions. Zones are used to determine how to manage different sections of the park depending on their intended use. Zones can be organized based on physical environment, visitor use, and development. program. The use of measurable indicators and standards was ranked as the most important element category (Table  9). The element category that was ranked at number two was monitoring; in addition, this was the element that experts agreed upon the most, as it had the lowest standard deviation in Round 2. Interpretation was ranked as the least important element to be included in visitor management planning efforts. Table 9 describes these results as they compare with the rankings from Round 3.

Delphi Round 3
The third Delphi round provided an opportunity for experts to modify their rankings based on comments and suggestions from their peer experts to see if experts moved closer to a consensus regarding element categories necessary for effective visitor management planning. Table 9 displays the descriptive statistics for Rounds 2 and 3. Five of the categories decreased in standard deviation in the third round, meaning they moved toward a greater consensus among experts in terms of relative importance of each element category. The five elements that moved toward consensus include citizen participation, context, financial planning, implementation strategy, and the use of indicators and standards. The category that moved toward consensus the most between rounds was financial planning. The category that moved the furthest away from consensus between rounds was monitoring.
Results of the Wilcoxon test showed no statistical differences between the rankings for Round 2 and Round 3. As is shown in Table 9, six of the 15 categories were assigned the same rank between Rounds 2 and 3. These categories include indicators and standards, zones, financial planning, carrying capacity, organizational strategy and interpretation. Of those six that had no change in rank, five remained consistent in their Median score, and were those categories that were identified as the least important. The last element category that remained the same and was ranked the most important in both rounds was indicators and standards, suggesting that experts have stable opinions about the elements that are on either end of the spectrum, while those that fall in the middle are more variable.

The Delphi Approach
Given the dispersed knowledge of visitor management planning around the world, and the need to share expert experiences and perceptions regarding visitation planning in protected areas, the Delphi Technique was the appropriate method for the generation of ideas surrounding visitor use planning on an international level. The number of participants in the first round, combined with thoughtful and rich responses from panel members, allowed for trustworthy qualitative analysis. Using the Delphi Technique allowed for shared idea generation among experts that likely would not have had the opportunity to exchange ideas otherwise. Despite one panel member voicing their concern of inherent pressure to move toward the norm, panel members were only asked to modify their ranking if they found reason to reconsider, with no obligation to do so. Similarly, element descriptions were redefined for the third round based on the expert comments; thus, the majority (78%) of participants found it necessary to change their rankings based on new information provided. While this is important, arguably more so was the rich information generated during Round 1. With the openended form of questioning, participants were free to share their ideas in a safe environment that would ensure confidentiality, in turn gaining thoughtful and honest responses from each panel member.
Limitations of the Delphi Technique for this study may have been expert selection and the complexity of the topic. It was unclear if the selected panel members captured the diversity of visitor management experts around the world, as the use of chain referral may have overlooked some experts altogether. The wealth of published journal articles addressing visitor management all over the world may have resulted in missing those experts who did not publish in the English language. Most panel members were North American-based researchers with international experience. Due to the higher proportion of North American experts, and the researchers staying close to the words of the expert panelists to describe element categories, some category descriptions may rely more heavily on examples within the U.S. National Park Service. While this may limit the global perspective, the higher proportion of North American experts was expected. We suggest that had we focused on visitor use management  Ranked "most important" by the majority of experts Adaptive management: Due to the flexibility of the VERP framework and process, including management results, public input, and continued monitoring, there is opportunity for change where needed. The framework allows for freedom, allowing the managers to make important visitor use and resource conservation decisions based on the specifics of an individual situation, and based on changes that emerge over time. The assessment and evaluation components of VERP are mostly formative leaving room for in-process, adaptive adjustments. Ranked "second most important" by the majority of experts Science-based process: VERP is a science-based process that allows rigorous collection of visitor use data and attitudes to help ensure that decisions are data driven and defensible. There are steps for inventory and assessment, creation of measurable metrics to assess social and resource conditions related to goals and objectives, and steps for monitoring and changing methods as part of a larger ongoing process. This process is also linked to a clear definition of resources carrying capacity to justify and support the uses that it may be planning for. Using a scientific process, VERP promotes standardization, making it possible to compare across management units through creating common definitions and a step-by-step, systematic management process. This makes VERP a reliable, observable, and transparent process that fits well with the National Park Service (NPS) philosophy of conserving resources for future generations while managing for a quality visitor experience.
Note. VERP = Visitor Experience and Resource Protection.
in general, the observed bias would have been limiting. However, due to our focus on VERP specifically, the North American emphasis was necessary, ensuring that experts understood the context in which VERP was originally developed before being applied in an international setting. We were also limited by our reliance on the Internet and Qualtrics software, as some experts were working in the field with little or no computer access, and others missed invitations due to Qualtrics generated emails that may have been automatically directed into spam inboxes. Another limitation could be the reliance on researcher interpretation to facilitate dialogue among experts regarding such a complex topic, as this may result in researcher bias or misunderstanding (Ruschkowski et al., 2013). We argue that the potential bias was likely mitigated as a result of the cyclical nature of the Delphi process with space for comments, changes, and reconsideration of ideas. However, despite our best efforts to stay close to the words and ideas shared by the expert panelists, one expert commented that one of the descriptions provided by the research team did not capture the essential ideas, thereby  potentially limiting the panelists' ability to properly rank its importance.

Lessons Learned and Management Implications
The results of the Delphi study suggest that there is agreement that the 15 element categories identified by experts are important for inclusion in a visitor management planning process, but they were not all equal in terms of importance (Table 9). The two element categories that were continuously ranked within the top most important for both rounds are (a) indicators and standards, and (b) monitoring. These are both well-known best principles and practices among management practitioners (Eagles & McCool, 2002;Manning & Anderson, 2012;McCool et al., 2007;Nilsen & Tayler, 1997). While monitoring moved from Rank 2 to 3 between rounds, it still remained highly important to experts. The least important categories-interpretation and organizational strategy-were identified as either confusing or irrelevant (interpretation), or outside of the domain of planning frameworks themselves, addressing issues that surpass the scope of this study (organizational strategy). While organizational strategy was considered among the least important categories to inform visitor management planning, it was identified as one of the most limiting factors of VERP specifically. Seemingly counterintuitive, this in fact suggests that experts recognize that the organizational atmosphere in which VERP is implemented highly impacts its outcomes, sustainability, and effectiveness (McCool et al., 2007), yet this does not fall within the domain of visitor management planning, and is not necessarily a weakness of the framework itself. With monitoring being an important element category, having a high level of organizational capacity becomes increasingly important. Due to the long-term commitment of staff and resources that successful monitoring requires, the capacity of the organization to support such efforts is essential (Eagles & McCool, 2002;Manning & Anderson, 2012;McCool et al., 2007). This study provided the first step in broadly identifying the elements that are most important for effective visitor management frameworks, discovering that the agency that is tasked with implementing such frameworks may need to be the next area of research. Further research should move toward identifying what it is about the organization itself that limits or facilitates success, rather than the framework that is being implemented. This Delphi study established that the challenges associated with visitor management in general and with VERP specifically are those factors that are out of the domain of the planning approaches themselves. This was determined through both the rankings generated for element categories of visitor management planning in general, and through rating the facilitating and limiting factors of VERP. Hence, future visitor management frameworks should be developed with the careful consideration of the political, social, and economic constraints that surround such planning efforts, and their ability to limit success in some cases. Despite the various socioeconomic and biophysical contexts where planning efforts have occurred, they are most often limited by lack of funding and organizational strategy; which in turn limits the capacity of staff to make informed management decisions that can then be successfully implemented. Thus, while common obstacles have been identified, approaches to solving those challenges would vary based on the nature of the protected area's biophysical and socioeconomic setting, the management objectives, and the knowledge base and values of the managers themselves. Through better understanding the role of the implementing organization, we may move closer to understanding agency capacity to initiate and support a visitor management planning effort.
There is continued discussion among professionals regarding the development of new visitor management frameworks. This research contributes to that conversation, bringing together new ideas and lessons learned to inform the ongoing process of developing effective and sustainable visitor management frameworks. New developments are currently moving toward one set of guidelines meant to enhance collaboration among agencies, ensuring each organization use the same terminology and methodologies to enhance the commitment to visitor use management efforts (Interagency Visitor Use Management Council, 2015). In accordance with efforts to reorganize visitor management approaches across agencies, this research suggests that the frameworks currently developed may be sufficient, but rather the implementing agency itself may require attention to determine approaches for organizational assessment (Lusthaus, Adrien, Anderson, & Carden, 1999) prior to implementing any sort of plan. Due to the successful applications of VERP in some cases (Jacobi & Manning, 1999;Lah, 2000;, we see that when the implementing agency has the capacity, VERP achieves intended outcomes. Other studies have displayed that VERP may be easily integrated with other planning processes such as transportation initiatives (Budruk, Laven, Manning, Valliere, & Hof, 2001;, thus providing further evidence that the framework itself may not be what is limiting current visitor management approaches. We suggest that the development of new visitor management frameworks should first consider the organizational environment in which it will be implemented, and careful consideration of organizational capacity should be a top priority. Thus, framework development may be transitioning toward capacity development for the agencies themselves. Organizational assessments need not be a long, drawn-out process, but should be a systematic process meant to determine staff, funding, and time capacities (Lusthaus et al., 1999) to inform if a long-term, adaptive management planning effort would be possible.
In conclusion, the Delphi Technique employed via the Internet was successful in achieving the goals of this study, with some limitations. The technique could be successfully employed for other recreation management studies, taking into consideration the lessons learned from this project. Examples of enhancing the Delphi Technique for future studies include using caution when choosing survey software to assist in participant recruitment and database generation. Another suggestion would be to allow for sufficient time to identify experts using the method of chain referral, as rushing the process will limit the expertise and diversity of experiences that are necessary for an impactful Delphi study. We also believe that the most significant results emerged in Rounds 1 and 2, where open-ended questions allowed for a richer understanding and sharing of expert perceptions. Such studies that may benefit from the Delphi Technique could be those that expand upon this research with a focus in determining important organizational components to support the implementation of long-term, adaptive management plans. Another application could be to emphasize the focus of best management practices to determine if best practices are agreed upon internationally, as visitor management being a data-driven, systematic process was one of the most important elements of visitor use planning in this study. Lastly, this study has informed the direction of future research in the field of protected area management through reemphasizing the need for organizational assessment and strategy relating to agency success in supporting a long-term, adaptive planning process. This is an essential step toward guiding visitor management approaches during a time of intensified recreation impacts in protected areas around the globe, and through the process of rethinking how to approach visitor management framework development for the future.