Using consensus methods to standardise judgement-based guidelines required for player management decision-making processes: A case study in professional rugby union

Standards are pivotal for generating the evidence required to manage players in professional sport environments like rugby union. Resultantly, using a three-step qualitative approach, this study aimed to formulate a consensus as a subjective standard for evidence generation pertaining to player management. The consensus statement intended to identify evidence on peaks/troughs in player external training loads using Global Positioning System (GPS)-based information in the High-Performance Unit (HPU) of a Gallagher Premiership rugby union club. Initially, a systematic review adhering to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) framework was conducted to unravel the factors considered (literature-based cues) when identifying peaks/troughs in player external training loads using GPS information. Next, thematic analysis conducted on the data obtained from 7 semi-structured interviews with HPU staff highlighted that they consider 6 factors with 38 elements (practitioner-based cues) during player external training load management. Thereafter, guided by the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument and by utilising selected elements representing 4/6 factors (healthy player, GPS information, longitudinal durations and practitioner judgements on information), a consensus among practitioners for identifying peaks/troughs in player external training loads was developed with the participation of five HPU members using the nominal group technique (NGT). Practitioners reached an agreement with regard to 12 indicators to subjectively identify peaks/troughs in player external training loads within the considered environment.


Introduction
In recent times, there appears to be a significant growth in the use of data for decision-making pertaining to player management across different sport organisations.However, since data has no meaning in isolation, 1 it must transform to its higher-order dimensions like information and evidence to meaningfully support decision-making.As a theoretical construct, Dammann 2 proposed a health informatics framework that illustrated the unidirectional transitioning between data, information, evidence and knowledge.Importantly, this model suggests that information is created when data is contextualised, information compared to a standard formulates evidence and repeated evidence or the success, and agreement arising from the use of evidence to test hypotheses creates knowledge. 2In professional sport environments, practitioners often rely on the evidence generated from different information sources (e.g.medical information, training information) when making decisions relating to player management. 3][6] In such contexts, under the absence of adequate standards or research-based guidelines, practitioners engaged in a collective decision-making process may create evidence from information sources based primarily on their own personal biases rather than aligning with holistic organisational objectives. 7This in turn has the potential to create noise in decision-making relating to player management.
From an applied viewpoint, in an environment like rugby union, we have illustrated that collective decision-making processes occur when practitioners make judgements regarding player external training load management. 6One key outcome of this process is to identify and act upon instances that a player experiences higher than normal (peak) or lower than normal (trough) external training loads. 6][10][11][12] Practically, prior research illustrates that the information contextualised from microtechnology data (e.g.Global Positioning System (GPS)) play a key role in creating the evidence needed to manage the external training loads of rugby union players in professional environments. 3In such contexts, from a physiological external training load monitoring perspective, 13 under the presence of adequate standards, GPS-based information can notify a practitioner when a player experiences an external training load peak or trough.While the final practitioner judgement on whether such evidence truly transitions to knowledge (e.g.likely training-induced adaptations due to repeated peaks) may depend on additional contexts to the decision (e.g.practitioner experiences, internal training load information), such a notification could still help alert potential risks and warrants investigation during player training load management.
Moreover, in a team sport environment like rugby union, the evidence necessary to manage the preparation and performance of an athlete is unique to that specific organisation.That is because each sporting club has its own operational model (e.g.financial strategy, playing style and training philosophy) leading to a distinct set of organisational goals. 14Accordingly, on a micro-level, the standards required for generating evidence from information sources (e.g.identifying peaks/troughs in external training loads from GPS information) are subjective to the considered sporting environment.Therefore, unless guided objectively by research-based evidence, the absence of a relevant subjective standard (i.e.indicators signifying the collective agreement among practitioners) that aligns with the organisational goals may result in individuals formulating evidence for the existence of player external training load peaks/troughs from GPS-based information based fundamentally on their individual biases and beliefs.
Clinical literature discusses how consensus development methods can organise subjective judgments among practitioners to create judgement-based guidelines for clinical decision-making and to overcome challenges arising due to the lack of research-based evidence. 15Therefore, consensus development techniques appear to be suitable for developing subjective standards that illustrate the agreement among a group of individuals associated with a collective decisionmaking process.Moreover, their use has already emerged in sport where consensus statements are being defined as guidelines for managing athletes. 16,17Furthermore, within clinical and sport literature, Delphi, 18 consensus development conference 10 and nominal group techniques (NGTs) 19 are the primary methods utilised to develop a consensus. 20An aspect common to all these methods is the use of cues (i.e. the factors or dimensions considered by the group when making their decisions) during their execution. 15Clinical literature discusses three main types of cues: literature-based (derived from existing literature on the topic), practitionerbased (from views of the consensus members) and contextual (factors specific to the environment, e.g.restrictions due to resources). 20Since the cues selected in the consensus question could bias the final outcome, it is hence important that they are selected in a systematic manner prior to a consensus development session. 20lthough consensus statements derived as judgementbased guidelines are emerging in sport literature, it is still unclear if practitioners within a professional sport environment can systematically reach a consensus and articulate subjective standards that signify their collective agreement during decision-making.Therefore, this article aims to bridge that gap in the sport literature by exploring the formulation of a consensus as a subjective standard among practitioners within the performance department of a professional rugby union club (signifying the collective agreement aligning with the holistic organisational goals) to create evidence on the existence of player external training load peaks/troughs using GPS-based information.

Case study design
This study focuses on the external training load management of players in the High-Performance Unit (HPU) of a professional rugby union club competing in the Gallagher Premiership in England.The specific details of the relevant HPU have been presented previously 6 with outcomes from optimising an information flow necessary for player management. 21Furthermore, ethical approval for the current study was obtained from the affiliated university.
Figure 1 illustrates a high-level overview of the steps undertaken to develop the consensus for identifying external training load peaks/troughs from GPS-based information within this case study.Moreover, the Appraisal of Guidelines for Research and Evaluation (AGREE) II statement was used as a reference standard to guide the consensus development process. 22

Identifying the cues
As specified previously, recommendations on the methods for consensus development when formulating clinical guidelines clearly illustrate that the cues used in the consensus questions could influence the individual and group judgements. 15,20Therefore, our first task was to articulate the three types of cues (literature-based, practitioner-based and contextual) that could be utilised by the practitioners when developing a consensus to identify peaks/troughs in the external training loads experienced by rugby union players.
Literature-based cues.We first conducted a systematic review adhering to the 2009 Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement to extract any literature-based cues relating to the considered consensus development. 23For conducting the systematic review, in relation to the aims of the study, we framed the question as: 'In sport literature, for field-based invasion team sport athletes, how are peaks and troughs in the training loads relating to injury risks, performance adaptations, wellness, fatigue, and illness identified using the information derived from Global Positioning System (GPS) based metrices?'.
Using Boolean operators, articles published from the year 2000 to September 2021 were extracted from EBSCO (Academic Search Complete, SPORTDiscus, MEDLINE and CINAHL Complete), PubMed and Scopus databases.Titles and abstracts of the retrieved papers were independently reviewed by two authors (JR and MZ) against the specified inclusion and exclusion criteria (refer to Supplementary material 1) and the articles not meeting these requirements were excluded prior to fulltext review.Selected full-text studies were further analysed against the inclusion criteria and a search of references within the selected studies were conducted to ensure that all relevant articles for the review were included.All articles were assessed for quality independently by authors JR and MZ based on the quality index proposed by Downs and Black 24 for randomised and non-randomised studies.A consensus was reached regarding any disagreements between the two authors relating to the study selections or quality assessments after discussions.Data synthesis was conducted by author JR and corresponding results were discussed among the authors prior to making inferences.All information on the search terms used, inclusion criteria and detailed results from the review have been provided separately in Supplementary material 1.From the findings, any relevant literature-based cues for the consensus question were identified.Additionally, as per recent expert opinions for consensus development in sports, 25 the systematic review results were shared with the consensus group 1 week prior to the first consensus development session.
Practitioner-based and contextual cues.From a practical viewpoint, practitioners consider different factors (cues) when making decisions relating to player training load management.Such cues or factors form the different contextual layers that bind a final decision relating to the external training load management of rugby union players within a considered environment.Hence, author JR conducted semi-structured interviews (each less than 30 minutes) with all HPU staff members (three physiotherapists, three strength and conditioning coaches and one sports scientist) responsible for the players' external training load management (during the timeframe of the study) within the considered environment to identify factors they considered during decision-making.Details of the interviewees have been outlined in Table 1.The relevant open-ended questions used in the semi-structured interviews have been provided in Table 2.They were specifically designed to extract details on the practitioner interaction with GPS-based information during player external training load management.Interview sessions were audio recorded and transcribed verbatim by the lead researcher prior to data analysis.For data analysis, the six-phase thematic analysis technique introduced by Braun and Clarke 26 was implemented to unravel the specific factors (as themes) that practitioners consider during player external training load management.For theme development, the lead author first developed semantic and latent codes from each interview transcript.The developed coding structure was evaluated independently by the authors DW and GR and agreement was reached between them and the lead author prior to the theme development stage.Next, based on the inherent features of the codes and guided by an inductive approach, lower-order themes depicting elements of the different factors were unravelled from the developed codes.Thereafter, those lower-order themes were grouped to form the final higher-order themes depicting the high-level factors that practitioners consider during player external training load management.Moreover, the developed themes were validated independently by authors DW and GR.
In the current study, no separate contextual cues were defined as there were no separate case study-specific factors (e.g.resource limitations) that needed consideration for the consensus development.Furthermore, guided by existing literature, a list of definitions (Supplementary material 2) relating to the study was developed based on the agreement between the authors, which were shared with the consensus group prior to the consensus development sessions.

Developing the subjective standard (consensus)
Methods for developing consensus.The selected cues were used to define the consensus questions needed to develop the subjective standard (consensus) to identify peaks/ troughs in player external training loads using GPS information.As justified previously, the aim of this study was to develop a consensus statement that was subjective to the case study organisation.Consequently, it was impractical to invite external experts for consensus development as they would not be aware of the operational dynamics and wider organisational objectives of the considered environment.Therefore, the aim was to develop the subjective standard from the expertise within the considered club such that it represented the collective agreement of the HPU.Moreover, among the three consensus methods introduced previously, Nominal Group Technique (NGT), which focuses on obtaining consensus among a group of individuals using face-to-face meetings 19,27 was well suited for the current study as it enabled a rigorous dialogue and equal contribution between practitioners during consensus formulation.Therefore, NGT was selected as the method to develop the consensus.Furthermore, based on the guidelines of senior management team at the club, five HPU members (Head of Medical, Head of Strength and Conditioning, Head of Sports Science, First Team physiotherapist and Senior Academy Strength and Conditioning Coach) from the seven individuals previously considered for the semi-structured interviews (representing the expertise within the club relating to the consensus question) were selected for the consensus development.The relevant participant characteristics have been provided in Table 1.
Determining consensus.In the current study, a dichotomous scale (agree/disagree) was used to rate the agreement or disagreement for an indicator by each member of the consensus group.We utilised guidelines available in content validity (CV) research when defining the exact criteria for reaching consensus. 28Similar findings from CV research have also been used previously by sport researchers when developing consensus statements related to sports. 17ence, aligning with the guidelines by Lynn, 28 for the current study, consensus for an indicator relating to each question was reached if all five members voted to either agree or disagree with it.
Conducting the nominal group sessions.Author JR led the nominal group meetings (relevant steps are outlined in Figure 2), and each session lasted a maximum duration of 2 hours and four such sessions were used to develop the subjective standards.The process was repeated separately to derive two independent consensus statements (i.e. for peaks and troughs).Initially, adhering to a time limit of 30-minutes, the five participants silently recorded the indicators (short phrases) that would illustrate a peak/trough in the player's external loads from GPS-based information.Subsequently, based on a round-robin format, the participants recorded the generated indicators on an online Microsoft Excel sheet.By considering one indicator at a time, the members then discussed them in detail and any common indicators among the recorded ones were grouped.The host (JR) intervened to move to the next item once the conversation regarding an indicator reached saturation (i.e.no more comments from the consensus team).Following the discussions, author JR recorded the indicators on a Google Form and the members used it to perform the first round of ratings.The indicators reaching consensus (all members agree or disagree) after round one rating were directly added to the final list and the ones not reaching consensus were taken back into the discussion.
A further round of discussions and voting was conducted on the indicators not reaching consensus from the initial round.
Resultantly, the indicators reaching consensus after round two were added to the final list If there were any items still not meeting consensus, based on a collective discussion with the nominal group, they were either modified, rejected or accepted to generate the finalised consensus statements.

Literature-based cues (systematic review)
Study selection.In the next sections of the results, we will only discuss the findings from the systematic review that directly relate to our research question.However, we have provided full details of the accumulated results from the systematic review in the Supplementary material 1.The PRISMA flow diagram shown in Figure 3 illustrates how the final list of 22 studies included for results synthesis were streamlined from the 3265 articles identified through database and reference list searching.Initially, there was 96.9% agreement between authors JR and MZ following initial screening of titles and abstracts.Additionally, any conflicts arising during the study selection were resolved after discussions between JR, MZ and GR.
Quality of selected studies.The finalised mean quality rating of the 22 studies included in the review from the 2 authors was 15.09 ± 1.66 out of the 32 possible points (range: 13-18).According to the quality ranges used by Hooper et al., 29  The relevant indicators signifying potential peaks/ troughs in player external training loads relating to injury risks and performance adaptations were articulated based on statistical methods like hypothesis tests, analysis of variance, linear, multiple and logistic regression, generalised additive models, generalised linear models and generalised estimating equations.Moreover, when association was the statistical aim (i.e. to determine the relationship between external training load measure/s and an outcome such as injury), it was possible to assess the strength of evidence on peaks/troughs generated from the latter statistical models based on outcome statistics like p-values, correlation coefficients, effect size, odds ratios (ORs), relative risks (RR), hazard ratios (HRs) and incidence rate ratios (IRRs) utilised within the studies.Similarly, prediction results were assessed using statistical processes such as area under the receiver operating characteristic curve (AUC-ROC), sensitivity, specificity and likelihood ratios.
For instance, to signify a peak, the results from a generalised estimating equation illustrated that there is a very likely harmful effect on time-loss overuse injuries if the 2-week cumulative total distance was greater than 59.185 m (OR = 2.25, 90% CI = 1.17-4.34)for a sample of 35 professional football players in the Netherlands. 30tracting literature-based cues from the systematic review results.The study designs of articles examined in the  review and their corresponding statistical outcomes were not consistent across the different studies.This made it challenging to relate the findings to the case study environment considered in the current article.As an example, an outcome specifying that 3-weekly sprint distances >1.453 m is associated with a greater injury risk when compared with <864 m (OR = 3.667, 95% CI = 0.88-15.21,p = 0.074) for Australian footballers, becomes challenging to interpret as the strength of evidence is defined in relative terms. 31Therefore, it wasn't possible to define specific indicators for identifying peaks/troughs in player external training loads using GPS-based metrics from the literature.However, it was evident that longitudinal durations considered in cumulative external training load information (e.g.1-2-3-4 weekly loads) was a key factor (i.e.utilised in majority of the studies) considered in the articles.In further two studies, loads were compared across training versus matches and adjustments in load due to previous injury history were also factored.However, because time was the most frequently used factor in the studies, longitudinal durations considered for cumulative GPS information was identified as a literature-based cue for the consensus question.

Practitioner-based cues
As shown in Figure 4, the results from the thematic analysis illustrated the existence of 6 different cues or higher-order themes (player categorisation, periodisation and game strategy, time factors, information factors, player and practitioner judgements and organisational factors), constituting 38 elements (lower-order themes) within them that HPU practitioners consider when making decisions relating to player external training load management.For illustration, Table 3 presents how the player categorisation theme (cue) was identified from the raw data.The description of each factor is specified below with examples from selected elements.
Factor 1: Time.Different time factors were considered by practitioners during player external training load management.Elements like the turnaround time after games, longitudinal durations considered for analysis (e.g.acute: chronic) and time allocated for decision-making were some of the time-based criteria factored by the practitioners.
"Troughs in the sense of like, have we managed to achieve a sprint exposure or multiple sprint exposures this week, yes or no, and if we're seeing consistently the player is going week to week without getting into that sprint zone, erm, and therefore potentially undertraining." Factor 2: Information.Practitioners considered different information sources like GPS information, player age, injury history, resistance training information, etc., when making inferences relating to player training load management.For instance, a practitioner highlighted the factoring of a player's age, training load and injury history information as stated below.
"If they report something subjectively, their age to training history, their injury history is massive on that.Yeah, particularly like, take for example a young guy and he's been put into a position.Say for example Player B, he was forced six games on the bounce and played all the games and he's got quite a high volume.He is more susceptible than someone who is three or four years a senior and has played a lot more.Do you know what I mean, erm, and then conversely if a guy has high injury history, tends to get (cranky) after a certain amount of time from a physical perspective." Factor 3: Player and practitioner judgements.This theme highlighted how the judgements (cognition) of practitioners and players were considered when making inferences relating to external training load management.For instance, the following highlights how the expertise of practitioners to derive evidence from an information source was factored during decision-making."It's also based off the experts that are surrounding me.So, there are experts who are looking at it more so than what I do on a daily basis and understand so much more and they feed it to me." "Some of our back threes are inherently low sensitive.So, to the athlete, we may know that their time at VHSR (very high-speed running) or their time in sprint is more indicative."Finally, as presented in Figure 5, four cue elements (healthy player, GPS information, longitudinal duration and practitioner judgements on information) corresponding to their respective high-level cues (player categorisation, information, time and player and practitioner judgments), that can be used to directly extract evidence from GPS information, were considered to define the below question for the consensus development.initially generated 15 indicators to identify peaks in the external training loads from GPS-based information.Following the initial round of discussions and voting, the practitioners reached a consensus on seven indicators.Group consent was reached on the remaining items after the second round of discussions and voting.After grouping similar indicators, the members reached a consensus on nine items.Specifically, six indicators from the nine (Table 4) illustrated conditions on the GPS information to subjectively identify a peak in the external training loads experienced by a healthy player in the regular training program.
Subjective indicators to identify external training load troughs from GPS information.Similar to the above, nominal group members initially listed 18 indicators to identify a trough in a player's external training load using GPS information.After grouping similar items, eight indicators were considered for first-round discussions and voting.The group reached a consensus on five items following round 1 and after another round of discussions and voting, group consent was reached on the remaining three indicators.Hence, six of the eight indicators (Table 5) signified a condition to identify a trough in player external training loads using GPS information.
At first glance, it may be tempting to scrutinise the use of percentages in the indicators corresponding to the consensus.Hence, we would like to restate that our objective was not on unravelling a set of indicators to identify peaks and troughs in player external training loads using GPS information which may be generalisable across sporting organisations.Instead, the study goal was to present the outcomes of our attempt to establish a consensus relative to the considered environment such that it illustrated the collective agreement among practitioners in the HPU when identifying peaks and troughs from GPS-based information.Consequently, the percentages used in the study illustrated the current subjective opinions of the HPU practitioners.We thus encourage readers not to blindly consider such percentages and use them without validating in their respective sporting environments.Moreover, the choice of percentages in the consensus items with comparisons between

Discussion
This study explored the possibility of articulating a consensus as a subjective standard among HPU staff for identifying peaks and troughs in player external training loads using GPS-based information at a Gallagher Premiership rugby union club.First, adhering to the PRISMA framework, a systematic review was conducted to extract details on the existing evidence on how peaks and troughs in player external training loads have been identified using GPS-based information.Resultantly, longitudinal durations considered for cumulative GPS information was identified as a literature-based cue from the systematic review.Second, practitioner-based cues or the factors considered by staff when managing external training loads of players in the considered environment were identified using thematic analysis conducted on the data extracted from seven semi-structured interviews with HPU staff.There were six core factors or cues (player categorisation, periodisation and game strategy, time, information, player and practitioner judgements and organisational factors) comprising 38 elements that were considered by HPU staff when managing the external training loads of the players (Figure 4).Third, guided by the AGREE II statement and utilising selected cue elements, a consensus for identifying peaks and troughs in the player external training loads were developed using a nominal group with five HPU members.The finalised consensus resulted in generating 12 indicators (peaks = 6 and troughs = 6) to subjectively identify peaks and troughs in the player external training loads using GPS-based information in the considered environment (Tables 4 and 5).

Evidence from the literature
3][34] Those research assessments were a reason that we did not consider evidence from the identification of external training load peaks/ troughs using measures like the acute:chronic workload ratio (ACWR). 35Instead, for the systematic review, we focused specifically on reverting to the first principles by examining studies that identified peaks/troughs directly from GPS-based external training load metrics.Interestingly, it wasn't possible to extract specific researchbased evidence to identify external training load peaks and troughs using GPS information.The reasoning for not articulating direct indicators from the review aligned with the ones observed previously. 34owever, it appears that the real problem in using GPS information for player external training load management, decision-making is not with the metrics themselves, but rather on what question that information aims to answer and consequently, the nature of how that information is used in practice.From an applied perspective, practitioners would likely still look to use GPS information as a feedback pathway during player external training load management since it's the best technology at their disposal in current contexts and also because technological developments occur as a continuously improving process.There is thus large scope for the development of judgement-based guidelines within organisations in this specific area of practice due to the lack of research-based evidence to guide practitioners in decision-making.through various sequences due to different contexts, to create the other layers of the final training load management decision.
"You evaluate the sport first, and then you use every other tool at your disposal to evaluate what's happened within that.So, when we have a rugby session, we evaluate, did we see what we wanted to see live and you chat to coaches.(Yeah), my perception of that was (and at that moment and this is my thoughts on it).When you look back at the video, does the video confirm or tell us something different to what we actually see.Then we start to (work) that out and then we highlight the positives and negatives within the (backroom).We start to use and utilise different data and GPS is one of those."Interestingly, the above statement specifies that within the considered environment, evidence generated from GPS information was only acting as a single contextual layer of the overall external training load management decision.This highlighted that information articulated from GPS data was used to inform or verify a practitioner's judgement within the club rather than drive their decision-making.We believe that this emphasises an important philosophical construct for using data in decision-making relating to player management and should warrant further scientific investigation by researchers.That is because there is a fundamental difference in having data-driven versus data-informed decision-making processes.Specifically, in an industry like manufacturing, data can indeed drive decisions (data-driven) since most processes tends be automated.However, player management workflows in sport are more human-oriented than automation driven.In such latter contexts, a practitioner is the key decision-making entity.Therefore, data should be used to inform a practitioner (data-informed) 36 or in more scientific terms, evidence generated from data (e.g.existence of an external training load peak identified from GPS information) can be used to test the hypothesis for falsifying the practitioner's belief . 37Moreover, if the evidence generated from data does indeed manage to disprove the practitioner opinions, new knowledge may be generated relating to player management in the considered environment.

Consensus indicators and their implementation
It was evident from the results that the NGT was well suited to articulate a consensus as a subjective standard by practitioners to organise judgment-based guidelines for decisionmaking within the considered environment.Examining the identified consensus indicators illustrated that the practitioners in the case study environment considered 1 week and 4 weeks as adequate acute and chronic time windows respectively during player external training load management.It thus appears that the choice of such time periods may have been influenced by the practitioner judgements on the systematic review shared with them prior to the nominal group sessions.This is because the chosen time windows align with those used previously in the literature.However, since the study doesn't provide concrete evidence to assess such a statement, we will leave it as a hypothesis at this stage to provide a basis for a potential future qualitative research exploration.Specifically, to evaluate how practitioners chose such time windows relating to player training load management during decision-making.Moreover, the indicators utilising percentages demonstrated that practitioners still perceived player training load management from ratio-based perspectives, especially relating to training volume.
From an applied viewpoint and in reverting to Dammann's 2 hierarchy, the 12 consensus indicators with agreement can be implemented in future as a subjective standard to generate evidence during collective decision-making processes in the HPU.This can be achieved by integrating the indicators into a data visualisation interface 38 by utilising a colour coding scheme 39 to signify a peak/trough in player external training loads.Furthermore, the guidelines by Cole and Altman, 40 which specify the use of natural logarithms for dealing with percentage differences could be used in the comparative percentage-based indicators to ensure that the comparisons are symmetric and additive.We wish to explore such an integration in the future.However, while the derived consensus could be implemented as a standard within the considered environment, further research explorations are required prior to a fullscale rollout.This is because the indicators must be improved by adding objectivity (e.g.time series analysis), mainly around the usage of time windows, percentages and ratio-based measures.Furthermore, although consensus methods may capture collective knowledge, they are equally capable of capturing collective ignorance. 15herefore, practitioners need to be mindful of such limitations that may exist when developing consensus statements constrained to the expertise within a considered environment.Resultantly, at present, we are quantitatively evaluating the impact that evidence generated from subjective standards like the ones articulated in the current study have on the agreement among practitioners engaged in collective decision-making processes.Thus, examining their potential to reduce decision noise while also validating if such evidence truly transitions to knowledge based on repeatability and agreement arising due to the outcomes of hypothesis tests pertaining to practitioner beliefs.

Study limitations
The subjective standards in the current study were developed in relation to the considered case study environment by utilising expertise within the HPU.Therefore, the consensus statements were articulated to meet the objectives of the given environment.Accordingly, although the methods can be generalisable across other case study organisations, the indicators themselves may not be directly applied to similar settings without validation.The objective of this study was to determine if there was consensus for specific indicators defined by the nominal group members, therefore, to reduce complexity, a dichotomous scale was used.However, if there are requirements to assess the level of consensus, ordinal measurements like Likert scales can be adopted for the voting.

Conclusion
When managing the external training loads of rugby union players in a case study environment, due to the absence of standards, practitioners can generate evidence from information sources based on their personal biases and beliefs.In this study, as the first step to combat noise in decision-making due to such subjectivity, we developed a consensus (signifying the collective agreement among staff) for identifying peaks and troughs in player external training loads using GPS-based information in the HPU of a professional rugby union club in England.The results indicated that practitioners were capable of utilising existing expertise within a case study environment to generate a consensus using the Nominal Group Technique.Moreover, a fundamental organisational objective of change management initiatives like the one discussed in this article is to create a culture for continual improvement.Therefore, as the next step, after incorporating necessary objectivity (e.g.mathematical and statistical analysis to replace percentages) into the indicators, the consensus statements can be integrated as standards into a data visualisation interface to create subjective evidence when managing player external training loads.Since our focus in this study was on a specific case study environment having unique organisational objectives, we do not wish to generalise all the results.Instead, we invite other researchers to utilise the methodological procedures in this article to implement similar systems in their respective environments and unravel the aspects which may be generalisable across organisations.Specifically, the methodological sequence we have adopted could be tested to introduce subjective standards to any other collective decisionmaking process associated with player management.

Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Figure 1 .
Figure 1.High-level overview of the steps conducted in developing the consensus.

Figure 3 .
Figure 3. Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) flow diagram of systematic review.ACWR: acute:chronic workload ratio; GPS: Global Positioning System.

Factor 4 :Factor 5 :
Organisational or environmental.This theme signified the different organisational or environment-based factors that were considered during decision-making relating to player external training load management."Noteverybody within a decision-making process at times has an equal weighting within the decision and equal understanding of the context around the decision, which might change or an equal understanding of actually the intricacies of the decision."Player categorisation.Practitioners categorised players based on different criteria and factored the resulting categories when making inferences relating to player external training load management.For instance, staff considered the effects of load in relation to the playing position of an individual or had greater caution about the training load exposures of travelling reserve players when making decisions.

Factor 6 :
Periodisation and game strategy.The following factor highlighted the importance of considering specific conditions around the periodisation strategy at the club as well as tactical criteria corresponding to games.For example, staff considered periodisation objectives defined for the program and team selection criteria when making decisions regarding player external training load management.

"
We are starting to figure out what our program gives us and what the coaches want from the program.Therefore, we need to prepare the players to survive in that and not just survive but thrive."

'
By considering only current available Global Positioning System (GPS) based metrics, what specific information conditions in them (using preferred comparative views of longitudinal periods) would indicate a peak or trough in the external training load experienced by a healthy rugby union player (do not consider injured or legacy players) in the regular training program?' Consensus statements from the nominal group sessions Subjective indicators to identify external training load peaks from GPS information.The 5 nominal group members

Figure 4 .
Figure 4. Factors considered by the practitioners during player external training load management in the considered environment.
first team Young players exposed to repetitive load creates risk of injury Main squad player load harder to manage Need to be aware about managing training loads of travelling reserve players Matchday versus travelling reserve versus non-23 player Travelling reserves greater possibility to have training load troughs Important to manage travelling reserves load troughs Academy players in travelling reserve is not frequent time windows (indicators 1, 2, 5, 6, 10, 11, 14 and 15) were articulated by the practitioners based on their interactions with a web-based R Shiny application developed by author JR.That interface was designed to contextualise (graphically) the different GPS information percentages defined in the indicators.The 5 nominal group members interacted with the designed data visualisation during an additional 1-hour meeting and reached an agreement on the relevant percentages.

Figure 5 .
Figure 5. Cues considered for defining the consensus question.

Figure 6 .
Figure 6.An example of how the different factors associated with a player training load management decision are layered during execution.

Table 1 .
Characteristics of the High-Performance Unit (HPU) members interviewed to extract practitioner-based cues and participated in the consensus development sessions.

Table 2 .
Questions used for the interviews with High-Performance Unit (HPU) staff.
S. No. Key question 1 Would you consider managing the external training load of a player from an injury risk mitigation perspective, performance adaptation view or both? 2 As a practitioner, what would concern you more, a player exposed to an external training load peak or a trough? 3 Which is easier, managing the external training load of an injured player or that of a healthy player? 4 Would you make inferences on player external training load management from individual GPS metrics (e.g.number of VHSRs) or from the interaction between multiple metrics?And could you please explain which of those measures you would use frequently? 5 Are there specific longitudinal durations you may consider (e.g.previous 2 weeks) when making inferences relating to player external training load management from GPS-based information?6 Would there be instances that you wouldn't rely on GPS-based information when making decisions relating to player external training load management?7 Do you think there can be noise in decision-making due to the different ways in which a practitioner can generate evidence from GPS-based information?GPS: Global Positioning System.
12/22 studies can be rated as 'Fair' (15-19 score) and the rest (10/22) would receive a 'Poor' (≤ 14 score) rating.The complete details of the quality ratings are available on Table C in the Supplementary material 1.
Study aims and outcome measures.From the selected articles, 18/22 evaluated the association between external training loads quantified using GPS-based metrics and injuries (overuse, non-contact, soft tissue, time loss and preventable).Similarly, a further 3/22 articles assessed the relationship between external training loads and performance adaptations like changes in aerobic and anaerobic fitness.Only one study aimed at using prediction as an outcome.None of the selected studies had evidence on identifying external training load peaks/troughs associated with player wellness, fatigue and illness.Furthermore, only one study was focused on rugby union and the others based their experimentation on football (8/22), Australian football(8/  22), rugby league (4/22) and Gaelic football(1/22).loads beyond the previous 4 weeks.However, within the articles, no justifications were provided for considering specific time windows in the data for the assessments.