Factors affecting the manual linking of clinical progress notes to problems in daily clinical practice: A retrospective quantitative analysis and cross sectional survey

This cross sectional study examines how patient characteristics, doctor characteristics, and doctors’ education and attitudes affect the extent to which doctors link progress notes to clinical problems. The independent effects of patient characteristics on the linking of notes was examined with a mixed model logistic regression. The effects of doctor characteristics and doctors’ education and attitudes on the link ratio was analyzed with univariate analysis of variance. A survey was used to obtain arguments and attitudes on linking notes. For “patient characteristics”, the odds of linking increased with an increase in the number of problems or hospital days, decreased, with an increase in the number of involved doctors, medical specialties or the number of notes. For “doctor characteristics”, the link ratio increased with more work experience. For “doctors’ education and attitudes”, the link ratio increased with more familiarity in linking notes and belief in the added value of problem oriented charting. “Overview” was the most cited reason for linking; “I don’t know how” the most cited reason for not linking. There is a huge variation within and between all disciplines. Important arguments, for and against, are found. Recommendations for policymakers and medical leadership are given to maximize the benefits.


Introduction
In recent years, Electronic Health Records (EHRs) have been implemented by an ever increasing number of hospitals. EHR systems have various forms and their implementation is a complex undertaking. 1 Fifty years ago, Weed 2 proposed a problem-oriented approach to improve the structure and content of clinical documentation. He also suggested to document the course of treatment according to the subjective, objective, assessment, and plan (SOAP) format. 2,3 The problem-oriented medical record (POMR) is a concept to organize data around medical problems. Although the POMR is widely adopted in EHR systems, only 15 original research studies have been published in the past 40 years. 4 An accurate problem list is a core element of the POMR and provides many advantages for delivering good clinical care. 5 It is intended to serve as a multi-disciplinary index for all health professionals. An accurate problem list can support a meaningful patient-clinician dialogue that fits with the trend of care becoming more patient-centered. 6,7 Although the term "problem list" seems to be a fairly common term in the literature 4,8 there still is discussion about many aspects of the problem list. 9,10 Moreover, there are several factors that are critical for the successful use of the problem list, such as structure, multi-disciplinary use, management, and policy. 11,12 The problem section, according to Health Level Seven International (HL7), lists and describes clinical problems or conditions currently being monitored for the patient. 13 The concept of "problem" therefore has a broader scope within this definition than diagnosis in health and clinical management and can be used for all matters that require attention. Unfortunately it is still one of the great failures of clinical informatics to effectively integrate the management of the problem list into the clinician's workflow. 14 Linking notes to problems, as defined by HL7, is of great importance to gain a better understanding of the patient's story and the clinical reasoning of the healthcare professionals involved. Furthermore, linking notes is useful for other aspects, such as accountability, continuity of care, education, and supervision. Nowadays, almost every EHR, including the systems from the dominant EHR vendors such as Epic and Cerner, use a patient-centered problem list where data can be linked to problems. Despite the benefits of linking, the multi-disciplinary use of the POMR, and in particular linking notes, is not a simple and straightforward matter. [15][16][17][18][19] The problem list can be used as an index, especially with increasing multi-disciplinary chart complexity or where patients transfer across settings. Ideally, a patient-centered problem-oriented overview could be created with data from different sources. 20 This also fits integrated practice units accountable for the total care for a medical condition and its complications, whereby value for the patient is created by providers' combined efforts across the entire care cycle, rather than "focused factories" concentrating on narrow sets of interventions. 21 Although many articles state that linkage between problems and data is essential for successful use of the problem list, linkage itself has hardly been studied. 12 Linking notes to problems is mentioned in articles, but it is mostly subjective. We are not aware of any study that has examined factors that influence the linking of notes to problems. Better understanding of these factors can contribute to further improvement of information quality and the continuity of care. Figure 1 provides an overview of the characteristics to be analyzed about patients and doctors, as well as the education/attitudes of doctors, that may affect the linking of progress notes to items on a problem list.

Set of factors under review
Patient characteristics include the number of problems a patient has in the chart, the number of doctors and medical specialties involved with a patient, the number of progress notes per patient, and the number of days that a patient has stayed in the hospital. Doctor characteristics include age, sex, medical specialty, and number of years of work experience. A third set of factors consists of doctors' education and attitudes. These include education and experience with linking notes, personal and team attitudes toward linking notes, and the use of a directive for management of the problem list.
We expect a positive influence with an increase in any of the patient characteristics given the advantage of maintaining an overview of the patient's case. For doctor characteristics, we do not expect any difference in age or sex. We expect that non-surgical specialties link more notes than surgical specialties because the latter often only concentrates to a specific problem to deal with, whereby linking notes may be of less value to them. We expect more linked notes with an increase in work experience because there will then be more experience with long-term treatment of patients where linking notes contributes to maintaining an overview of the patient's case. We hypothesized that more education and experience with linking notes, a positive personal and team attitude toward linking notes, and the existence and use of directives for the problem list, would be positively related with the linking of notes because of the belief in the added value of this method.

Setting
In a Dutch university medical center, the linkage of progress notes over a 4-year period was studied.

Progress notes module and link functionality
An EHR progress notes module built upon a home grown EHR has been used with link functionality that is generic for all medical specialties 22 and is based on the POMR and SOAP principles 23 which are now ubiquitous. 24 The starting point for adding and managing items on the problem list is a progress note. Problems are automatically added by what is entered in the Assessment-Diag/ Hyp field (see screenshot in Figure 2). If a diagnosis is entered as part of a note, this note is automatically linked to this problem. One can write multiple diagnosis in one progress note. Allergies and active issues from the medical history are also visible on the problem list. A progress note, with or without a diagnosis, can be manually linked to one or more problems on the list by clicking the corresponding items. Based on the linked notes one can generate a problem-oriented overview that is, an overview of all notes linked to one or more problems. The problem list can then act as an index for all linked notes.

Transaction log file and EHR database
A separate log file automatically recorded all user-system interactions, such as mouse clicks and application events. Each record of the log file contained items, such as patient identification, user identification, user role, performed action, and a date/time stamp, but no medical data. Two external professional testers validated the log file functionality. Querying the EHR database provided more detailed information for analysis of the progress notes.

Survey
To obtain doctor perceptions about the linking of notes, they received an on-line survey questionnaire. To link answers from the survey to data from the log file, the survey was not anonymous. Permission for this was granted by the privacy officer as a representative of the board of directors. In total, there were 58 questions involving various topics, such as user education, work experience, viewpoints about POMR, and use and usability of the system. More than half of the questions were about the POMR. Two open-ended questions about the linking of progress notes were not mandatory. We pilot tested the questionnaire with several doctors to assess intelligibility and clarity. For the actual survey, we selected doctors who had written at least 200 progress notes in 2012. After excluding doctors who were not able to respond because of extended leave or retirement, the total population was 700. These doctors were invited to take part in the survey by e-mail at the end of April 2013. A first reminder was sent 2 weeks later, followed by a second reminder at the end of May.

Analysis
Data has been extracted from the log file and EHR database using MS SQL Server Management. Statistical analyses were performed with IBM SPSS Statistics 25.0 (SPSS Inc., Chicago, IL). Firstly, the distribution of all notes over the days of the week and over the hours of a day was investigated. The calculated link ratio is the number of manually linked notes divided by the total number of notes that could be linked manually. Therefore we excluded automatically linked notes and notes from patients with an empty problem list. Although linking a note to a single problem or linking a general note may seem unnecessary, a note can always be linked to one or more problems. This makes a link ratio of 100% the theoretical, but not necessarily the ideal, maximum. An automatically linked note can also be linked manually. This is the case if a note is written with a diagnosis, and is also linked to an already listed problem. Overall analysis of the data was done using descriptive statistics. Significance for all analyses was set at p < 0.05.
Three sets of factors have been examined in accordance with the set of factors under review. For the set of "patient characteristics", we counted the number of problems, regardless of the status of a problem, number of doctors involved, number of grouped specialties involved, number of progress notes, and number of hospital days, whereby each unique day where at least one progress note was written counts as a hospital day. Taking into account the nesting of notes within patients within doctors, we used a generalized linear mixed model with random intercepts for a mixed model logistic regression to study the independent effects of patient characteristics on the linking of notes expressed as an odds ratio.
For the set of "doctor characteristics", doctors were categorized by their specialty in the following eight groups: (1) anesthesiology, (2) cardio-lung, (3) internists, (4) neurology, (5) pediatrics, (6) supporting (radiology, radiotherapy, and nuclear medicine), (7) surgeons, and (8) others. Univariate analysis of variance was used to study the effects of age, sex, medical specialty, and work experience on the link ratio. Also, the link ratio of the doctors was analyzed in the course of time. For variation between medical specialties and variation within a medical specialty, link ratios were calculated for all doctors who had written at least 200 progress notes from 2010 to 2013. A box plot was used to show the variation between doctors and medical specialties. We performed a non-respondent analysis of the survey, using chi-squared tests to study differences between responders and non-responders with respect to the medical specialty and the number of written progress notes, which were extracted from the log file.
For the set of "doctors' education and attitudes", the answers from the questionnaire were used in relation to the respondents' link ratios. Univariate analysis of variance was used to study the effects in relation to the given answers. The information gathered from two open-ended questions from the questionnaire, asking the main reason to link or not to link notes, were first split into separate arguments and then categorized. Results from the other questions were grouped together and displayed in a table.

Results
In total, 1765 doctors and 1,878,705 notes referring to 193,703 patients were initially included. The extracted data from the transactional log file and EHR database with the numbers of patients, doctors and progress notes are summarized and represented in a data selection diagram ( Figure 3). Figure 4 plots the distribution of all included notes over the days of the week. Of all notes, 202,534 were automatically linked and 435,008 were manually linked.
The average percentage of manually linked notes on weekdays is 23.7% and on weekends 16.5%. The distribution of all included notes over the hours of the day shows a similar picture where the average percentage of manually linked notes from 8 am until 6 pm is 24.2% and from 6 pm until 8 am is 12.7%. Monday to Friday from 8 am to 6 pm the average percentage of manually linked notes is 24.6%. Table 1 shows the effect of patient characteristics on the linking of notes. The table displays the percentage of notes and corresponding odds ratio for each patient characteristic.

Patient characteristics
An increase in the number of doctors, medical specialties or progress notes was independently associated with a smaller chance of linking. In contrast, an increase in hospital days or the number of problems was independently associated with a greater chance of linking. Table 2 shows the effects of doctor characteristics on the linking of notes. The table displays the link ratios for each doctor characteristic and categories with 95% confidence intervals. Age was not used because of the correlation between age and work experience (Cramer's V = 0.643).

Doctor characteristics
Significant variation was observed between medical specialties. Pediatrics had the lowest link ratio, while supporting had the highest link ratio. Also, significantly higher link ratios were found in doctors with more than 10 years of work experience. We found no significant differences in the link ratio between male and female doctors. Figure 5 shows the variation between all doctors who wrote 200 or more notes, as well as the variation within a medical specialty. Within each specialty, there were doctors who did not link at all and doctors who linked very extensively. The same outcome was observed with regard to the respondents for the lowest (pediatrics) and highest (supporting) link ratios. Figure 6 shows the link ratio of 1161 doctors who wrote 200 or more notes on 151,356 patients. About a quarter (24%) of the 1161 doctors (n = 274) manually linked 50% or more of their notes, representing 70% (302,967 out of 435,008) of all manually linked notes. Eleven percent of the 1161 doctors (n = 123) did not link any notes (link ratio = 0.00). Table 3 shows the effect of doctors' education and attitudes on the linking of notes. The table displays the link ratios for answer options to various aspects of doctors' education and attitudes.

Doctors' education and attitudes
Significantly higher link ratios were found in doctors who were familiar with linking notes and who believed that linking notes made the record more comprehensible. Attention to the POMR during medical training was significantly related to a lower link ratio. Doctors who paid a lot of attention to the problem list had higher link ratios, but the difference was not significant. We found no significant association between link ratios and training hours or favoring a hospital-wide directive.
The link ratio of the doctors over time showed that the percentage of doctors who link >50% of the notes remained stable, while the link ratio of doctors who linked <50% continued to decline.

Survey results
The results of the questionnaire were based on the response of 258 respondents (response rate = 37%). We found no differences between respondents (258) and non-respondents (442) with respect to specialty (p = 0.635) or the number of written notes (p = 0.329). In the questionnaire, we asked the doctors about the most important reason to link or not to link progress notes to problems. From the 258 doctors, there were 115 answers with one to three reasons on why to link, and 179 answers with one to three reasons on why not to link (Figure 7). "Overview" was by far the most mentioned reason to link notes, while the most mentioned reason not to link was "I don't know how." Table 4 shows the responses to the questions regarding education, experience, personal and team attitudes toward linking notes, and directives for the problem list. The results showed that many doctors had little training, most doctors were experienced and familiar with the functionality, splitting notes or linking them to multiple problems was rarely used, and a majority of the doctors were for a hospital-wide directive. Noteworthy is the question, "In daily practice I link the following average percentage of notes to problems?." Doctors generally made a good estimate of the number of self-linked notes. The deviation of the response to the questionnaire and the measured score based on the EHR database for 71% of the doctors were less than 20%. In 9% of cases, there was an underestimation, while an overestimation was observed in 20%.

Principal findings
For "patient characteristics", the linking clearly increased with an increase in the number of hospital days or the number of problems. Contrary to our hypothesis, an increase in the number of involved doctors, medical specialties or the number of notes clearly decreased the linking. For "doctor characteristics", the linking increased with work experience of more than 10-15 years, which was in agreement with our hypothesis. The variation of linking notes between medical specialties, but also between doctors within a specialty, was enormous. Some doctors did not link at all, while others linked very extensively. In contrast to our hypothesis, internists had a slightly lower link ratio than surgeons. For "doctors' education and attitudes", doctors who were familiar with linking notes, paid attention to problem list accuracy, and believed in the added value of linking notes had a higher link ratio. Remarkably, the attention to the POMR during medical training seemed to have a negative effect on the linking of notes, possibly due to the expected additional administrative actions. Over time, the link ratio decreased within the group of doctors who linked 50% or less, while the group who linked 50% or more of their notes remained stable. During the evening and at night, significantly fewer notes are linked. By far, "Overview" was the most commonly given reason to link progress notes to problems, while the most common reason not to link was "I don't know how" followed by "Forget", "Too complicated", "Time consuming", and "No added value".

Strengths of the study
The main strength of this study is that this is the first study that has measured the linking of notes from a generic multidisciplinary application and that provides comprehensive insight into factors that affect the manual linking of notes. Another strength of this study is the combination of data sources, a detailed transactional log file, the EHR database, and survey data. For the first two sources, data were collected over a relatively long period. The log file itself was validated twice by independent professional testers. Because the survey was not anonymous, we were able to investigate specific characteristics. Furthermore, during the almost 4 years of data collection, this module has hardly been changed.

Weaknesses of the study
The response rate to the survey (37%) was relatively low; however, our analysis of non-responders did not reveal any difference between respondents and non-respondents with respect to specialty or the number of written notes. Furthermore, the questionnaire was not anonymous. While this allowed the possibility to link the results to the log file, the lack of anonymity may have biased respondents. However, this effect is probably small, because there were no right, wrong, or socially desirable answers in our survey. We also had no reason to believe that questions were answered dishonestly. The measurements were done in only one university hospital, which may limit the generalizability. Apart from that, comparison with other hospitals is difficult, because they use different systems that are highly configurable, what applies to both homegrown and commercial systems. Many variables play a role in the way in which doctors use the system. 25 Therefore, we mainly focused to compare different groups of doctors using the same system for a number of years in which the system has hardly changed.
Additionally, patient characteristics were measured at the last date of the log file, as well as the measurements for the other factors, such as the number of linked and unlinked notes. If, during the addition of new notes, a change occurs in these factors that could also influence linking, it is averaged out.
The fact that automatically linked notes that were also linked manually have been excluded, could lead to a small underestimate of the link ratio.
There may be other factors that we have not explored, which could explain the variation in linking notes to problems. Factors such as the nationality of doctors or patients, factors that affect physician professional satisfaction due to poor EHR usability, 26 working conditions for clinicians, 27,28 the location or level of care such as an intensive care unit, combined with operational policies, that may affect the department's behavior and performance. 29 In addition, rewarding or financial incentives could be latent variables that might influence care provided by doctors, 30,31 and also the linking of notes so that, for example, all transactions for a specific problem are billable. Also, if one is in doubt but is responsible, the choice may be made not to link to a specific problem. So often cited arguments for not linking notes (Figure 7) can also be explained by other factors.

Relation to other studies
We are not aware of any other study that has analyzed the effect of patient-and doctor-related factors on linking notes. According to Chowdhry et al., 4 only one study explored the usability of a fully electronic POMR in daily clinical practice. Important reasons that are mentioned in this study from Bossen 32 for not linking notes are "too complicated" and "time consuming". Although manually applying these relationships takes time, doctors who are familiar with linking notes seems to manually link notes to problems relatively easily given the higher link ratio. In general, limitations of the POMR, such as complexity of maintenance and missing guidelines, especially multi-disciplinary, have been described by Salmon et al. 33 In our study, factors like too complicated, time consuming, and no policy are often mentioned as similar reasons not to link. Although all doctors are trained before and after implementation, usability and functionality remain a point for improvement. A possible approach for improvement is the use of problem-oriented templates that are associated with increased note quality without an increase of total charting time. 34 Linking notes to problems is often mentioned in articles, but these are often opinions or expectations. There are several articles in which alternatives have been suggested where the linking of data to problems can take place automatically because of the integration and application of medical knowledge. [35][36][37] Suggestions for linking data, or the automatic linking of data in the case of evidenced-based relationships, can also give an improvement.
Although the problem list is a fundamental item of patient-centered care, different clinicians often have different ways of working; therefore, significant variation is observed in the use of the problem list. 10,38-40 "Overview" was the most frequently reported reason to link progress notes to problems, which is also an important success factor mentioned in other studies. 11,12 Our survey shows several diagnosis-related reasons on why not to link progress notes, such as diagnosis from other doctors", "no definitive diagnosis", or "more than one diagnosis". The notion  (14) There is, within my specialty, a directive for linking notes to problems No (40) Don't know (43) Yes (17) These directives are aligned with the directives of other specialties No (25) Don't know (56) Yes (19) Generally, I stick to these directives  of serving clinicians, patients, and other stakeholders in a broad sense, yet lacking a standard for using problem lists and for the linking of notes, is incompatible at the expense of the information quality. Another important aspect of linking notes to problems is the human-computer interaction, particularly the usability of computer interfaces. Usability is especially important when performing complex tasks given that determining a health problem is a complex, collaborative activity that involves information gathering and clinical reasoning. 41 Other research claims that improvements in EHR usability are essential in accurate recording diagnosis in EHRs. 42 Authors recognize and endorse the results of the literature review conducted by Zahabi et al. 43 to usability guidelines for EHR interfaces focused on the diagnosis process and documentation. Despite existing research and further development of EHRs, improvement of usability cannot be taken for granted. 44 A recent large study of US physicians measuring the usability of the EHR systems shows that the usability is perceived as poor. 45 In a previous study from Cillessen et al., 23 the overall usability of the clinical notes application, which contains the progress notes module for linking notes, was investigated. The results show that the overall physician usability satisfaction score was acceptable given the score of 3.2 on a scale from 1 (highly dissatisfied) to 5 (highly satisfied). Others argue that the functionality of adding relationships between different data, such as linking notes to problems, is an important factor that can further improve the usability of EHRs, increasing physician satisfaction with EHRs. 46,47 Finally, we discuss the problem list and the linking of notes in relation to the application of standard terminologies such as ICD and SNOMED CT. ICD is used primarily by professional coders working with clinical records for billing, research, statistics or other purposes. In the progress notes module, diagnoses can be entered as free text or as ICD-10 codes, but coding was rarely used. ICD is not intended or suitable for use directly by clinicians at the point of care and is not detailed enough to meet the needs of hospital specialists. 48 SNOMED CT is the most comprehensive healthcare terminology available used in electronic health record systems to facilitate clinical documentation and reporting and to retrieve and analyse clinical data. 48 Although there are organizations that have successfully implemented SNOMED CT, there is still much work ahead to bring SNOMED CT into routine clinical use. 49 Despite this work, the importance of using standards such as ICD, SNOMED, and others, cannot be understated, especially for the benefit of clinical decision support systems. 50 The increase in the use of standard terminologies, which are also becoming increasingly mandatory, may improve the quality of the problem list, which as a result, may have a positive effect on linking notes to problems.

Meaning of the study
The more complex the medical record, measured in the number of doctors, medical specialties, and notes, the more the link ratio decreases where we would actually expect the opposite. Linking notes to problems is appreciated in daily clinical practice to create an overview, but takes time, which is a frequently cited reason for not to do so, which may partly explain the low link ratio between 6 pm until 8 am. Accuracy of the problem list and belief in the added value seem to be important factors for linking notes. A minority of doctors, represented in all disciplines, link their notes very persistently. It is discouraging for them when colleagues, especially in multi-disciplinary use, affect the integrity of the problem list or do not link their notes, which in turn discourages others from linking notes and thus can deteriorate further over time. Better and regular training could increase the link ratio given the main reasons not to link notes, "I don't know how" and "Forget." There are several recommendations to maximize the benefits of the linking of notes. First, policymakers must ensure a multi-disciplinary guideline for linking notes, which should at least pay attention to the content and management of the problem list and when to link which data. Second, medical leadership should emphasize the added value of linking notes. Third, doctors must be (repeatedly) trained on how to use the linking of notes. Fourth, policymakers and informaticians should optimize the desired way of working with, for example, easy to use templates or mappings. Finally, medical staff should conduct open and closed record reviews periodically for feedback.

Future research
More research is needed to define the required functionality and to improve the usability to maintain a multi-disciplinary problem list. In particular when a patient's health condition (expressed as one of more complaints, issues, findings, diagnoses, risks, barriers, etc.) and its course are identified and monitored by healthcare provider(s) and/or the patient and patient family. 13 This also fits with the call for a fundamental paradigm shift in the way health services are delivered, managed and funded from the World Health Organization (WHO). 51 Patient access to the EHR may improve communication and increase engagement and self-management. 52 Especially if more people or organizations are involved in the care of a patient the linking of notes is useful because it contributes to transparency. It would be useful to conduct observational studies in those circumstances where the link ratio is low.

Conclusions
Linking notes to clinical problems is seen as added value to keep an overview. Linking notes is not self-evident and varies from no use at all to extensive use. We found important arguments from doctors for and against the use of linking notes. Our key recommendations for policymakers and medical leadership to maximize the benefits of the linking of notes; ensure a multi-disciplinary policy, emphasize the added value, train doctors sufficiently and repeatedly, improve functionality and optimize usability to optimize the desired way of working, conduct record reviews periodically for feedback. financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years, and no other relationships or activities that could appear to have influenced the submitted work.

Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.

License for publication statement
The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide license to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to (i) publish, reproduce, distribute, display, and store the Contribution, (ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, (iii) create any other derivative work(s) based on the Contribution, (iv) to exploit all subsidiary rights in the Contribution, (v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, (vi) license any third party to do any or all of the above.

Ethical approval
Not applicable.

Transparency declaration
FC affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.