Hypothesis / aims of study
A good urodynamic test should result in a high-quality trace which can be reliably interpreted, to guide assessment and treatment of lower urinary tract symptoms. A method to measure the quality of the trace was systematically developed, the ‘UTraQ’ system – Urodynamic Trace Quality system (1). This study aims to improve the inter-rater reliability of the scoring method by clarifying the assessment questions used, improve user guidance material and to find the minimum score defining a good quality trace.
Study design, materials and methods
Twenty two traces were collected from attendees at ICS-certified urodynamic courses and submitted to seven experienced urodynamicists (clinicians and allied health-care professionals) from four different centres. Where scores differed significantly between raters, clarification was sought on the reasons for the difference and questions edited accordingly. In addition, pop-up guidance notes have been added to the Excel score sheet, to reduce any confusion or ambiguity. These notes have also been added to the spreadsheet, to facilitate use as a printed out hard copy, rather than just an on-screen system. The scores were then adjusted to reflect the answers to the revised questions, where a different response was agreed.
Results
Figure 1 shows the scores for each trace submitted, with each point representing the score given by one rater. The system responds to variations in data quality, while maintaining broad agreement between raters. The variation between raters is ± 10% of the mean. Some variation is to be expected, since there is a small amount of subjectivity in the assessment, for example “Was a good quality cough test carried out at the start of the test?”. Scorers would be unlikely to measure the cough responses precisely, thus the answer might be dependent to some extent on personal judgement. Some other variation can be explained by the fact that traces were produced by urodynamic machines with which the scorer was not familiar.
In the process of improving the system, adjustments were made to the phraseology of questions, to clarify any that were unclear. An example is the original question “Were all of the pressure axes displayed with the same height per cmH2O?” was changed to “Were the maximum values of each pressure scale / axis the same value in cmH2O?”, since reference to axis height was found to be confusing.
Some questions, though felt to be focusing on non-essential detail, were retained as this will enable the scoring system to differentiate between ‘good’ and ‘excellent’ trace quality, even though both grades would be acceptable for a confident diagnosis to be made.
Interpretation of results
The results to date indicate that the scoring system is sensitive to the quality of the trace presented, since despite some variation in scores between scorers, there is a contour of agreement between them. During this process, it has been suggested that, rather than simply ‘good’ or ‘poor’, a global assessment of ‘some useful data despite quality shortcomings’ can be added. This should facilitate the highlighting of areas for improvement without rejection of useful diagnostic data. Indeed, it is envisaged that this tool will be used in audit of a urodynamicist’s or a department’s performance, giving objective guidance for and evidence of areas for improvement.
Notable from our data is the need for scorers to gain some practice with the system, as scores varied more on the first three traces, but then became closer after that. It is also of note that the values currently used in the guidance refer to the ranges of normal values for adults only.
The system has some questions that are specific to water-filled systems. Since the questions were originally tested on the ICS standard water-filled systems, future work will be needed to verify the application of UTraQ to air-filled systems, as some values used in quality assessment, e.g. normal initial resting pressures, are as yet unknown for air-filled systems.
The data collection will continue until 60 traces have been scored, at which point the cutoff for acceptable quality (currently suggested as 75% (1)) will be reviewed. In the meantime, the ICS trace peer review service, run under the auspices of the ICS Urodynamics Committee, will continue and expand. For that purpose, the updated trace score sheet will be maintained on the ICS website at https://www.ics.org/folder/committees/urodynamics-public-documents/d/trace-score-reviewer-feedback-sheet-revised/download