BUILDING BETTER CLINICAL TRIALS

Orikami, a data science company in Nijmegen, the Netherlands, used CTTI's Novel Endpoints Recommendations as an internal quality guide to develop a digital biomarker to monitor fatigue in multiple sclerosis (MS) patients. As a result of this effort, the biomarker is integrated into Orikami's MS Sherpa app, which monitors fatigue along with two other digital biomarkers (cognition and mobility) to check MS progression more frequently and objectively in support of informed, shared decision making between neurologists and patients.

Background: Orikami is a Dutch MedTech organization specialized in applied data science and development and implementation of digital biomarkers to enable personalized healthcare for people with chronic diseases.When talking to patients with MS, Orikami discovered that often their main complaint was fatigue, but patients reported that they struggled to communicate about fatigue effectively with their neurologist, care team, or even their spouse. Fatigue, after all, is subjective and often difficult to describe in concrete terms. However, a common trend began to emerge: while patients struggled to detect fatigue in themselves, they told us their partners often served as an early warning sign. These partners could detect fatigue by looking at their loved one's eyes. Orikami was intrigued and began exploring published literature about the relationship between eyes and fatigue. Several publications noted a correlation between drowsiness and eye tracking or blinks in driving, and there was even a small amount of literature specific to fatigue and MS with promising results. Orikami organized a co-creation session with neurologists, MS patients, and MS patients' relatives that confirmed this information, and the organization decided to launch the development of a novel digital concept to identify fatigue in MS patients on the basis of eye movements. Orikami wanted to develop a digital endpoint that was sensitive to changes in fatigue, as well as highly acceptable, highly scalable and easy to administer. With this in mind, they quickly decided that the front-facing camera of smartphones was the most logical choice to register the endpoint. The idea was that MS patients could simply open an app on a phone, watch a series of dots that jump around, and have reaction time (an indicator that according to the literature correlates highly with fatigue) sent to themselves and their doctor, offering the clinician valuable insights into the patient's MS experience.

Orikami, a start-up at the time, dove into the development process and quickly realized they did not have a clear enough idea of exactly how to build and validate their concept. The team would hit a roadblock, then realize they needed to rework steps already taken and implement new processes. Realizing their method was not optimally efficient, Orikami began searching for guidance on best practices for developing a novel digital endpoint. For example, the choice to use available smartphones created a lot of difficulties in synchronizing the timings of the stimulus of the smartphone screen and eye tracking. Orikami found out that the internal clock of the camera and the internal clock of the smartphone screen was not synchronized, so for valid eye tracking, they needed to get a grip on these basic data assumptions. To make the development even more complex, the time synchronization isn't stable over different devices and different operating systems. Orikami found themselves in a troubling position: they had started a very costly clinical trial, and some of basic data issues were not yet resolved.

The team came across CTTI's Novel Endpoints Recommendations, which address the barriers Orikami had encountered and clarify the best pathways for identifying, selecting, and developing novel endpoints derived from mobile technologies. The step-by-step process puts patients' needs first and encourages cooperation between patients, their caregivers, health care professionals, and developers. These recommendations not only proved to be an effective quality check for Orikami, but also helped the team articulate their concept into practical words and steps to facilitate good discussion with stakeholders and clients. Most importantly, they offered clear guidance on which dimensions in technical validity must be firmly established before proceeding to the clinical trial stage.

Orikami went back to the drawing board and began translating CTTI's recommendations (a six-step plan) into a development framework that the team could bring to life:1. Identify which aspect of health is a Meaningful Health Aspect (MHA) by interviewing MS patients to determine what aspects of their condition bothered them most. Note that MHAs are not determined by what can be effectively measured, but by what patients report as the symptoms they struggle with most in their daily lives. Orikami had already completed this stage with their subsequent interviews, so could confidently proceed with the knowledge that fatigue was a primary concern for MS patients. Cognition and mobility were also identified as MS MHAs. 

2. Define the scope of assessment and determine if identification of fatigue as a biomarker will positively affect MS patients. For example, even if fatigue could be better tracked, would it improve the lives of patients? Since fatigue is highly correlated with MS disease progression, improved identification of this health aspect could trigger a meaningful change in treatment that may improve patient lives.  

3. Find the appropriate and exact measurement for fatigue. Was eye movement and reaction time the right concept of interest to target? Orikami's research concluded that delayed visual response was in fact a significant indicator. 

4. Find the right mobile technology. Should Orikami work with only one platform, or could they target a diverse range of devices? Is the front-end camera theoretically capable in terms of framerate, contrast sensitivity, and resolution? 

5. Determine the standards for use. For example, at what, if any, point would dim lighting make detection of eye movement unreliable? How close to the eyes should patients hold their phone? Is the movement of holding the smartphone disturbing the signal? The Orikami team assessed many factors and created guidelines to ensure they developed the appropriate proof-of-concept tests in this phase.

6. After rigorously going through each of the previous steps, Orikami was ready to show validity in a clinical study. At the time of this writing, that study is ongoing. Orikami's strong protocol is testing a range of setups of stimuli on the screen to see how people with MS and healthy controls react to these. The study is exploring the sensitivity of change on published measures of fatigue, as well as correlations with disease activity, relapses, and progression.For a detailed look at Orikami's journey with CTTI's Novel Endpoints Recommendations, visit this blog authored by the development team.  

Orikami's fatigue biomarker is now integrated into Orikami's MS Sherpa app, which monitors fatigue along with two other digital biomarkers (cognition and mobility) to improve decision-making between MS patients and their physicians. CTTI's guidelines continue to serve as an important component of Orikami's quality steps in all biomarker endpoint development, ensuring the entire development team is on the same page. 

Orikami is currently applying this development concept to other biomarkers in MS and Parkinson's disease. Although CTTI's recommendations were brought in late to help the organization right-track their fatigue biomarker effort, they are now used as the starting point and foundational framework for biomarker endpoint development. Orikami has expressed confidence that all developers can use CTTI's process as a starting point and baseline architecture to create digital biomarkers that consistently and reliably provide information to help improve patients' lives.