From pilot to practice. How to add value through data science in healthcare
AI, machine learning, and data science are hot topics in healthcare. In recent years, many AI pilots have been launched by healthcare institutions. But knowing that 50% to 80% of data science projects fail, it is crucial to look beyond these pilots. Impact Steward Frank Peters takes you along the journey. How do you set up successful data science projects that deliver sustainable results?
Why data science projects often disappoint
AI projects often fail to meet their – sometimes sky-high – expectations. But what does success actually look like? A lack of clear objectives is one of the main reasons for the ‘failure’ of these projects. If you do not know what you are working towards, how can it be a success?
Another significant reason why data science projects fail lies at the core: the availability and quality of data. Data is often incomplete, biased, or noisy.
A common problem is a one-sided focus on technology instead of real-world impact. Don’t put the cart before the horse. Meaningful application requires knowledge and experience from healthcare practice.
How to use data science effectively?
To move AI projects beyond the pilot phase and into practice, a structured, clear, and process-oriented approach is needed. The CRISP-DM model (Cross Industry Standard Process for Data Mining) is one of the most widely used and recognised data science frameworks. Following this approach is also valuable for healthcare institutions.
The foundation: Understand the ‘business’
“No clear goal, no success.”
The first step of the CRISP-DM model, Business Understanding, is about understanding the context in which the project is conducted and formulating a clear and measurable question. What problem is being solved, and what ‘value’ is being created?
Involve domain experts from the start
Healthcare projects often involve sensitive data and ethical considerations. Understanding the needs of doctors, patients, or policymakers enables the development of models that align with real-world practice. Involve them in the process, as they understand the medical context and can help prevent irrelevant or incorrect assumptions.
Understand the decision-making process and context
An AI model should enhance an existing process, not replace it. Analyse how decisions are currently made to identify missing or improvable information. This prevents wasting valuable time on irrelevant analyses and solutions, ensuring that AI adds value to affordable, efficient, and sustainable healthcare.
Make it concrete and define success criteria
But how do you measure this value? Establish ‘success criteria’ that can be measured. Make them very, very concrete. Suppose a hospital wants to develop an AI model to identify patients who would benefit most from knee or hip replacement surgery. To achieve a meaningful outcome, the research question must be clear and measurable. Instead of asking, “How can we prevent unnecessary knee replacements?” make it specific: “Which preoperative factors predict an improvement of at least 11 points on the Oxford Knee Score (OKS) after surgery, thereby preventing unnecessary procedures?” This way, you can measure and determine if the model is valuable.

Fig. 1 The CRISP-DM framework
The foundation: Good and structured data
“Garbage in is garbage out.”
The quality of data determines the success of any data science project, especially in healthcare, where data privacy and ethics are paramount. The Data Understanding and Data Preparation phases in CRISP-DM lay the foundation for reliable analyses.
Diligence comes first
In healthcare, data privacy and ethics are understandably essential. Before working with data, ensure that it can be used and check for potential biases in the model. Also, verify the accuracy and completeness of the data. Poor or incomplete data leads to inaccurate models, which in healthcare can have serious consequences, such as incorrect predictions affecting medical decisions. For example, an AI model for skin cancer diagnosis was trained on a dataset with predominantly lighter skin tones, making it less accurate for patients with darker skin and more likely to miss cancer cases. This resulted in clinical bias and increased the risk of unequal care, which was only corrected after adding more diverse training data. Therefore, ensure a precise understanding of which data is used for which question or objective.
Saving time
Data Preparation is often overlooked in the rush to deliver ‘results.’ However, this step ultimately saves time. With structured and cleaned data, less time is needed for preprocessing, allowing more time for developing and interpreting models that perform better with high-quality data.

“The CRISP-DM model is one of the most widely used and recognised data science frameworks. Following this approach is also valuable for healthcare institutions.”
Frank Peters, Impact Steward
The technical core: Model development and evaluation
“The technical core.”
With a clear research question and structured data, the project’s technical core—Modelling and Evaluation—can begin. There are many different models (dozens), and not every model is suitable for every type of question. Proper knowledge of these models requires expertise that many institutions often lack and therefore seek externally.
A suitable model
Which type of model should be chosen? A Random Forest model, Naïve Bayes, k-Nearest Neighbours? Choose models that fit the problem and data. This is called hypothesis-driven modelling. Each model has its pros and cons and is more or less suitable for the issue at hand.
Test it out
Whether the chosen model is indeed the right one must be tested first. Take the time to optimise, evaluate, and validate models. This prevents choosing a model that performs poorly in practice. At the start of the COVID-19 pandemic, many models attempted to predict the severity of infections, but many were trained on small and incomplete datasets containing bias (e.g., only severely ill hospital patients). As a result, they later proved unreliable for general populations.
The ultimate goal: Implementation and impact
“The proof of the pudding”
In the deployment phase, insights are translated into practice. How do you move from reliable insights to actual actions or a usable solution in daily practice? A data science project is only successful if it has an impact: saving time for professionals and achieving better outcomes for patients. In short, real value.
Transparency and explainability first
Ensure users understand how the model works. In healthcare, this is crucial for building trust. The Epic Sepsis Model (Sepsis is a life-threatening condition where the immune system overreacts to an infection, potentially causing organ failure), used by many (American) hospitals, was a black-box model without clear explanations of why a patient was classified as high-risk. Doctors only received a warning (“Sepsis risk high”) without insight into the factors behind this conclusion. As a result, they distrusted the model and ignored the warnings, especially due to frequent false positives.
Continuous learning
Like any process, machine learning models require maintenance. Monitor their performance and evaluate them regularly. This is essential to ensure they remain accurate.
From pilot to practice
In summary, every data science or AI project requires a serious, process-oriented, value-driven approach. The CRISP-DM model supports this by focusing on the right research question and data quality. This increases the likelihood of transitioning from pilot to practice.
Impact Stewards specialises in supporting these processes and can guide organisations in successfully implementing projects. Want to learn more about how Impact Stewards can help with your challenge? Contact us.
Curious what we can do for you?
Frank Peters
frank.peters@impactstewards.com