Algorithms are commonly used in diagnostics and healthcare for producing risk scores, diagnoses, alerts, etc; guiding clinical decisions; and helping to standardize care. Some are so simple that you might not think of them as an algorithm: BMI, for example. Others are recognized as what we typically think of as an algorithm, such as estimated glomerular filtration rate to reflect kidney function, the model for end-stage liver disease (MELD) score for predicting the prognosis of patients with liver disease and so on. Increasingly, with the advent of machine learning and AI, much more complex algorithms are being developed that can process much larger data samples and find patterns that otherwise would be hard to identify. Such new algorithms are not always simple mathematical equations and can sometimes be seen a “black box”. Here, challenges might arise when confirmation of appropriate algorithmic function is necessary. Also, operating these new complex algorithms in real-time, in a clinical setting, might require new and more complex ICT infrastructure including multiple software systems, located at different (virtual) locations, and not always within the span of control of the laboratory or hospital. This might increase the risk of algorithmic failure and generation of incorrect results.

As for any clinical laboratory test, there is a need to assure the quality of (complex) algorithms used as clinical test. Simple mathematical calculations are generally within the span of control of the laboratory and associated with a very low risk of failure after proper initial validation and implementation. Complex algorithms, however, might be associated with additional risks for reporting erroneous results. To manage these risks and ensure proper algorithmic function, running complex algorithms in a clinical setting might require additional quality control (QC). CLI chatted with Dr Huub H. van Rossum (The Netherlands Cancer Institute, The Netherlands) to find out more about how he and his team have developed QC for their mSTOP algorithm for the early identification of non-small cell lung cancer patients who do not respond to immune checkpoint inhibitor therapy.

Complex algorithms and prediction models are increasingly being used in clinical laboratories, can you give us an example of such an algorithm, please?

Firstly, in our clinical lab, and I think in many clinical labs, some simple algorithms are commonly used. For example, the CKD-EPI algorithm, which is probably one of the most commonly used algorithms in medicine, is mathematically actually quite complex. It’s not easy to calculate it by hand, and, additionally, you need to know the right parameters, which are in the formula, because they relate to the method and standardization you use. We also use a couple of other algorithms, such as the correction for hemolysis. The use of this algorithm is somewhat debatable, but we did quite a lot of development and validation of these rather simple algorithms [1,2]. These kinds of algorithms are generally computed in software available in the labs or controlled by the labs and so is within the span of control within the lab. This means that the lab can ensure the quality of the algorithm and verify that it’s correctly programmed according to their own practices. In general, this is easily done and there’s no reason to suspect that things have changed or even that there is a risk that things are changed.

Secondly, however, different kinds of models of algorithms are increasingly being used, such as clinical risk scores, which are now also quite common, that are being developed that only focus on laboratory data, such as the ROMA (Risk of Ovarian Malignancy Algorithm) or the GAAD for hepatocellular carcinoma, KELIM score for ovarian cancer follow-up and so on. These can be in many different clinical fields but as I’m in a cancer centre, I’m more familiar with that area. So, for instance, within the gynecology field, the ROMA score is perhaps the most well known, which only uses some tumour markers (CA-125 and HE4) along with the patient’s menopausal status to estimate the likelihood of an pelvic mass being ovarian cancer. In the Netherlands, though, we generally use other risk scores that also incorporate imaging characteristics as well as tumour marker data. However, this is not under the control of the labs but the physicians, and, generally in this situation, it’s therefore operated at another place. So, we don’t control it and we don’t have any insights on its functioning, and the only thing we can do is that we can inform them that they have all the relevant information to be able to use those scoring systems in an appropriate way.

Thirdly, the other type of algorithm that is increasingly being used are more advanced algorithms that are not necessarily easy to describe by a simple algorithm or algorithmic function. These can be AI based, are very complex to program, and are often performed outside of the lab.

In our setting, we developed the longitudinal machine learning (ML)-based Serum Tumor marker-based Outcome Prediction (mSTOP) model to identify patients with advanced non-small cell lung cancer (NSCLC) recently started on immune checkpoint inhibitor treatment who will not benefit from this therapy. mSTOP is based on the measurement of both CEA and CYFRA 21-1 just before start of treatment and at 6 weeks after start of treatment.

The complexity of this model exists in two ways. The first is that it uses ML-based models, and so it’s not like a simple regression or linear function, but the algorithm itself is more complex. We cannot just compute it in our middleware or in our Laboratory Information System (LIS). And second, it’s a longitudinal model, which means that it uses multiple results over time. Finally, one further thing that we have incorporated – because this model is specifically for a certain population of lung cancer patients – is that we try to also check whether the test is appropriately being used. This means that we can use the clinical information to identify these patients for whom the test is suitable. All this information is then joined together and we developed this whole specific, external, separate script that computes the data and the result is then reported back to the LIS, which we report to the physician. As you can imagine, all these factors are quite complex – data is fed in from different sources (so not just data under the control of the lab but also data that originates from the electronic healthcare records), and the calculation is also done externally. So, we developed the script that can run all that, which is based on the R programming language (Fig. 1).

Fig. 1. Technical set-up of mSTOP ICT infrastructure
EHR, electronic health record; LIS, Laboratory Information System.
From: van Rossum HH, van der Schaar M, Buma AIG, Moritz R, van Balen DEM et al. Clinical validation and implementation of mSTOP, a machine learning-based, longitudinal prediction model for the early identification of non-small cell lung cancer patients who not benefit from immune checkpoint inhibitor treatment. Clin Chem Lab Med. 2026;64(9):2075–2083. PMID: 42054314 (https://doi.org/10.1515/cclm-2025-1624); published under the Creative Commons Attribution 4.0 (CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/deed.en).

What is quality control and why is it needed in the clinical lab?

Running quality control (QC) checks is something we do routinely, and it’s really a quantitation of the quality of the results. This can be done by either internal QC, running a sample on your analyser, and having your own tolerance levels in which you say the assay is within our quality specification (so we can just continue) or not (in which case something has to happen). There are a lot of other QC systems as well, and we can add these to the QC plan based on the clinical risk associated with assay failure [3]. One other thing we do for several high-risk assays is that we continuously determine the average of the patient results (patient moving average or patient-based real-time QC), and this average should be in a certain range. If it’s outside of that range, it might also indicate something’s wrong and standard procedures for follow-up, including running internal QC, are in place [4,5]. So there are different ways to assure the quality of clinical tests. QC, by definition is a measure of the quality: usually to check this you have a QC sample that you know the value of and when you test it the result needs to reflect that value – or at least be within a certain tolerance. There are a lot of other systems available to help assure the quality of the results, such as patient moving averages of patient-based real-time QC that are not necessarily truly controlling the quality; rather, they flag when there is a change that the quality of a test (or sample used) is outside certain quality specifications, which is different.

What does QC look like when applied to complex algorithms?

There are different aspects to this, actually. One aspect is the impact of the assay – how does the normally occurring analytical variation, or technical error of the assays that produce the input data of complex algorithms, affect algorithmic output and performance. There is currently quite some focus on this. When I talk to clinicians they are more pragmatic about this and say that this variation was also present in the data used to obtain and validation the complex algorithm, so why the fuss? Another relevant aspect generally recognized is to ensure proper algorithmic use and the clinical performance of a test. This is largely done by ensuring the algorithm is used for the proper clinical population. This is relevant as the diagnostic performance metrics like positive prediction values (PPV) and negative predictive values (NPV) are highly dependent on the prevalence of a clinical condition in the tested population. If the prevalence or clinical characteristics change significantly change, the clinical performance of a complex algorithm might also significantly change. This can occur when the test is applied in an altered the clinical settings, but also by major developments in clinical practice of the original intended population. This is relevant as the PPV and NPV are probably most relevant to the physician as they represent the probability that a positive or negative test or algorithmic value is truly associated with a certain clinical outcome.

In our papers we really focused on a third type of QC; a set-up only used to control the technical algorithmic functioning and ensuring the algorithm keeps reporting the same output from the same input data.

How have you applied QC to work with your mSTOP model?

This is very technical, and again there are different aspects to QC.

QC for the ICT infrastructure

As mentioned above, our algorithm was based on the R programming language, and in our view, this raised two potential QC issues. One is that the system for running the R code also includes software package updates, and the second is that it’s actually published on a part of the server that it’s not necessarily fully under our control. Another challenge is that mSTOP is based on ML, so it’s sometimes very difficult to predict exactly what is going on. This means that when we thought about developing the mSTOP model we could see some risk of changes occurring that we might not be aware of that could impact mSTOP values or reporting, and we realized that for mSTOP to operate in real time in a clinical setting we would have to address this risk. So, we schedule a daily QC run, just like we do for any other clinical assays. However, this is not a true test running on an analyser platform, but involves having input data that is sent to the whole mSTOP ICT system so that we can test that the ICT infrastructure is still working appropriately. So, we use some scenarios based on clinical settings: for instance, low tumor marker value, a positive mSTOP, a negative mSTOP, an mSTOP value close to the clinical cut-off and decision level, etc. All these scenarios were actually based on the original validation study, and the algorithm needs to come up with the exact same value as the original study (Table 1). This is simply a technical QC of both the mSTOP algorithm as well as the ICT infrastructure developed to operate it. It doesn’t include, let’s say, if we have a drift in assay, how does this affect the performance of the model? It’s really just to check whether the ICT infrastructure still works appropriately.

Time frame

Another question is how often to run the QC test or in this care run the procedure? For mSTOP, we thought about this carefully, because we report the results in real time. Particularly, the clinical procedure and impact of erroneous results was assessed. In the mSTOP setting, decisions largely depend on other parameters and are not irreversible, like for example, when you report a blood glucose level and the physicians immediately stop/give insulin, etc. Therefore, we scheduled the QC on a daily basis and aligned this with the other internal QC as we would do for other laboratory tests in our lab, which we judged to be an appropriate time scale.

Future proofing the algorithm

An aspect that is relevant here is that clinical practice changes. This is a good thing because it basically means that practice is developing and is improving patient care. However, if we have such a test set up and validated for a specific setting using a certain treatment and that changes, then we should also control the clinical performance of such a test in the new generation of treatment regimens. The point here is to ensure that a model can still be used in the future and what needs to be done to make keep that model relevant. For mSTOP a positive test indicates a large probability that the patient will not respond to treatment. Currently, there are no effective next-in-line treatments available, only chemotherapy that might increase the life expectancy with 3 months or so, but is also associated with severe adverse effects and reduced quality of life. mSTOP should then trigger the discussion on how to proceed clinically and discuss patient end-of-life preferences. Potentially, when in the future a more effective less toxic next-in-line treatment becomes available, the consequences of a positive mSTOP might be significantly different. In such a scenario, mSTOP might even enable improved survival of a group of patients, as early identification of non-responding patients is essential to enable on-time conversion to next-in-line treatment and to not miss this clinical opportunity!

Another challenge is the generalizability of the mSTOP algorithm to other CEA and CYFRA 21-1 assays. The mSTOP algorithm is developed and validated for our in-house Roche CEA and CYFRA 21-1 assays. As for many tumour markers, results obtained by difference assays are not interchangeable [6]. It is highly relevant to investigate whether and how mSTOP could be applied on other analytical platforms.

Table 1. Scenario-based multi-parametric quality control (QC) set-up for mSTOP
Both Cyfra 21-1 and CEA values represent (µg/L) units. BL, baseline; FU, 6 weeks follow-up.
From: van Rossum HH, van der Schaar M, Buma AIG, Moritz R, van Balen DEM et al. Clinical validation and implementation of mSTOP, a machine learning-based, longitudinal prediction model for the early identification of non-small cell lung cancer patients who not benefit from immune checkpoint inhibitor treatment. Clin Chem Lab Med. 2026;64(9):2075–2083. PMID: 42054314 (https://doi.org/10.1515/cclm-2025-1624); published under the Creative Commons Attribution 4.0 (CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/deed.en).

What future developments do you envisage in this field?

There’s a lot of focus on algorithms and data now, and I’m sure people are trying to develop a lot of models. Sometimes this is a goal of its own, although I think a better way is to really think about the clinical user – it is really important to think about how the clinicians can use the algorithm to truly improve patient outcomes, quality of life and/or reduce healthcare costs. Here a diagnostic algorithm might not always be useful, if other diagnostic procedures would always be performed that can also address several other questions and can immediately be used for other additional analysis. Also, focussing on the area under the receiver operating curve (AUC-ROC) could potentially undermine algorithmic sensitivity and specificity individually, while the algorithmic application might be most suitable for ruling in or out. We developed mSTOP in a cancer setting, and it’s a longitudinal model. We now have the entire infrastructure and system in place to develop and validate such models and implement them for real-time clinical application under local IVD regulation. We are currently trying to develop similar longitudinal models for cancer patients, either based on different tumour markers or also perhaps different settings. Currently, we’re working on a project focusing on breast cancer to try to improve the interpretation of changes in tumour markers so that clinicians can use tumour marker data to make better follow-up decisions. Similarly for lung cancer patients receiving targeted therapies. If we know what a change in a tumour marker means clinically (a more accurate probability of having either a response to treatment, non-response to treatment or relapse after initial response), then the physician has a better basis to make clinical decisions based on the tumour marker results obtained.

For biomarkers, the potential clinical value of an algorithm is very dependent on the clinical field and particularly on the setting that you’re in. For example, having a lot of treatment options available is very different from when there are no treatment options, and this really affects what the model should be able to do. If there are a lot of options, you really want to be on time to identify non-responding or recurrent cancers, because you don’t want to miss the therapeutic window. If you don’t have a treatment option, there’s really no reason to monitor the patient, because nothing can be done.

In the field of cancer, major developments in biomarkers are currently ongoing. There are new technologies, such as cell-free DNA (or circulating tumour DNA) liquid biopsy technologies, that are entering the clinical practice. These probably allow for monitoring of many more types of cancers in a more accurate way. Similarly to the more traditional tumour markers, I think also the longitudinal interpretation of these values, need to be interpreted by means of algorithms. I think, then, at least in cancer care, these kinds of follow-up models might be very powerful and valuable. Having methods available to develop longitudinal models independently of the type of biomarker, and ways to truly implement them in clinical laboratories worldwide, constitutes a major goal of the laboratory medicine field to really be of value to improve the clinical cancer practice.

 References
1. Colak S, Tasdemir O, van der Schaaf M, Opdam F, van den Noort V, van den Broek D, van Rossum HH. Design, validation and performance of aspartate aminotransferase- and lactate dehydrogenase-reporting algorithms for haemolysed specimens including correction within quality specifications. Ann Clin Biochem. 2024;61(4):239-247. PMID: 31495183 (https://doi.org/10.1177/0004563219878475).
2. van Rossum HH. Demonstrating the feasibility of accurately and reliably correcting potassium results for mildly hemolytic samples using a new experimental design. Clin Chim Acta. 2021;522:83-87. PMID: 34418365 (https://doi.org/10.1016/j.cca.2021.08.019).
3. van Rossum HH. Technical quality assurance and quality control for medical laboratories: a review and proposal of a new concept to obtain integrated and validated QA/QC plans. Crit Rev Clin Lab Sci. 2022;59(8):586-600. PMID: 35758201
(https://doi.org/10.1080/10408363.2022.2088685).
4. van Rossum HH, van den Broek D. Ten-month evaluation of the routine application of patient moving average for real-time quality control in a hospital setting. J Appl Lab Med. 2020;5(6):1184–1193. PMID: 32533149 (https://doi.org/10.1093/jalm/jfaa071).
5. van Rossum HH, van den Broek D. Design and implementation of quality control plans that integrate moving average and internal quality control: incorporating the best of both worlds. Clin Chem Lab Med. 2019;57(9):1329–1338. PMID: 30903753 (https://doi.org/10.1515/cclm-2019-0027).
6. van Rossum HH, Holdenrieder S, Ballieux BEPB, Badrick TC, Yun YM, Zhang C et al. Investigating the Current Harmonization Status of Tumor Markers Using Global External Quality Assessment Programs: A Feasibility Study. Clin Chem. 2024;70(4):669–679. PMID: 38385453 (https://doi.org/10.1093/clinchem/hvae005).

The interviewee

Dr Huub H. van Rossum PhD, EuSpLM
Faculty member at the Netherlands Cancer Institute; Founder and chair of the International Federation of Clinical Chemistry and Laboratory Medicine Working Group on Tumor Marker Harmonization (IFCC WG-TMH), and International Society of Oncology and Biomarker  (ISOBM) working group on tumor marker harmonization. Board member of the International Society of Oncology and Biomarkers (ISOBM)

Department of Laboratory Medicine, The Netherlands Cancer Institute, Amsterdam, 1066CX, The Netherlands

Email: h.v.rossum@nki.nl

For further information, see:

1. van Rossum HH, van Delft FA. Scenario-based multi-parametric QC for quality control of complex algorithms used in clinical care. Clin Chem Lab Med. 2026;64(10):2188–2191. PMID: 42321983 (https://doi.org/10.1515/cclm-2026-0759).
2. van Rossum HH, van der Schaar M, Buma AIG, Moritz R, van Balen DEM et al. Clinical validation and implemen-tation of mSTOP, a machine learning-based, longitudinal prediction model for the early identification of non-small cell lung cancer patients who not benefit from immune checkpoint inhibitor treatment. Clin Chem Lab Med. 2026;64(9):2075–2083. PMID: 42054314 (https://doi.org/10.1515/cclm-2025-1624).