Testing non-inferiority for accuracy of quantitative microbiological methods

van den Heuvel, Edwin; Heidari, Maliheh; Möllenhoff, Kathrin; IJzerman-Boon, Pieta

article

Testing non-inferiority for accuracy of quantitative microbiological methods

Authors Edwin van den Heuvel, Maliheh Heidari, Kathrin Möllenhoff and Pieta IJzerman-Boon compare statistical analysis methods for determining comparable accuracy of alternate and compendial microbiological methods.

FOR THE VALIDATION of alternate microbiological methods, it is necessary to demonstrate that the accuracy of the alternate method is equivalent or non-inferior to the compendial method. Based on an experiment with multiple test samples from different concentrations per microbiological method, researchers often conduct difference testing assuming normally distributed data. Moreover, researchers typically conduct balanced experiments, with the same number of samples per concentration. In this paper we evaluate two statistical approaches for demonstrating non-inferiority using Poisson distributed counts. The first approach does not make any assumptions on the relation between the mean enumeration and the concentration. The paper shows that (optimal) sample sizes for this approach depend on the selected concentrations, which can suggest a non-balanced experiment. The second approach models the relation between the mean counts and the concentration for both microbiological methods with the non-linear Mitscherlich function. This model-based approach has more power than the non-model-based approach for testing non-inferiority.

Download

Download this article

To download this article, please login or create an account. If you choose to create an account you'll gain access to our website content, as well as the opportunity to subscribe free-of-charge to European Pharmaceutical Review's journal and email communications.

Do you want to create an account?

Step 1 - Download this article

Please enter your email and choose a password for your account. Passwords must include at least 8 characters, a mixture of both uppercase and lowercase letters, at least one letter and number and at least one special character, e.g. ! @ # ? ]

Email address {{errors.email}}

Password {{errors.password}}

Confirm password {{errors.confirm_password}}

Step 2 - Subscription preferences

Please select whether you would like to receive European Pharmaceutical Review's journal in printed and/or digital form, and choose which types of email communication you would like to receive.

Check all that apply:

Select all

Printed version of European Pharmaceutical Review journal (Sent free-of-charge by post 5 times a year)

Digital version of the European Pharmaceutical Review journal (Sent free-of-charge via email 5 times a year)

Step 3 - Personal details

Please enter your personal details.

Title

Title other

First name

Last name

Job title

Your employer

Telephone

Step 4 - Location details

Please enter your location details. We may use these details to personalise the content you see.

Address 1

Address 2

Town/City

Zip/Post code

Country

Step 5 - Areas of interest

Please select your areas of interest. We may use these details to personalise the content you see.

Tick all that apply:

Select all

Step 6 -

Tick all that apply:

Select all

I would like to receive relevant information and insights from the sponsor of this issue, FUJIFILM, Merck and Nelson Labs

By completing this form you are agreeing to our terms and conditions and privacy policy. As part of your content download, and in compliance with GDPR, we will share your data with the specific sponsor(s)/partner(s) of this content who may wish to contact you. You may opt-out at any time by using the unsubscribe link in one of our, or our sponsor's, marketing emails, or by contacting [email protected].

Privacy policy / terms & conditions

Please review and accept European Pharmaceutical Review's privacy policy and terms & conditions

By completing this form you are agreeing to our terms and conditions and privacy policy. As part of your content download, and in compliance with GDPR, we will share your data with the specific sponsor(s)/partner(s) of this content as we believe you have a legitimate interest in this and other similar content. You may opt-out at any time by using the unsubscribe link in one of our, or our sponsor's, marketing emails, or by contacting [email protected].

Introduction

To validate an alternate microbiological method (AMM), the guideline USP <1223>¹ recommends enumerating several test samples of different concentrations with both the AMM and compendial method (CMM). It recommends executing an equivalence or non-inferiority test, demonstrating that the ratio of mean counts of the AMM and CMM is within a certain margin. When equivalence or non-inferiority is demonstrated for each concentration, the accuracy of the AMM is considered equivalent or non-inferior to that of the CMM.

Equivalence or non-inferiority has rarely been implemented. Green and Randell² proposed a two-sample t-test per concentration for an experiment with 30, 75, 150, 225 and 300 colony forming units (CFU) and 12 samples per concentration and method. Chae et al.³ used a two-way ANOVA (analysis of variance) for an experiment with 50, 150, 350 and 500 CFUs, while Costanzo et al.⁴ used the R2 from a linear regression analysis on six concentrations to evaluate accuracy. Nguyen et al.⁵ calculated and visualised the recoveries at single concentrations, but without using formal statistical methods. Gordon et al.⁶ used a two‑sample t-test for concentrations five, 10, 30, 100 and 200 CFUs with 15 samples per concentration and method. They came close to an equivalence approach, since they also calculated a power value to indicate that they could make a reliable statement. Only Jones et al.⁷ conducted a non‑inferiority approach on a four-concentration experiment using a log‑linear model for the relation between the mean counts and the concentration.

One advantage of such a model-based (MB) approach is that the full sample size is used to estimate a mean count for any concentration level, while a non-model-based (NMB) approach uses only the data per concentration. Thus, the MB approach is expected to be more efficient than the NMB approach. Secondly, the model helps to evaluate non-inferiority in between levels of concentrations. However, the MB approach does require an appropriate fit to the data and the calculation of the standard error, as the estimate of the accuracy is mathematically more complex. In this article we will compare the two statistical approaches.

Methods

To demonstrate accuracy of the AMM with respect to the CMM we will consider a non-inferiority approach for both the MB and NMB approach. Thus, the AMM must not count substantially less than the CMM. Non-inferiority at concentration λ is then formulated by

with μ₁(λ) and μ₀(λ) the mean count of the AMM and CMM at concentration level λ, respectively, and with δ ∈(0,1) the so-called non-inferiority margin. This margin is often selected equal to δ =0.3^6,7 or δ =0.2.⁸ The ratio μ₁(λ)/μ₀(λ) is considered the recovery of the AMM with respect to the CMM at concentration level λ.

To be able to test Equation 1, an accuracy experiment is conducted. It is assumed that multiple concentrations (m) are created with levels λ₁<λ₂<…<λ_m and n_i samples per method and concentration i∈{1,2,…,m}. The observed counts are denoted by Yhij for method h∈{0,1} at concentration i∈{1,2,…,m} and test sample j∈{1,2,…,ni}. The counts Y_hij are considered independently Poisson distributed: Y_hij~P(μ_h(λ_i)), contrary to the normality assumption frequently used in the literature.^2,4,6,8 Based on this data a statement on accuracy of the AMM with respect to the CMM over the range of concentrations [λ₁,λ_m] is formulated.

Please note: due to the complex format of the equations, we cannot guarantee the 100 percent accuracy of the versions within the text on this page – please download the digital issue or QA/QC Microbiology/RMM In-Depth Focus supplement for access to the accurate equations.

In the NMB setup we assume that the mean μ_h(λ_i) is equal to μ_h(λ_i)=exp{β_hi}, with β_hi an unknown parameter, without making assumptions on the relation between the mean count μ_h(λ_i) and concentration λ_i. The null hypothesis in Equation 1 is now reformulated into H₀:β_1i-β_0ii. The parameter β_hi can be estimated with maximum likelihood estimation (MLE): β_hi=log (Y_hi), with Y_hi=Σ_jⁿⁱ =1(Y_hij⁄nⁱ) the average count for method h at concentration λ_i. Using the asymptotic likelihood theory, the estimated standard error of the estimator β_hi is equal to τ_hi=1⁄[n_iY_hi]^1/2, making the estimated variance of the difference β_1i -β_0i equal to τ_i² =n_i -1([Y_0i]-1+[Y_1i]^-1). An approximate (1‑α)×100% lower confidence limit on β_1i-β_0i is then equal to

(2)

with z_q the qth quantile of the standard normal distribution function.

For the MB approach we will assume the three-parameter non‑linear Mitscherlich function⁹ of the form:

(3)

which is an extension of the relation used in Jones et al.⁷ The null hypothesis of inferiority at positive concentrations (λ>0) for the MB approach can now be written into

(4)

The parameters in the Mitscherlich function have a specific interpretation. The intercept β_h0 is the mean count when no microorganisms are present in the test samples. It may represent a false positive rate for blank samples when β_h0 is positive, but it may also represent enumeration of non-culturable microorganisms for the (alternate) microbiological method. A negative intercept would indicate that there is a range of concentrations that is not detected or enumerated by the microbiological method. The slope β_h1 relates to the probability of enumerating a single organism. This is best seen when the microbiological method is proportional (β_h0=0 and β_h2=1). The slope represents the proportion of CFUs counted in the test samples when β_h1∈(0,1], and at a concentration of λ=1 this can be seen as a probability. The slope β_h1 is then viewed as the expected proportion of test samples with one microorganism that are detected positively. When the slope is larger than one, the microbiological method may enumerate more than just CFUs (potentially including viable but non-culturable microorganisms) or it is more sensitive than a microbiological method that counts precisely what is expected to be present. The power parameter β_h2 can also be viewed as a measure of sensitivity over the range of concentrations. When β_h2 is less than one, the microbiological method starts to saturate at higher concentration levels. This may typically occur for fast growing organisms, since the colonies start to overlap and enumeration of all colonies becomes more difficult. A power larger than one indicates that the microbiological method is less sensitive at lower concentrations. This may imply that at lower concentrations the microbiological method does not enumerate all microorganisms in the test samples.

The Mitscherlich function in Equation 3 can also be estimated with MLE, using Poisson regression with the identity link function. A closed-form expression of the MLEs does not exist in general. The (1-α)×100% lower confidence limit for the difference log(μ₁(λ))‑log(μ₀(λ)) at concentration λ is now of the following form:

(5)

with μ_h(λ)=β_h0+ β_h1λ^βh2 and τ_h²(λ) the estimator for the first-order approximation of the variance τ_h²(λ)=VAR(log(β_h0+ β_h1λ^βh2)) using the delta method.¹⁰

Whenever the lower confidence limit L_NMB (λ_i) or L^3P_MB (λ_i) is above the non-inferiority margin log(1-δ), the null hypothesis in Equation 1 at concentration λ_i is rejected and the AMM is claimed to be non-inferior to the CMM at concentration λ_i. If, for the NMB approach, non-inferiority is demonstrated at all concentrations λ₁,λ₂,…, λ_m, it is assumed that non-inferiority holds over the full range [λ₁,λ_m], although evidence was only provided at concentrations λ₁,λ₂,…, λ_m. For the MB approach, non-inferiority can be determined at each concentration λ∈[λ₁,λ_m], because L^3P_MB (λ) is a continuous function of λ. We will compare the NMB and MB approach on non-inferiority in the concentrations λ₁,λ₂,…, λ_m only.

Comparisons of the NMB and MB approach

We will consider an accuracy experiment with five concentrations λ₁=5, λ₂=10, λ₃=20, λ₄=40, and λ₅=80. Enumerating beyond 80CFU is frequently an issue for the compendial method due to overlap in growth of CFUs. Going below 5CFU may require substantial sample sizes due to the large relative standard deviations (RSD(%) = (100%)⁄√(μ_h(λ)). We will only consider the non‑inferiority margin of δ=0.3.

The minimal sample size can be calculated per concentration for Equation 2 using asymptotics, a type I error of α=0.05, and a family-wise type II error of β=0.20. The latter means that we need higher powers at single concentrations to ensure an overall power of 80 percent for all five concentrations simultaneously. We have numerically optimised the sample sizes and the results are provided in Table 1 for recoveries of 100 percent, 95 percent and 90 percent at each concentration. Calculating sample sizes for the MB approach is less straightforward due to the complexity of calculating the standard error, so we will use the sample sizes of the NMB approach.

To compare the two approaches, we conducted simulations for three scenarios. These scenarios represent different shapes of the recoveries (Figure 1) for potentially realistic examples. They all show a recovery within the non-inferiority margin of δ=0.3 and the experiment should, in principle, declare non-inferiority across the full interval of concentrations. In the first scenario (the blue line in Figure 1), the AMM (μ₁(λ)=0.8λ^1.1) has no false positives and is less sensitive at lower concentrations than at higher concentrations. The CMM (μ₀(λ)=0.2+1.2λ^0.9) has a very small false positive rate and becomes less sensitive at higher concentrations. The recovery at 5CFU is below 100 percent (at 88.5 percent), but quickly increases to a level above 100 percent, making the AMM superior to the CMM for concentrations above 8.6CFU.

In the second scenario (the red line in Figure 1), the AMM μ₁(λ)=1.2+0.65λ^1.075 is worse than in the first scenario, since it has a moderate false positive rate and is less sensitive across the full range of concentrations. The CMM (μ₀(λ)=0.2+1.1λ ^0.95) is very similar to the first scenario and even closer to the ideal proportional relation (μ₀(λ)=λ). The recovery at 5CFU starts around 92.3 percent and drops to 89.2 percent at 10CFU. From that point the recovery increases slowly and the AMM becomes slightly superior for concentrations beyond 57.5CFU. In the third scenario (the green line in Figure 1), the recovery declines from 108.4 percent at 5CFU to 86.5 percent at 80CFU. The CMM is the ideal method with μ₀(λ)=λ, while the AMM (μ₁(λ)=0.1+1.2λ^0.925) is rather sensitive at lower concentrations, but has more difficulties at higher concentrations.

Figure 1: Recoveries for three scenarios

Simulation of the accuracy experiment was straightforward. At each concentration λ_i, we simulated n_i independent Poisson distributed counts Y_0ij for the CMM having mean count μ₀(λ_i) from one of the scenarios and additionally simulated n_i Poisson distributed counts Y1ij for the AMM having mean count μ1(λ i) from the same scenario. Based on the simulated experiment we calculated Equation 2 for the NMB approach and Equation 5 for the MB approach at all five concentrations. For both the NMB and MB approach we also calculated the power for claiming non‑inferiority at all concentrations used in the experiment. Thus, we compared min_i=1,2,…,mL_NMB(λ_i) and min_i=1,2,…,mL_MB(λ_i) with the non-inferiority criterion log(1-δ). For each setting we simulated 1,000 simulations. We performed all simulations and analyses with [R].

The simulation results on the power of both approaches are provided in Table 2. We only use the sample sizes corresponding to a recovery of 100 percent and 95 percent, since the sample sizes with 90 percent recovery were unrealistically large. For all three scenarios we observe that the MB approach is more powerful than the NMB approach at each concentration and therefore the MB approach also outperforms the NMB approach over all concentrations simultaneously. Since the sample sizes have been calculated for a recovery of minimally 95 percent across the range of concentrations, we do not obtain a power of at least 80 percent at each concentration, since the recoveries were less than 95 percent at certain concentrations and scenarios. However, the MB approach does obtain approximately 80 percent power or more for all three scenarios when we choose the sample sizes (34, 23, 16, nine and five). More importantly, for scenario three, the MB approach already reaches a power well above 80 percent for the optimal sample sizes (25, 17, 10, six and four) that were calculated for a recovery of 100 percent.

Conclusion

We have provided details on two statistical analysis methods for testing non-inferiority of AMM accuracy with respect to the CMM. The NMB approach has the advantage of not making assumptions on the relationship between mean counts and concentration. This method was used to determine appropriate sample sizes and showed that imbalanced accuracy experiments are most appropriate. The test statistic for non-inferiority showed proper power values when sample sizes were calculated for the appropriate recoveries. The MB approach, using the three‑parameter non-linear Mitscherlich function, shows better power values than the NMB approach. The MB approach can achieve 80 percent power or more for recoveries at levels below 100 percent with sample sizes calculated at 100 percent recovery for the NMB approach.

About the authors

Edwin van den Heuvel studied mathematics at the University of Amsterdam. He received his PhD in mathematical statistics in 1996 from the same university. He has worked for six years for the consultancy firm IBIS UvA BV as a professional statistician, and eight years for the pharmaceutical company MSD as a statistical manager. He then returned to academia where he became a full professor in medical statistics at the University of Groningen and four years later a full professor in statistics at the Eindhoven University of Technology. He has held adjunct professorship positions at McMaster University and Boston University and is member of the statistics expert committee of the United States Pharmacopoeia. His research focuses on the development of statistical methods for the analysis of complex structured data sets and he is an expert in statistics for microbiology.

Maliheh Heidari studied Applied Statistics for her BSc at Shahid Beheshti University (SBU), and she obtained her MSc in Mathematical Statistics at Allameh Tabataba’i University (ATU) in Tehran, Iran. She is a PhD student in Statistics at Eindhoven University of Technology (TU/e). Her research is focused on efficient designs and analysis for the validation of rapid quantitative microbiological methods.

Kathrin Möllenhoff graduated in Mathematics from Wuppertal, Germany in 2008. Afterwards she completed her PhD at the Ruhr-University in Bochum where she defended her doctoral thesis in 2017. Her research focuses on the development of statistical methodology, particularly needed for clinical research and in drug development. Her two main research interests are methods for trials with small sample sizes, as required for instance for rare diseases, and survival analysis. After two years as a PostDoc in Bochum and in Cologne at the Center for Medical Statistics and Bioinformatics, Kathrin moved to TU Eindhoven where she started working as an Assistant Professor at the Department of Mathematics and Computer Science. Currently she is an Assistant Professor for biostatistics at Heinrich-Heine-University in Düsseldorf, Germany.

Pieta IJzerman-Boon holds a Ph.D. in Statistics from the University of Twente, the Netherlands. She has 11 years of experience in clinical statistics plus 11 years in non-clinical statistics, where she is a Principal Statistician at the Center for Mathematical Sciences, MSD Oss. She is also a Senior Statistician in a research collaboration with the Eindhoven University of Technology to develop statistical methods for microbiology. She gave multiple presentations and (co-)authored papers on this topic. She is an ARMM workstream member of BPOG.

References

USP<1223>. Validation of alternative microbiological methods, In United States Pharmacopoeia, 2015, vol. USP40-NF35, pp. 1756-1770. Rockville, MD: U.S. Pharmacopoeial Convention.
Green S, Randell C. Rapid microbiological methods explained. Microbiological Contamination Control in Pharmaceutical Clean Rooms, 2004; 157.
Chae GT, Stimson J, Emelko MB, et al. Statistical assessment of the accuracy and precision of bacteria- and virus-sized microsphere enumerations by epifluorescence microscopy. Water research. 2008; 42(6-7): 1431-40.
Costanzo SP, Borazjani RN, McCormick PJ. Validation of the Scan RDI for routine microbiological analysis of process water. PDA Journal of Pharmaceutical Science and Technology, 2002; 56(4): 206-19.
Nguyen K, Mills A, Sage A, Jones D. Validation of the Growth Direct™ system to perform pharmaceutical water bioburden analysis. European Journal of Parenteral & Pharmaceutical Sciences, 2015; 20(3).
Gordon O, Goverde M, Staerk A, Roesti D. Validation of Milliflex® Quantum for bioburden testing of pharmaceutical products. PDA Journal of Pharmaceutical Science and Technology, 2017; 71(3): 206-24.
Jones DL, Parsaei N, van den Heuvel ER. Primary validation of the Growth Direct environmental monitoring system and media, Institute of Validation Technology, 2020; 26(1).
Michelutti L, Bulfoni M, Nencioni E. A novel pharmaceutical approach for the analytical validation of probiotic bacterial count by flow cytometry. Journal of Microbiological Methods, 2020; 170:105834.
Box GE, Lucas HL. Design of experiments in non-linear situations. Biometrika, 1959, 46(1/2): 77-90.
Billingsley P. Convergence of probability measures. John Wiley & Sons; 1968.

Cookie	Description
cookielawinfo-checkbox-advertising-targeting	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Advertising & Targeting".
cookielawinfo-checkbox-analytics	This cookie is set by GDPR Cookie Consent WordPress Plugin. The cookie is used to remember the user consent for the cookies under the category "Analytics".
cookielawinfo-checkbox-necessary	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-performance	This cookie is set by GDPR Cookie Consent WordPress Plugin. The cookie is used to remember the user consent for the cookies under the category "Performance".
PHPSESSID	This cookie is native to PHP applications. The cookie is used to store and identify a users' unique session ID for the purpose of managing user session on the website. The cookie is a session cookies and is deleted when all the browser windows are closed.
viewed_cookie_policy	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
zmember_logged	This session cookie is served by our membership/subscription system and controls whether you are able to see content which is only available to logged in users.

Cookie	Description
cf_ob_info	This cookie is set by Cloudflare content delivery network and, in conjunction with the cookie 'cf_use_ob', is used to determine whether it should continue serving “Always Online” until the cookie expires.
cf_use_ob	This cookie is set by Cloudflare content delivery network and is used to determine whether it should continue serving “Always Online” until the cookie expires.
free_subscription_only	This session cookie is served by our membership/subscription system and controls which types of content you are able to access.
ls_smartpush	This cookie is set by Litespeed Server and allows the server to store settings to help improve performance of the site.
one_signal_sdk_db	This cookie is set by OneSignal push notifications and is used for storing user preferences in connection with their notification permission status.
YSC	This cookie is set by Youtube and is used to track the views of embedded videos.

Cookie	Description
bcookie	This cookie is set by LinkedIn. The purpose of the cookie is to enable LinkedIn functionalities on the page.
GPS	This cookie is set by YouTube and registers a unique ID for tracking users based on their geographical location
lang	This cookie is set by LinkedIn and is used to store the language preferences of a user to serve up content in that stored language the next time user visit the website.
lidc	This cookie is set by LinkedIn and used for routing.
lissc	This cookie is set by LinkedIn share Buttons and ad tags.
vuid	We embed videos from our official Vimeo channel. When you press play, Vimeo will drop third party cookies to enable the video to play and to see how long a viewer has watched the video. This cookie does not track individuals.
wow.anonymousId	This cookie is set by Spotler and tracks an anonymous visitor ID.
wow.schedule	This cookie is set by Spotler and enables it to track the Load Balance Session Queue.
wow.session	This cookie is set by Spotler to track the Internet Information Services (IIS) session state.
wow.utmvalues	This cookie is set by Spotler and stores the UTM values for the session. UTM values are specific text strings that are appended to URLs that allow Communigator to track the URLs and the UTM values when they get clicked on.
_ga	This cookie is set by Google Analytics and is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. It stores information anonymously and assign a randomly generated number to identify unique visitors.
_gat	This cookies is set by Google Universal Analytics to throttle the request rate to limit the collection of data on high traffic sites.
_gid	This cookie is set by Google Analytics and is used to store information of how visitors use a website and helps in creating an analytics report of how the website is doing. The data collected including the number visitors, the source where they have come from, and the pages visited in an anonymous form.

Cookie	Description
advanced_ads_browser_width	This cookie is set by Advanced Ads and measures the browser width.
advanced_ads_page_impressions	This cookie is set by Advanced Ads and measures the number of previous page impressions.
advanced_ads_pro_server_info	This cookie is set by Advanced Ads and sets geo-location, user role and user capabilities. It is used by cache busting in Advanced Ads Pro when the appropriate visitor conditions are used.
advanced_ads_pro_visitor_referrer	This cookie is set by Advanced Ads and sets the referrer URL.
bscookie	This cookie is a browser ID cookie set by LinkedIn share Buttons and ad tags.
IDE	This cookie is set by Google DoubleClick and stores information about how the user uses the website and any other advertisement before visiting the website. This is used to present users with ads that are relevant to them according to the user profile.
li_sugr	This cookie is set by LinkedIn and is used for tracking.
UserMatchHistory	This cookie is set by Linkedin and is used to track visitors on multiple websites, in order to present relevant advertisement based on the visitor's preferences.
VISITOR_INFO1_LIVE	This cookie is set by YouTube. Used to track the information of the embedded YouTube videos on a website.

Recommended

Testing non-inferiority for accuracy of quantitative microbiological methods

Shares

Share this post

Email preferences