AI Retrofit: Tackling Challenges with Cutting-Edge Technologies

Visual inspection is an essential step in parenteral drug production, as it ensures the safety of the drug product in its container. Manual inspection and automated systems can achieve a similar level of sensitivity, but there are specific strengths and weaknesses for both methods. Pharmaceutical companies rely on AVI to overcome challenges associated with manual inspection, such as low inspection throughput and performance variations. However, AVI machines have a trade-off between the configuration of the AVI sensitivity and the false rejection rate (FRR). The more variations and details that an inspection task considers, the higher the number of falsely identified defects in safe products. This article discusses a project that explored AI capabilities to leverage the inspection performance on an existing application and gain additional business insights by tackling a real-world challenge.

Project Scope

Glass ampoules, as illustrated in Figure 1, offer protection against air and contaminations and are widely used as the primary packaging for pharmaceutical drugs. They are hermetically sealed by melting the thin top with an open flame, and once closed, provide superior imperviousness to gases and liquids.

One specific defect in the inspection of ampoules are black spots in the tip area (Figure 2), which occur during the fill and finish process. Today’s available machine vision technology is capable of catching these tiny black spots. The technology combines high-resolution cameras with a backlight setup, which highlights the tiny black spots as dark pixels on a white background in the image. The challenge of such systems occurs when additional bubbles of the liquid product stick in the tip area of the ampoule. Based on their size, geometric form, and location, these bubbles can look similar to black spots, which ultimately challenges classic machine vision algorithms to distinguish between bubbles and real defects. This common industry issue often results in an increase in the FRR.

The optical setup provided the resolution to visualize defects and therefore fulfilled the basic requirement of using advanced AI algorithms to tackle a real-world challenge in an existing production environment. Enhancing the inspection tasks with AI algorithms covered the central part of the project scope. An agile approach combined with specific requirements for a minimum viable product (MVP) enabled the team to integrate the AI box, collect images and store them in a classified image database, and deploy the first AI model within 24 hours. Following this approach not only established a realistic benchmark using AI algorithms in a qualified process, but also provided valuable business insights regarding the inspection process in a GxP environment.

Integrating AI into an Existing AVI MachineI Machine

A successful AVI process, illustrated in Figure 3, ensures ideal interaction between mechanical engineering, optical imaging, image processing, and automation. In this project, the ampoules are smoothly rotated and presented to the tip inspection stations, which encompasses the pre-process automation step. Two cameras capture the tip area from different angles, with five images per camera, to establish a 360-degree view during rotation. After processing these images and inspecting for defects, the AVI machine rejects defective products according to the result of the visual inspection process.

The main challenge in integrating the AI retrofit was determining how to enable access to the images in the vision system and then process them through a neural network and return a result within the given timeframe of the AVI process. Different solutions are available to solve this problem. The evaluation process for this use case resulted in the integration of the deevio AI-Box. The deevio AI-Box is a proprietary vendor solution that provided a powerful AI platform equipped with all the necessary interfaces to ensure a smooth integration into an automated process that was parallel to the traditional image processing algorithms, emphasized in the visual inspection process flow shown in Figure 3. Additional pre-processing steps, such as cropping images, further increased the AI processing time by a factor of two and optimized the bandwidth by focusing on the essential parts in the images.

Figure 4 illustrates the straightforward integration of the deevio AI-Box into the existing system architecture by connecting the AI box to the switching hub. A simple file transfer protocol and additional settings in the inspection stations relayed the images to the deevio AI-Box, which processed these images through neural networks within milliseconds to comply with the given time constraints of the machine. Finally, an inspection performance of 420 ampoules per minute, or seven ampoules per second, was achieved.

Key Success Factors in Using AI

The foundation for every successful AI project is to build a database that consists of high-quality images and high-quality data labels. Because the team used the existing vision system already in production, the high quality of the images itself was given, which made it easy for a human to distinguish between images without any defects, classified as OK, and images containing defects , which were classified as not OK (NOK). The starting point to train the first AI model was the image collection of a test set of ampoules, including defective samples and good samples, that were processed through the machine. Having the AI box running in parallel to production continuously increased the image database with real production experience and consequently enabled continuous but controlled learning of the algorithm. Figure 5 illustrates the AI learning approach within the continuous improvement process.

Two factors improved significantly: the accuracy of the AI model and the labeling process. First, it is imperative in the development of the AI model to have the images labeled by subject matter experts (SMEs) who are familiar with the visual inspection process, specifically with the defects. Second, determining how to handle the sheer amount of data produced by an AVI machine during the labeling process is also integral. During the MVP phase, over 3,000 images were collected and classified into a sustainable database. In addition to the user-friendly interface provided by deevio’s user interface tool, the confidence score of each image was also archived, which enabled a pre-sort of the latest images. This function made the labeling experience quite convenient and allowed the team to efficiently maintain the database and create a smooth labeling process.

Proof of Value

After the successful integration of the MVP, we had an AI system running that enabled us to gain further understanding of AVI in combination with AI. As a result, the project team started to explore the proof of value by focusing on three specific use cases.

Use Case 1: Benchmark of an Existing System Equipped with AI Algorithms

While the AVI machine ran in production, the AI system continuously collected images in parallel. This setup allowed us to compare the detection rate (DR) and the FRR with an existing and qualified system that was in production for over 6 years. Figure 6 shows the evolution of the inspection performance throughout the project timeline. The goal was to outperform the average baseline of 95% DR and 1.7% FRR generated on the AVI machine specifically for black spots. The first 3 weeks after the implementation were used to collect images and establish a database.

Optimizations on the first AI model achieved a DR of 99.2%, which outperformed the baseline by 4.2%. However, the FRR of 6.2% needed to be improved during the following weeks. Adding images, improving data label quality, and the recall optimization approach of the AI model—which is the performance metric targeted to decrease the FRR—were the primary goals in the improvement process. Finally, 3 months after implementing the deevio AI-Box, a second AI model was deployed that outperformed the baseline with a DR of 98.0% and a significant drop of the FRR from 4.9% to 1.3%, which is 0.4% better than the target FRR.

The performance of the AI model can still be further improved by optimizing the recall of the AI model and providing more well-labeled data in the future to decrease the FRR value close to 0%. The consequence might be that the DR value gets slightly worse. However, the addition of further data can solve this problem. All decisions taken in the optimization cycle are risk based and consider quality and business aspects. Creating a benchmark of the two systems not only confirmed the positive impact of AI, but also transformed the added value into tangible and comparable data that can be useful for business case considerations.

Use Case 2: Develop the Process Behind the Technology

In addition to the performance of new technology, its integration into an existing pharmaceutical environment is important, including its challenges. Therefore, processes were developed to integrate AI into a GxP environment. An important consideration was the evaluation of the most suitable algorithms to solve the specific problem statement. The deep learning (DL) method seemed to be a promising approach for our use case. DL is based on neural network architectures with multiple layers and has been successfully used to solve multiple problems in image recognition. The visual inspection process illustrated in Figure 3 identified AI algorithms as an algorithm set complementary to the traditional machine vision algorithms. Therefore, the process to deploy AI models according to Figure 5 was developed and adjusted to the AVI standard continuous improvement process to enable a smooth integration into the existing AVI process.

Previous experiences have taught us to use a supervised learning approach because of the possibility of freezing a model before the deployment on an AVI system. This enabled controlled learning and was an essential outcome for the qualification concept of an AI algorithm in a GxP environment. Furthermore, in the pharmaceutical validation processes, it is crucial to understand how and why AI models work. Therefore, integrated gradients were introduced to answer these questions and increase the understanding of AI models. This function aims to explain the relationship between a model’s predictions and its features by creating a colored mask that is overlaid on the original image. Based on the model’s decision, all important areas and pixels are then highlighted according to a color scheme that is instantly and intuitively visible to the user, which ultimately provides a better understanding of the model’s predictions.

Use Case 3: Business Insights from Implementing AI Algorithms into Existing Equipment

This specific use case explored the business perspective derived from the MVP. The development of an AI solution that was relatively easy and fast to implement and test as well as being cost efficient solved a real-world challenge. Further, it ultimately increased the awareness and understanding of AI technology, specifically the importance of image acquisition, capabilities of the hardware components, integration in current system architecture, and handling of large data sets. All these outcomes were consolidated and summarized as user requirements or implementation guidelines to support future AI projects. Finally, important lessons were learned to support a data-driven and AI-enabled organization, such as exploring required infrastructure, capabilities, interfaces, and processes; defining an AI governance; and establishing other valuable inputs to the AI strategy.

Key Takeaways

The impressive performance achieved by the AI MVP outperformed the existing system in only 3 months and highlighted the powerful potential of AI solutions. The AI retrofit project not only confirmed an improvement in the inspection performance but also enabled valuable insights into implementing AI technology in the manufacturing environment. On a technical level, user requirements—such as the importance of high-quality image acquisition, powerful hardware, and software components—were identified. One key success factor was establishing and maintaining a sustainable database, which set the foundation for other AI solutions.

The implementation of AI into an existing AVI machine also allowed us to develop guidelines and concepts to cope with existing processes and pharmaceutical regulations. Another significant outcome was using critical thinking and a risk-based approach when relying on AI algorithms. It is never a one-size-fits-all solution. Finally, the AI retrofit approach started with a clear value proposition, but it also leveraged additional business insights to generalize further AI implementations and enabled the organization to explore and improve AI capabilities for future projects.

Next Steps

An essential next step for the project team is to tackle the technology challenges identified during this project, such as the feasibility of updating the hardware components in the existing machine, equipping the vision system with the needed computing power, and integrating the AI solution in the AVI system, including different communication layers and user interactions (e.g., visualization or recipe and user management). The project’s successful outcome was also directly linked to the ability of the project team to develop new out-of-the-box ideas beyond the AI retrofit project, which consequently triggered new AI initiatives within the network and identified opportunities to improve AI capabilities. Finally, the project team is interested in fostering discussions around AI topics with other pharmaceutical companies, AI suppliers, and communities to exchange best practices and lessons learned to drive AI solutions in the pharmaceutical industry.