TWI731484B - Method and system for building medication library and managing medication via the image of its blister package - Google Patents

Abstract

Disclosed herein is an improved system for managing a medication and a method of building a medication library via images of a blister package of the medication. The method comprises steps of: (a) respectively obtaining a first image and a second image of a blister package of the medication, in which the first image and the second image include a first side and a second side of the blister package of the medication, respectively; (b) subjecting the first image and the second image to an image rectification treatment to respectively produce a first rectified image and a second rectified image; (c) juxtaposing the first rectified image and the second rectified image to produce a combined image; (d) processing the combined image to produce a reference image; and (e) building the medication library with the aid of the reference image. The system comprises an image processor, which is programmed with instructions to execute the present method.

Description

Method and system for establishing drug database and managing drugs through drug blister packaging images

The present disclosure relates to the field of drug management systems, in particular to a method and system that can classify and identify drugs through drug blister packaging images.

For the smooth operation of the hospital system and the provision of high-quality patient care, drug verification is a basic prerequisite for ensuring the safety of patients' medication. Although automated dispensing cabinets (ADCs) have been introduced into medical institutions for 20 years, there are still lapses in pharmaceutical matters. Since the administration of the fully automatic dispensing cabinet still relies to a large extent on the clinical staff’s precise compliance with the standard procedures, in particular, it relies on the human eye to identify the drug through the appearance of the drug, which will inevitably lead to the appearance of identification errors. . In order to reduce human error, such as storing similar medicines in the wrong medicine cabinet by mistake, or clinical staff may get medicines that look similar in appearance from a neighboring medicine cabinet, it is necessary to make further improvements to the medicine management system.

Deep learning is a method based on multi-level learning data representation in the machine learning series. These methods are obtained by combining simple but non-linear modules, each of which transforms a level of representation into a higher and more abstract level. Through a sufficient combination of these transformations, more complex functions (such as classification tasks) can be learned, such as the application of deep learning to clinical pharmaceutical management. Based on the basic function of deep learning to recognize the appearance of objects, supplemented by the systematic image processing of the appearance of drugs, deep learning is expected to bring solutions to the shortcomings of current drug dispensing technology.

However, considering that there are many types of drug packages that are similar in appearance, and general drug packages are composed of front and back sides. In order to increase the identification efficiency in a short time, it is necessary to provide an image processing method that can accurately obtain the appearance of the drug packaging, so that it can be applied to a deep learning system to improve the effectiveness of deep learning and the accuracy of identification.

In view of the above, it is necessary in the prior art to provide an improved method and system for drug management (for example, to classify and identify the drug via a drug blister package).

In order to provide readers with a basic understanding, the following provides a brief summary of the present disclosure. This summary of the invention is not an extensive overview of the disclosure, and is not used to identify the key/essential elements of the invention or outline the scope of the invention. Its sole purpose is to present some concepts of this disclosure in a simplified conceptual form as a prelude to the more detailed description presented later.

As embodied and broadly described herein, the purpose of the present disclosure is to provide an improved drug management system and an implementation method for identifying clinical drugs through the system, thereby greatly improving the efficiency and accuracy of dispensing drugs.

One aspect of the present disclosure relates to a computer-implemented method for building a drug database of drug blister packaging images. In some embodiments, the method includes: (a) obtaining a first image and a second image of the drug blister package, respectively, wherein the first image and the second image respectively include the drug blister package A first side and a second side; (b) performing image rectification on the first image and the second image respectively to generate a corrected first image and a corrected second image; (c) ) Juxtapose the corrected first image and the corrected second image to generate a merged image; (d) process the merged image to generate a reference image; and (e) use the reference image to create the Drug database.

According to some embodiments of the present disclosure, in step (b), the first image can be corrected by the following steps: (b-1) provide a reference rectangle; (b-2) select the first image Four corner points in the image, among which the four corner points constitute a quadrilateral; (b-3) Based on the correspondence between the reference rectangle provided in step (b-1) and the quadrilateral selected in step (b-2), calculate A first homography matrix of the correspondence between the reference rectangle and the quadrilateral; and (b-4) perform perspective correction on the first image according to the first homography matrix to Obtain the corrected first image.

According to some embodiments of the present disclosure, the quadrilateral in the aforementioned step (b-2) can correspond to an outline of the first side of the drug blister package.

Preferably, in the aforementioned step (b), the following steps are used to perform image correction on the second image: (bi) Based on the corresponding relationship between the corrected first image and the second image, calculate the A second homography matrix; and (b-ii) performing perspective correction on the second image according to the second homography matrix to obtain the corrected second image.

According to some embodiments of the present disclosure, step (c) includes: (c-1) extracting the first and second sides of the drug blister package from the corrected first image and the corrected second image Contours to generate a first contour image and a second contour image respectively; (c-2) identifying the corner points of each contour in the first contour image and the second contour image to determine the coordinates of the corner points; c-3) According to the coordinates of step (c-2), rotate the first contour image and the second contour image of step (c-1) to form a first processed image and a second processed image, respectively; And (c-4) the first processed image and the second processed image of the regularization step (c-3) to generate the combined image.

Preferably, step (c-2) is performed by a linear conversion algorithm or a centroid algorithm.

Preferably, step (d) is performed by a machine learning algorithm.

According to some embodiments of the present disclosure, the first side and the second side are the front side and the back side of the drug blister package, respectively.

Another aspect of the present disclosure relates to a medication management system, which includes a medication database established by the aforementioned computer-implemented method, an image capture device, an image processor, and a machine learning processor, designed to implement medication management.

Specifically, the drug database is used to provide a reference image; the image capture device is used to capture an image of a blister pack of a drug, and the image capture device includes a transparent plate for Place the medicine on it; a first image capturing unit is arranged on one side of the transparent plate for capturing a first image of the medicine at a first angle, wherein the first image contains the medicine A first side of the blister package; and a second image capturing unit, arranged on the other side of the transparent plate, for capturing a second image of the medicine at a second angle, wherein the second The image contains a second side of the drug blister package. The image processor is programmed to execute a first computer-implemented method for generating a candidate image through instruction programming, wherein the first computer-implemented method includes: (a) performing image correction on the first image and the second image respectively , To respectively generate a corrected first image and a corrected second image; and (b) regularizing the corrected first image and the corrected second image to generate the candidate image. The machine learning processor is programmed with instructions to execute a second computer-implemented method for comparing the candidate image with the reference image of the drug database established by the computer-implemented method described in claim 1.

According to some embodiments of the present disclosure, the aforementioned step (b) includes: (b-1) extracting the first side and the first side of the drug blister package from the corrected first image and the corrected second image The contour of the second surface to generate a first contour image and a second contour image respectively; (b-2) Identify the corner points of each contour in the first contour image and the second contour image to determine the corner point (B-3) According to the coordinates of step (b-2), rotate the first contour image and the second contour image of step (b-1) to form a first processed image and a second Two processing images; and (b-4) the first processing image and the second processing image of the regularization step (b-3) to generate the candidate image.

According to a preferred embodiment of the present disclosure, the first side and the second side are the front and back sides of the drug blister package, respectively.

According to some embodiments of the present disclosure, the first angle and the second angle are respectively 40 to 90 degrees with respect to the horizontal plane of the transparent plate.

In order to make the description of the present disclosure more detailed and complete, the following provides an illustrative description for the implementation aspects and specific embodiments of the present invention; this is not the only way to implement or use the specific embodiments of the present invention. The implementation manners cover the characteristics of a number of specific embodiments and the method steps and sequences used to construct and operate these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and sequence of steps.

I. Definition

For ease of description, the specific terms described in the specification, the embodiments and the appended patent scope are collectively described here. Unless otherwise defined in this specification, the scientific and technical terms used herein have the same meaning as understood and used by those with ordinary knowledge in the technical field of the present invention. In addition, without conflict with the context, the singular nouns used in this specification cover the plural nouns; and the plural nouns used also cover the singular nouns. Specifically, unless the context clearly indicates otherwise, the singular form "one" (a and an) used in the scope of the patent application herein and appended includes plural forms. In addition, in this specification and the scope of the patent application, expressions such as "at least one" and "one or more" have the same meaning, and both of them mean that they include one, two, and three. Or more.

Although the numerical ranges and parameters used to define the broader scope of the present invention are approximate numerical values, the relevant numerical values in the specific embodiments are presented here as accurately as possible. However, any value inevitably contains standard deviations due to individual test methods. Here, "about" usually means that the actual value is within plus or minus 10%, 5%, 1%, or 0.5% of a specific value or range. Or, the word "about" means that the actual value falls within the acceptable standard error of the average value, depending on the consideration of a person with ordinary knowledge in the technical field of the present invention. In addition to the experimental examples, or unless otherwise clearly stated, all ranges, quantities, values and percentages used herein (for example, used to describe the amount of material, length of time, temperature, operating conditions, quantity ratios and other similar Those) have been modified by "about". Therefore, unless otherwise stated, the numerical parameters disclosed in this specification and the accompanying patent scope are approximate values and can be changed according to requirements. At least these numerical parameters should be understood as the indicated effective number of digits and the value obtained by applying the general carry method. Here, the numerical range is expressed from one end point to another point or between two end points; unless otherwise specified, the numerical range described here includes the end points.

As used herein, the term "blister pack" (blister pack or blister package) covers any type of layered packaging in which the product is contained between sheet materials; wherein these sheet materials can be used in the technical field. It is bonded or encapsulated by methods well known to those skilled in the art. For example, these sheets can be bonded by heat and/or pressure activated glue. These are commercially available as individual sheets (for manual packaging) or as continuous rolls on rolls (for machine packaging). The main structure of blister packaging is a cavity or bag made of a formable mesh (usually thermoplastic). The cavity or bag is large enough to contain the items in the blister pack. Depending on the application, the blister pack may have a backing of thermoplastic material. For the pharmaceutical field, blister packaging is often used as a single-dose packaging for tablets, and contains drug information printed on the reverse side of the blister packaging. In addition, these sheets are available in various thicknesses. Generally speaking, the side of the cavity or bag containing the single-dose pill is regarded as the front side, and the side with the drug information printed on the other side is regarded as the back side.

The “rectification” or “correction” of the image used in this article refers to the use of specific methods to correct a distorted original image that has been shot at a non-frontal angle and thus deforms the target object in the image content to have a frontal angle and A process of orthophoto that unifies the scale and coordinates. Specific methods can be obtained through mathematical principles, such as matrices or multivariate linear equations.

The term "juxtapose" used in this article refers to placing two objects side by side. In the present disclosure, it specifically refers to the step of placing two images, especially two images with the same size after image processing, side by side. In the text, in order to clearly explain the relevant content, similar words such as "alignment", "side by side", and "merge" can be used to replace the description.

II. Implementation of the present invention

The present invention aims to provide an improved image processing, combined with an inductive deep learning method, so as to solve the problem of identification errors faced by conventional clinical dispensing. In addition, the present invention also aims to develop an automatic medication verification (AMV) device that uses a real-time operating system to reduce the workload of clinical staff.

Specifically, the so-called inductive deep learning refers to a method of processing or highlighting the features or images before inputting the features or images into the algorithm, so as to learn the feature information in a more emphasized manner. In order to better divide the identified target and expose the inherent descriptive features, various image processing steps are generally used to achieve feature processing. These steps are to integrate the front and back cut images of the appearance of the drug blister package into one. In a fixed-size template, it facilitates the subsequent classification process of the learning network.

1. Method of establishing drug database

The first aspect of the present disclosure relates to a method 100 for establishing a drug database in a computer-readable storage medium, and is used in conjunction with Figure 1 (a flowchart according to an embodiment of the present disclosure). Explain it.

It should be noted that before the method 100 is implemented, one or more original images of a drug blister package can be captured by any known method, such as an image capturing device (for example, a camera). Then, in step 102 of the method 100 of the present disclosure, the captured image is automatically forwarded to the device and/or system embedded with instructions for executing the method 100 of the present invention for the subsequent step of establishing a drug database .

In the method 100, the first image and the second image of the overall appearance of a drug in the blister package are first obtained respectively (step 102). The first image includes the first side of the drug blister package, and the second image includes the second side of the drug blister package. Specifically, a blister package is a single-dose package of tablets and contains drug information printed on the reverse side of the blister package. In a specific embodiment, since the blister package is formed by pressing two upper and lower sheets of material, it has two sides. Generally speaking, the side of the cavity or bag containing the single-dose pill is regarded as the front side, and the side with printed drug information is regarded as the back side. In other words, in the present disclosure, the first side or the second side of the drug blister package refers to the front side or the back side of the drug blister package, respectively.

After obtaining the first image and the second image including the complete surface of the drug blister package, image rectification is performed on the first image and the second image respectively to generate the corrected first image and the corrected second image ( Step 104).

Specifically, in step 104, image correction may be performed on the first image and the second image respectively. In accordance with Figure 1, in step 104, steps 1041, 1043, 1045, and 1047 are used to perform image correction on the first image; and steps 1042 and 1044 are used to perform image correction on the second image. First, step 1041 is to provide a reference rectangle. In the present disclosure, the reference rectangle is specifically a virtual front view image, and is aligned with the X-axis and Y-axis of the Cartesian plane coordinate system. It is preferably based on the reference background object that is also a rectangle, and is selected in the specific photographic field of view. A virtual rectangle with a specific aspect ratio inside. In the preferred embodiment of the present case, the fixed size for placing the transparent plate carrying the drug to be photographed can be used as the background reference object, and the four corner points of the transparent plate with the same aspect ratio in the front view shooting field can be selected to form A regular quadrilateral, as a reference rectangle. In another embodiment of this case, the outline of the drug blister package (corresponding to a quadrilateral with a fixed size and an aspect ratio) presented in a specific frontal field of view can also be used as the reference rectangle.

Then, in the first image, any four corner points that can form a quadrilateral are selected (step 1043). Specifically, the four corner points of a quadrilateral can be formed by arbitrary selection, and the quadrilateral can have any shape, such as a trapezoid, a rhombus, a parallelogram, an arbitrary quadrilateral, or a regular quadrilateral. In a preferred embodiment of the present disclosure, the quadrilateral is a quadrilateral formed corresponding to the four corners of the aforementioned transparent plate; in another preferred embodiment, the quadrilateral corresponds to the first drug blister package. Silhouette of one side. Specifically, depending on the situation, the quadrilateral is made to correspond to the front or back contour of the drug blister package.

After defining a quadrilateral for the first image, the step of finding the first homography matrix can be continued, as shown in step 1045 in Figure 1. The specific method is to calculate the first homography matrix of the correspondence between the reference rectangle and the quadrilateral according to the correspondence between the reference rectangle in step 1041 and the selected quadrilateral in step 1043. In the specific embodiments of the present disclosure, the outline of the drug blister package presented in a specific frontal field of view and the outline of the drug blister package selected in the first image can be based on the coordinate correspondence between the two in the coordinate system, Find the first homography matrix. In another specific embodiment, the corresponding relationship between the reference rectangle formed by the four corner points of the transparent plate in the field of view of the front view and the quadrilateral formed by the same corner points in the first image is used to obtain the first The homography matrix. The solution method of the homography matrix is common knowledge in the field, so I won't repeat it here.

After the first homography matrix is obtained, perspective correction can be performed on the first image obtained in step 102 to obtain the corrected first image. The corrected first image obtained through the aforementioned steps 1041-1047 is a front-view image showing the first side of the drug blister package, so as to facilitate the subsequent steps of contour extraction and regularization.

Continuing to cooperate with Figure 1, in step 104 of the method 100, the second image can also be corrected in the following steps. The steps include: calculating a second homography matrix between the first image and the second image based on the corresponding relationship between the corrected first image and the second image (step 1042); and comparing the second image according to the second homography matrix Perform perspective correction to obtain a corrected second image (step 1044). Specifically, the corrected first image and the second image both contain the same reference background object (for example, the aforementioned transparent plate), and the corrected first image should be a front view image, so it can be based on the corrected first image The relationship with the second image, and the second homography matrix between the two is obtained. Then, according to the second homography matrix, perspective correction is performed on the second image to obtain the corrected second image. As mentioned above, the corrected second image is a front view showing the second side of the drug blister package. According to a specific embodiment of the present disclosure, after correction, the relative position of the second surface of the drug blister package in the corrected second image is the same as the relative position of the first surface in the corrected first image.

It is worth mentioning that the image correction procedures of the first image and the second image can be performed simultaneously or sequentially, as long as the corrected first image and the corrected second image can be obtained separately. Optionally, if the first image is already in the front-viewing image state, only the image correction step (that is, steps 1042 and 1044) can be performed on the second image.

After the corrected first image and the corrected second image are obtained, in order to make the image have a fixed specification and format, the corrected first image and the corrected second image are normalized to generate a combined image (as shown in Figure 1). Step 106). This step is mainly to improve the accuracy of the appearance recognition of the subsequent machine learning program, and to process and highlight an image, so as to provide a clean image with predetermined characteristics (for example, fixed at a predetermined pixel size, etc.), and more It is better to remove the background part of the non-subject (ie, the image of the drug blister pack) in the clean image.

According to a preferred embodiment of the present disclosure, the aforementioned combined image specifically presents both the front and the back of the drug blister package, and in the original corrected first image and the corrected second image, the rest of the background of the non-drug blister package is removed section.

According to a specific implementation of the present disclosure, the specific regularization step 106 usually includes the following steps: (1061) extracting the contours of the first side and the second side of the drug blister package from the first image after correction and the second image after correction to generate the first contour image and the second contour image respectively; (1063) Identify the corner points of each contour in the first contour image and the second contour image to determine the coordinates of the corner points; (1065) According to the coordinates of step (1063), rotate the first contour image and the second contour image of step (1061) to form a first processed image and a second processed image, respectively; and (1067) The first processed image and the second processed image of the normalizing step (1065) are generated to generate a merged image of the present disclosure.

In step 1061, an image processor is used to process the corrected first image and the corrected second image respectively, with the purpose of extracting the contours of the first side and the second side of the drug blister package in the images, so as to generate a clear image. The first contour image and the second contour image of the defined contour. The image processor executes several algorithms to eliminate the background noise of the image and extract target features. Specifically, the extracted image in step 1061 is processed through the following image processing: (i) gray-scale conversion processing, (ii) noise filtering processing, (iii) edge recognition processing, (iv) convex hull calculation processing And (v) Find contour processing. It should be noted that the aforementioned processes (i) to (v) can be performed independently in any order, preferably in the order of (i) to (v).

Specifically, (i) use a color conversion algorithm to perform grayscale conversion processing to convert BGR colors (color) into grayscale; (ii) use a filter algorithm to perform noise filtering processing to remove background noise Minimize; (iii) use edge recognition algorithms to perform edge recognition processing to determine the coordinates of each edge of the drug blister package in the image; (iv) use convex hull algorithm (a type used in flat or other low-dimensional Find the convex hull algorithm of a finite set of points) perform the convex hull calculation process to calculate the actual area of the drug blister package in the image; and (v) use the contour definition algorithm to perform the contour finding process to extract And the main area of eye-catching drug blister packaging. By performing the aforementioned processes (i) to (v), a first contour image and a second contour image each with a defined contour are generated from the corrected first image and the corrected second image, wherein the background noise is Elimination, and the main part of the blister packaging is extracted and eye-catching to facilitate subsequent image processing procedures.

Step 1063 is continued, which defines the corner points of each contour in the first contour image and the second contour image. Regardless of the outline shape of the drug blister package, this step can determine the coordinates of at least one corner of each outline image. Preferably, the coordinates of the four corner points of the first contour image and the second contour image obtained from step 1061 are determined. The goal of this step is to predict at least three straight lines from the contour edge of the drug blister package, and then use geometric inference to obtain the quadrilateral closest to the blister package edge and the coordinates of each corner point relative to the plane coordinate system. Since the blister package may have multiple shapes and/or contours, different algorithms can be used to identify corner points from the first contour image and the second contour image. For example, if the outline shape of the drug blister package in the outline image is a conventional quadrilateral (such as a rectangle), a straight-line conversion algorithm can be used to identify the four corner points, and then establish the coordinates of the corner points. On the other hand, if the outline shape of the drug blister package in the outline images is an unconventional quadrilateral, such as an atypical polygon with three straight edges and one curved edge, then the centroid algorithm is used for corner recognition. Through the aforementioned design, regardless of the direction and shape of the drug package, the coordinates of each corner can be determined.

In step 1063, identifying the corner points of each processed image is not only to establish anchor points for the subsequent rotation step (1065) and regularization step (1067), but also to ensure that the entire drug blister package is contained in the first outline Image and the second contour image for subsequent analysis. It is worth noting that the coordinates of the four corner points determined for the aforementioned blister package outline must be arranged in a clockwise or counterclockwise manner. According to this, the first outline image and the second outline can be combined based on the determined coordinates of the four corner points. The image is rotated to a predetermined position (step 1065) to form a first processed image and a second processed image respectively. The predetermined location can vary with actual implementation requirements. In some embodiments, the predetermined position means that the short side and the long side of the drug blister package are respectively parallel to the X axis and the Y axis of the plane coordinate system. The better orientation method for blister packaging is to rotate the image only once to make the short and long sides parallel to the X and Y axes of the Cartesian coordinate system, respectively. In actual application, a variety of perspective conversion algorithms can be used to rotate the first contour image and the second contour image, which have a predetermined pixel size (for example, 448×224 pixels), so as to generate the first processed image and the second processed image with predetermined positions, respectively. Process images.

Next, in step 1067, the first processed image and the second processed image obtained in the foregoing steps are placed side by side and regularized to generate a combined image. It should be noted that the merged image obtained from this step can contain several combinations, which is helpful for establishing the drug database data as much as possible. The first processed image and the second processed image can be in an upright or inverted state, respectively. Preferably, the first and second processed images that are in the same direction and respectively present the front and back sides of the drug blister package are aligned with each other to generate a merged image that contains the most characteristic information of the drug blister package at one time.

Continuing to refer to Figure 1, in order to build a drug database, in steps 108 and 110, the combined images are used to train the machine learning algorithm embedded in the computer to generate reference images. The combined images containing drug information are classified based on the reference images stored in the drug database, and can be read later to identify candidate packages. In some embodiments, at least one combined image of multiple drugs can be input into a machine learning algorithm. In an exemplary embodiment, more than 10 to 20,000 combined images can be combined, such as 10, 100, 200, 300, 400, 500, 1000, 1,100, 1,200, 1,300, 1,400, 1,500, 2,000, 2,500, 3,000, 3,500, 4,000, 4,500, 5,000, 5,500, 10,000, 11,000, 12,000, 13,000, 14,000, 15,000, 16,000, 17,000, 18,000, 19,000, and 20,000 combined images are input to machine learning algorithms to build a drug database. Each image can be used to train the machine learning system to convert the image information into reference information, which can then be stored in the device and/or the built-in database of the system.

It should be noted that the machine learning program system suitable for the present disclosure can be any conventional visual object detection model in the technical field, which is optimized or not optimized under certain criteria according to actual needs. These models include but are not limited to : Formable parts model (DPM), region convolutional neural network (R-CNN), fast region convolutional neural network (Fast R-CNN), high-speed regional convolutional neural network Road (Faster R-CNN), masked area convolutional neural network (mask R-CNN) and YOLO. Preferably, the visual object detection model suitable for training the learning step of the present disclosure is optimized, and at least the parameters (such as the number of input image pixels, bounding boxes and anchor boxes) are optimized. And size) for optimization. According to the present disclosure, the merged image processed by the foregoing steps and subsequently input to the learning system should be a "full bleed image" that meets a predetermined size (for example, a fixed pixel). Therefore, the number and size of anchor frames can be minimized (for example, to only one anchor frame) to improve calculation speed and performance. In other words, through the above-mentioned steps of obtaining the front-view images and arranging the two front-view images into one, the machine learning system can also run smoothly and quickly when processing large amounts of data on large-scale drug packaging. What's more, the overall processing time can be shortened, which greatly improves the efficiency of establishing a drug database.

In addition, the subject matter described herein can be implemented using a non-transitory, tangible processor-readable storage medium storing processor-readable instructions. When executed by a processor of a programmable device, the instructions can control the programmable device to execute the method according to the embodiment of the present disclosure. Exemplary processor-readable storage media suitable for implementing the subject matter described herein may include (but are not limited to) RAM, ROM, EPROM, EEPROM, flash memory, or other stale memory technologies: CD-ROM, DVD, or Other optical storage, magnetic card type, magnetic disk storage or other magnetic storage devices, and other media that can be used to store the required information and that can be read by the processor. In addition, the processor-readable storage medium that implements the subject of the present invention may be located on a single device or computing platform, or may also be distributed on multiple devices or computing platforms. In some embodiments, the computing platform is an embedded system with real-time computing constraints.

The processed and eye-catching images of the present disclosure, and by arranging the two processed images into a "merged image", actually increase the calculation performance and accuracy. Based on the above technical features, the purpose of this disclosure is to establish a drug database based on drug blister packaging images, and use the drug database as a benchmark for drug identification, which can improve the accuracy of drug identification and eliminate human error during the dispensing process. , To improve the safety of drug use and the quality of patient care.

2. Drug Management System

Another aspect of the subject of the present invention is to provide a drug management system. Referring to FIG. 2 of the medication management system 200, the medication management system 200 includes an image capture device 210, an image processor 220, a drug database 230, and a machine learning processor 240, wherein the image capture device 210 The machine learning processor 240 and the machine learning processor 240 are respectively coupled to the image processor 220, and the medicine database 230 is connected to the machine learning processor 240 in a communication mode or physically.

The image capturing device 210 is configured to capture one or more images of a blister pack of a medicine. In some embodiments, the structure of the image capturing device 210 includes a transparent board 211 and a first image capturing unit 213 and a second image capturing unit 215 respectively disposed on both sides of the transparent board 211. The transparent plate 211 may be made of glass or acrylic polymer. Specifically, the first image capturing unit 213 is disposed on the upper side of the transparent plate 211 (as shown in FIG. 2), and captures a first image of a drug at a first angle θ, wherein the first image includes a drug blister package The second image capturing unit 215 is arranged on the lower side of the transparent plate 211 to capture a second image of the drug at a second angle, and the second image includes the second image of the drug blister package. Face (for example, the reverse). In a specific embodiment, the first angle and the second angle are 40 to 90 degrees of the first image capturing unit 213 and the second image capturing unit 215 relative to the horizontal plane of the transparent plate 211, for example: 40, 45 , 50, 55, 60, 65, 70, 75, 80, 85 or 90 degrees. In practice, the medicine is placed on the transparent plate 211, and the first image capturing unit 213 captures images of the medicine at a depression angle of 45 to 60 degrees, preferably 50 degrees. On the other hand, the second image capturing unit 215 performs image capturing of the medicine from below the transparent plate body 211 at an elevation angle of 90 degrees (as shown in FIG. 2). Preferably, the first image capturing unit 213 and the second image capturing unit 215 capture the front and back images of the drug blister package at the same time. For example, the first image capturing unit 213 and the second image capturing unit 215 are real-time digital cameras.

Unless otherwise specified, according to the present disclosure, the image processor 220 and the machine learning processor 240 each include a memory for storing a plurality of instructions, which can enable the processor to implement the method of the present invention, including the aforementioned configuration The method of drug database 230 and the method of identifying drugs in this aspect. In some embodiments, the image processor 220 and the machine learning processor 240 are set as two independent devices; or the two may also be set in the same hardware. In some embodiments, the image processor 220 and the machine learning processor 240 are communicably connected to each other. Specifically, the image processor 220 is communicatively connected with the image capture device 210 to receive images captured by the image capture device 210, and is configured to execute the image processing steps of the first computer implementation method of the present invention ( That is, the aforementioned steps 104 and 106) are used to generate merged images for building the drug database 230 and/or candidate images for subsequent identification. The drug database 230 established by the aforementioned method 100 of the present invention can be stored in a storage device connected to the machine learning processor 240 through a cable connection or a wireless network to provide a reference image, or optionally, the drug database 230 It is stored in the machine learning processor 240. In the exemplary embodiment described in FIG. 2, the drug database 230 of the present invention is connected to the machine learning processor 240 in a communication mode. The machine learning processor 240 can also be communicatively connected with the image processor 220 and configured to implement the image comparison of the second computer-implemented method of the present invention for drug identification. The method includes comparing the candidate image with the reference image in the drug database 230 established by the method 100 of the present invention.

In some embodiments, the medication management system 200 further includes a user interface (not shown) for outputting medication identification results, accepting instructions from external users, and feeding user input to the image processor 220 and Machine learning processor 240.

Various technologies can be used to implement the communication between the image capture device 210, the image processor 220, the medicine database 230, and the image learning processor 240. For example, the medication management system 200 of the present invention may include a network interface to allow the image capturing device 210, the image processor 220, the medication database 230, and the machine learning processor 240 to communicate through a network (such as a local area communication network). Communication via LAN, a wide area network (WAN), network or wireless network). In other embodiments, the system may have a system bus, which couples various system components (including the image capture device 210) to the image processor 220.

A number of embodiments are presented below to illustrate certain aspects of the present invention, so as to facilitate those skilled in the art to which the present invention belongs to implement the present invention. These experimental examples should not be regarded as limiting the scope of the present invention. Without further explanation, it is believed that those with ordinary knowledge in the technical field can use the present invention to the fullest extent based on the description herein. All publications cited herein are incorporated herein by reference in their entirety.

Example 1: Construction of a drug database

Strategies for image correction of the initial shot image

Collect more than 250 kinds of drugs currently on the market from the hospital pharmacy of Mackay Memorial Hospital (Taipei, Taiwan). In order to build a database of the drug database, as much as possible to obtain photos of all drug blister packaging. When performing image capture, a drug is randomly selected and placed on a special cabinet equipped with a transparent plate made of glass, and the specific location of the drug is not limited. Set up a light source around the transparent board to facilitate lighting, and install two BRIO webcams (Logitech, USA) on both sides of the transparent board (keep a distance from the board) to ensure the field of vision of the webcam Cover the entire board area. Among them, the web camera located above has a depression angle of 50 degrees with respect to the plane of the transparent plate; the web camera located below shoots vertically upwards. In the case where the captured image is not a front-view image, the concept of image correction is used to correct the captured image.

，其斜向水平的玻璃板體，所拍攝的第一影像為

，通常包含藥物泡型包裝的正面；另假設一虛擬垂直正上方攝影機

，假定該虛擬攝影機

所拍攝的虛擬影像為

，應呈現與第一影像

相近視野的正視視角。下方攝影機

，由

所拍攝的第二影像為

，通常是藥物泡型包裝的反面。第一影像

及第二影像

經影像糾正之後，分別以

與

表示。 Figure 3 exemplarily shows how to correct the first image and the second image of the drug blister package into the corrected first image and the corrected second image, respectively. As shown in Figure 3, first define the camera taken above the transparent plate as

, Its obliquely horizontal glass plate, the first image taken is

, Usually contains the front of the drug blister package; also assume a virtual vertical camera directly above

, Assuming that the virtual camera

The virtual image taken is

, Should be presented with the first image

The frontal angle of view of the similar field of view. Camera below

,by

The second image taken is

, Usually the opposite of drug blister packaging. First image

And the second image

After image correction,

versus

Said.

與第一影像

之間尋找一第一單應性矩陣

，具體方法是根據玻璃透明板體的實際長寬比例，在虛擬影像

及第一影像

中分別定義出可互相對應的各四個角點(例如第3圖，第一影像

中的四個圓點)。具體可以定義位於虛擬影像

正中間一玻璃的四個角點，同時第一影像

也具有該矩形的對應四個角點，進而求得

。求得虛擬影像

與第一影像

之間的第一單應性矩陣

之後，將第一影像

與第一單應性矩陣

相乘則可得到糾正後第一影像

。 First in the virtual image

With the first image

Find a first homography matrix

, The specific method is based on the actual aspect ratio of the transparent glass plate, in the virtual image

And the first image

Define the four corner points that can correspond to each other (for example, Figure 3, the first image

Four dots in). It can be defined in the virtual image

The four corners of the glass in the middle, and the first image

It also has the corresponding four corners of the rectangle, and then obtains

. Get a virtual image

With the first image

The first homography matrix between

After that, the first image

With the first homography matrix

Multiply to get the first image after correction

.

之後，還可以利用在虛擬影像

正中間呈現的玻璃四個角點(其與糾正後第一影像

中玻璃四個角點同)與第二影像

之間的對應關係，求得

，並將

乘上

後得到糾正後第二影像

。由於糾正後第二影像

追本溯源仍是從上方正視虛擬影像

得來，因此可知道糾正後第二影像

中，藥物泡型包裝反面影像出現的相對位置與糾正後第一影像

中藥物泡型包裝正面影像的相對位置相同(如第3圖所示)。 Then, get the first image after correction

Later, you can also use the virtual image

The four corners of the glass appearing in the middle (the same as the first image after correction)

The four corners of the middle glass are the same) and the second image

Correspondence between, find

And will

ride on

After being corrected, the second image

. Since the second image after correction

Tracing back to the source is still looking at the virtual image from above

So we know that the second image after correction

In, the relative position of the reverse image of the drug blister package and the first image after correction

The relative position of the front image of the traditional Chinese medicine blister package is the same (as shown in Figure 3).

Prepare and correct double-sided images (Rectified Two-sided Images , RTIs)

及糾正後第二影像

後，利用編程在影像處理器中的開發函式庫(Open Source Computer Vision 2，OpenCV 2)函式，對各影像進行裁剪以最小化背景雜訊並對其進行處理，以產生去背景之藥物泡型包裝正面輪廓影像

及去背景之藥物泡型包裝反面輪廓影像

。除了從糾正後第二影像

進行提取去背來獲得藥物泡型包裝反面輪廓影像

之外，由於糾正後第一影像

可視為虛擬影像

，因此也可以透過對去背景之藥物泡型包裝正面輪廓影像

直接乘上

，來產生去背景之藥物泡型包裝反面輪廓影像

。接著將去背景之藥物泡型包裝正面輪廓影像

及去背景之藥物泡型包裝反面輪廓影像

經旋轉後，規整產生合併影像(本發明稱為『校正雙面併整影像(RTIs)』)。每個RTI可與一預定模板(以下稱為校正雙面併整模板(rectified two-sided template，RTT)適配並且包含一藥物泡型包裝的正面與反面。預定模板是指規整合併影像成448 × 224像素之尺寸。總共取得18,000的RTIs，並利用CNN模型進行後續深度學習處理程序。 The first image after being corrected

And corrected second image

Then, use the Open Source Computer Vision 2 (OpenCV 2) function programmed in the image processor to crop each image to minimize the background noise and process it to produce a background-removing medicine Front profile image of blister packaging

And the back profile image of the drug blister package with the background removed

. Except from the corrected second image

Perform extraction to get back contour image of drug blister package

In addition, because the first image after correction

Can be regarded as a virtual image

, So you can also image the front profile of the drug blister pack with the background removed

Multiply directly

, To produce the back profile image of the drug blister pack without background

. Next, the front profile image of the drug blister package with the background removed

And the back profile image of the drug blister package with the background removed

After being rotated, a combined image is generated in a regular manner (this invention is called "corrected double-sided combined image (RTIs)"). Each RTI can be adapted to a predetermined template (hereinafter referred to as a rectified two-sided template (RTT)) and includes the front and back sides of a drug blister package. The predetermined template refers to the integration and image formation of 448 × 224 pixels in size. A total of 18,000 RTIs are obtained, and the CNN model is used for subsequent deep learning processing procedures.

Strategies for detecting irregular quadrilateral corners of drug blister packaging

及

進行旋轉，需偵測藥物泡型包裝輪廓的各角點。在待測藥物泡型包裝具有三直線邊緣及一曲線邊緣構成的不規則四邊形形狀之情況下，利用質心算法(表1)透過幾何學推理以測定曲線邊緣和未定義角點。 表1：用於偵測不規則四邊形形狀角點的質心演算法 輸入：三條邊緣線 L以及泡型包裝的輪廓形狀資訊 blisterContour輸出：透過直線交點以及幾何學推理找到的四個角點 P ₁ 、 P ₂ 、 P ₃ 及 P ₄ (以循環順序排列) 1: 程序 FINDCORNERS( L, blisterContour) 2:   P ₁, P ₄ ← 三條線 L的兩個交點。 3:   M ← 兩交點 P ₁ 、P ₄ 之間的中點。 4:   B ← 泡型包裝輪廓形狀的質心 blisterContour。 5:  

← 從 M至 B的位移向量。 6:    P ₂ ← P ₁ + 2

7:   P ₃ ← P ₄ + 2

As mentioned above, in order to image the drug blister packaging with the background removed

and

To rotate, it is necessary to detect the corner points of the outline of the drug blister package. When the drug blister package to be tested has an irregular quadrilateral shape composed of three straight edges and a curved edge, the centroid algorithm (Table 1) is used to determine the curved edges and undefined corners through geometric inference. Table 1: Centroid algorithm for detecting corners of irregular quadrilateral shapes Input: Three edge lines L and the contour shape information of the blister package blisterContour Output: Four corner points P ₁ , P ₂ , P ₃ and P ₄ (arranged in cyclic order) found through the intersection of straight lines and geometric inference 1: Program FINDCORNERS( L, blisterContour ) 2: P ₁ , P ₄ ← two intersection points of three lines L. 3: M ← the midpoint between two intersection points P ₁ and P _4. 4: B ← the center of mass blisterContour of the contour shape of the blister package. 5:

← The displacement vector from M to B. 6: P ₂ ← P ₁ + 2

7: P ₃ ← P ₄ + 2

Figure 4 exemplarily shows how to perform corner point recognition on drug packages with irregular quadrilaterals. As depicted in Figure 4, a blister package with three straight edges (L ₁ , L ₂ , L ₃ ) and a curved edge (C ₁ ) presents a random direction, where the intersection of L ₁ and L ₂ is designated as P ₁ , and the intersection of L ₂ and L ₃ is designated as P ₄ . The goal is to determine the coordinates of P ₂ and P ₃ , based on the four points ( P ₁ , P ₂ , P ₃ and P ₄ ) and the area surrounded by the four edges ( L ₁ , L ₂ , L ₃ and C ₁₎ Will cover the image of the entire package. First, determine the midpoint M between P ₁ and P ₄ on the edge L ₂ , and then use the algorithm cv2.moments (OpenCV 2) to determine the center of mass B of the drug blister packaging area. The displacement vector v is calculated based on the coordinates of the midpoint M and the center of mass B. Finally, it can be determined that the positions twice the displacement vector from the intersection points P ₁ and P ₄ are the coordinates of P ₂ and P _3. Through the foregoing process, the four intersection points or corner points P ₁ , P ₂ , P ₃ and P ₄ can be automatically sorted in a clockwise or counterclockwise direction.

Example 2: Evaluate the classification efficiency of a medication management system that implements machine learning based on the medication database of Example 1

Process and merge images

In order to evaluate the visual recognition performance of the RTIs of the present disclosure, the original images of the drugs (unprocessed and unmerged) and RTIs (processed and merged) are respectively input to the training model (optimized Tiny YOLO) for deep learning. Table 2 summarizes the training results. According to the data in Table 2, compared to the unprocessed original image, the “brighter” image of the present disclosure has high performance in training visual recognition. The higher the training F1-score, the better the visual recognition effect of the deep learning network on packaging images. Compared with the unprocessed images, the RTIs processed into RTT also significantly increase the training effect. Table 2: Comparison of unprocessed images and RTIs Experimental group Group I Group II Group III Image type Unprocessed image (front side) Unprocessed image (reverse side) Processed images (RTIs) as in Example 1 Training time (minutes) 362 462 twenty three Training type 65 83 5 Precision (%) 77.03 89.39 - recall 67.44 87.68 - F1-score 65.39 86.48 99.82

Real-time drug identification application

During operation, a drug is randomly selected and placed on a transparent plate equipped with glass. Set up a light source around the transparent board to facilitate lighting, and install two BRIO webcams (Logitech, USA) on both sides of the transparent board (keep a distance from the board) to ensure the field of vision of the webcam Cover the entire board area. On the other hand, the drug database built in Example 1 is first stored in the real-time embedded computing device JETSON ^TM TX2 (NVIDIA, USA). JETSON ^TM TX2, called the "developer kit", contains memory, CPU, GPU, USB port, mesh antenna and other computing components, so that image processing steps and machine learning steps can be performed on the same device Executed within. In operation, two webcams can be connected to JETSON ^TM TX2 via a USB cable. According to this, the front and back images of the selected drug can be simultaneously transmitted to the processor in real time. The two original images of the selected drug blister package are processed into an RTI, and then the RTI accepts the learning model Tiny YOLO ^{programmed in JETSON TM TX2 for subsequent procedures.} The visual recognition speed of Tiny YOLO model can reach about 200 frames per second (FPS). The total process (ie, from capturing images to visual recognition) takes about 6.23 frames per second. The recognition result can be displayed on an external display device in real time, such as a computer screen or a mobile phone interface. Therefore, the external user can almost get the identification result of the medicine after putting the medicine on the glass plate body at the same time.

Furthermore, in order to evaluate whether the system of the present invention can effectively reduce the probability of human error in the dispensing process, another comparison is made again. For example, an anti-anxiety drug: Lorazepam, whose appearance is quite similar to other drugs, so it is the most misidentified drug during the dispensing process. First, the RTI of lenapine is entered into the system of the present invention to compare with all drugs in the database. After calculation, the learning model proposes several drug candidates that may be misidentified based on the similarity of appearance after the analysis of the learning model. The candidate form can assist clinical staff in reviewing whether the selected drug is correct. Therefore, with the drug management system of the present disclosure, the accuracy of dispensing drugs can be improved, and human error can be minimized to ensure patient safety.

In short, the method and system for drug management of the present invention can achieve the aforementioned objectives by performing image processing steps in a semi-open mechanism to merge images into one, and executing the deep learning model in real-time. The advantage of this disclosure is not only that regardless of the orientation of the object (i.e., the drug) and the angle of the initial camera, the original image can be quickly corrected and organized into a consolidated image with a fixed size, which is not only beneficial to the machine The efficiency of learning can also increase the accuracy of drug classification through the appearance of the drug.

It should be understood that the foregoing description of the embodiments is only given in the form of examples, and various modifications can be made by those with ordinary knowledge in the technical field of the art. The above specification, examples and experimental results provide a complete description of the structure and use of the exemplary embodiments of the present invention. Although various specific embodiments of the present invention are disclosed in the above embodiments, they are not intended to limit the present invention. Those with ordinary knowledge in the technical field to which the present invention belongs, without departing from the principle and spirit of the present invention, Various changes and modifications can be made to it, so the protection scope of the present invention shall be subject to the scope of the accompanying patent application.

100 Method 102–110 Steps 200 Drug management system 210 Image capture device 211 Transparent board 213 The first image capture unit 215 Second image capture unit 220 Image processor 230 Drug database 240 Machine learning processor

In order to make the above and other objects, features, advantages and embodiments of the present invention more comprehensible, the description of the accompanying drawings is as follows.

FIG. 1 is a flowchart of a method 100 according to an embodiment of the present disclosure.

FIG. 2 shows a medication management system 200 according to an embodiment of the present disclosure.

Figure 3 shows an example of the present disclosure to illustrate how to correct the image through the homography matrix.

Figure 4 shows an example of the present disclosure to illustrate how to define the corner points of the package through the centroid algorithm.

According to the usual operation method, the various elements and features in the figure are not drawn to scale, and the drawing method is to present the specific features and elements related to the present invention in the best way. In addition, between different drawings, the same or similar element symbols are used to refer to similar elements/components.

100 Method 102–110 Steps

Claims (10)

A computer-implemented method for establishing a drug database of a drug blister package image includes: (a) obtaining a first image and a second image of the drug blister package, respectively, wherein the first image and the first image The two images respectively include a first side and a second side of the drug blister package; (b) performing image rectification on the first image and the second image respectively to generate a corrected first image And a corrected second image; (c) juxtapose the corrected first image and the corrected second image to generate a merged image; (d) train the merged image with a machine learning algorithm, To generate a reference image; and (e) use the reference image to establish the drug database, wherein step (c) includes: (c-1) extracting the corrected first image and the corrected second image, The contours of the first side and the second side of the drug blister package to generate a first contour image and a second contour image respectively; (c-2) Identify each contour in the first contour image and the second contour image (C-3) According to the coordinates of step (c-2), rotate the first contour image and the second contour image of step (c-1) to determine the coordinates of the corner point; Forming a first processed image and a second processed image; and (c-4) the first processed image and the second processed image of the normalizing step (c-3) to generate the combined image. The computer-implemented method according to claim 1, wherein in step (b), image correction is performed on the first image by the following steps: (b-1) providing a reference rectangle; (b-2) selecting the first image Four corner points in an image, where the four corner points constitute a quadrilateral; (b-3) based on the correspondence between the reference rectangle provided in step (b-1) and the quadrilateral selected in step (b-2), Calculating a first homography matrix of the corresponding relationship between the reference rectangle and the quadrilateral; and (b-4) performing perspective correction on the first image according to the first homography matrix, To obtain the corrected first image. The computer-implemented method according to claim 2, wherein the quadrilateral in step (b-2) corresponds to an outline of the first side of the drug blister package. The computer-implemented method according to claim 1 or 2, wherein in step (b), image correction is performed on the second image by the following steps: (bi) based on the difference between the corrected first image and the second image Calculating a second homography matrix between the two; and (b-ii) performing perspective correction on the second image according to the second homography matrix to obtain the corrected second image. The computer-implemented method according to claim 1, wherein step (c-2) is performed by a linear conversion algorithm or a centroid algorithm. The computer-implemented method according to claim 1, wherein the first side and the second side are the front side and the back side of the drug blister package, respectively. A medicine management system, comprising: an image capturing device for capturing an image of a blister package of a medicine, comprising: a transparent plate for placing the medicine on it; and a first image capturing unit , Arranged on one side of the transparent plate body to capture a first image of the medicine at a first angle, wherein the first image includes a first surface of the medicine blister package; and a second image The capturing unit is arranged on the other side of the transparent plate to capture a second image of the drug at a second angle, wherein the second image includes a second side of the drug blister package, a The image processor is programmed to execute a first computer-implemented method through instructions to generate a candidate image, wherein the first computer-implemented method includes: (a) performing image correction on the first image and the second image respectively, To respectively generate a corrected first image and a corrected second image; and (b) normalize the corrected first image and the corrected second image to generate the candidate image, as described in claim 1 The drug database established by the computer-implemented method is used to provide a reference image; and A machine learning processor, programmed with instructions to execute a second computer-implemented method, is used to compare the candidate image with the reference image of the drug database created by the computer-implemented method according to claim 1. The system according to claim 7, wherein step (b) comprises: (b-1) extracting the first side and the second side of the drug blister package from the corrected first image and the corrected second image The contours of the two sides to respectively generate a first contour image and a second contour image; (b-2) Identify the corner points of each contour in the first contour image and the second contour image to determine the corner point Coordinates; (b-3) According to the coordinates of step (b-2), rotate the first contour image and the second contour image of step (b-1) to form a first processed image and a second Processing the image; and (b-4) the first processed image and the second processed image of the normalizing step (b-3) to generate the candidate image. The system according to claim 7, wherein the first side and the second side are the front side and the back side of the drug blister package, respectively. The system according to claim 7, wherein the first angle and the second angle are respectively 40 to 90 degrees with respect to the horizontal plane of the transparent plate.

Images