TWI695347B - Method and system for sorting and identifying medication via its label and/or package - Google Patents

Abstract

Disclosed herein is an improved pharmaceutical management system and methods implemented by the system for sorting and identifying a medicine via its label and/or package. The method comprises steps of: (a) receiving a plurality of raw images of a package of a medication; (b) juxtaposing two of the plurality of raw images to produce a combined image, in which the two raw images are different from each other; (c) processing the combined image to produce a reference image; and (d) establishing the medication library with the aid of the reference image. The system comprises an image capturing device, an image processor, and a machine learning processor. The image processor is programmed with instructions to execute the method for producing a combined image.

Description

Method and system for classifying and identifying medicine through medicine packaging and/or labeling

This disclosure relates to the field of medicine management systems, and in particular, to a method and system for classifying and identifying medicines through labeling and/or packaging of medicines.

Efficient storage and management systems and/or methods of medical supplies and pharmaceutical products are important prerequisites for the smooth operation of hospital systems and the provision of quality patient care. In this case, the accuracy of prescription dispensing is critical to all medical institutions. Although fully automated dispensing cabinets (ADCs) have been introduced into medical institutions for 20 years, there are still medical negligence. Since the administration of fully automated dispensing cabinets still relies heavily on the precise adherence of standard procedures by clinical staff, it is unlikely that the final human error will be completely ruled out. For example, a pharmacy may store the wrong medicine in a given medicine cabinet, or a clinician may get a "look-alike" medicine from an adjacent medicine cabinet. Therefore, it is quite unreliable to identify the drug through the appearance of the drug with the human eye, let alone apply it to the drug management system.

Deep learning is part of a broader series of machine learning methods based on multi-level learning data representation. These methods are obtained by combining simple but non-linear modules, each of which will be a level The representation is converted to a higher, more abstract level. With sufficient combinations of these transformations, more complex functions (such as classification tasks) can be learned. Therefore, deep learning has made significant progress in solving the problems that have existed in the artificial intelligence community for many years, and can be applied to many technical fields. For the pharmaceutical management system, the excellent performance of deep learning in identifying the appearance of objects should bring solutions to the shortcomings of current drug distribution technology. However, given the wide variety of pharmaceutical packages involving similar appearances, performing deep learning alone cannot produce the desired results.

In view of the foregoing, it is necessary for the prior art to provide an improved method and system for drug management (eg, classification and identification of drugs via labeling and/or packaging of drugs).

In order to provide the reader with a basic understanding, the following provides a brief summary of the disclosure. This summary of the invention is not an extensive overview of the disclosure, nor is it intended to identify key/essential elements of the invention or outline the scope of the invention. Its sole purpose is to present some concepts of the disclosure in a simplified conceptual form as a prelude to the more detailed description that is presented later.

As embodied and broadly described herein, the purpose of this disclosure is to provide an improved drug management system and implementation method for identifying clinical drugs through the system, thereby greatly improving the efficiency and accuracy of dispensing.

An aspect of this disclosure relates to a computer-implemented method for creating a drug database. In some embodiments, the method includes: (a) receiving a plurality of original images of a drug package; (b) juxtapose two of the plurality of original images to generate a merged image, wherein the two The original images are different from each other; (c) processing the merged image to generate a reference image; and (d) using the reference image to create the drug database.

In some non-essential embodiments, the aforementioned method further includes simultaneously capturing the plurality of original images of the drug package before step (a).

According to some embodiments of the present disclosure, the drug is in the form of a blister package.

According to some embodiments of the present disclosure, step (b) includes: (b-1) processing the plurality of original images separately to generate a plurality of first processed images each having a defined contour; (b-2) identifying Corners of each defined contour of the plurality of first processed images to determine their coordinates; (b-3) based on the coordinates determined in step (b-2), rotate the first of step (b-1) Processing images to generate a plurality of second processed images; and (b-4) combining any two of the plurality of second processed images to generate the combined image of step (b).

According to some embodiments of the present disclosure, each of the plurality of original images in step (b-1) is subjected to the following processes: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) An edge recognition process, (iv) a convex hull operation process, and (v) a contour finding process.

According to some embodiments of the present disclosure, the aforementioned processes (i) to (v) may be independently performed in any order; and in the aforementioned processes (i) to (v), the steps (b-1) may be used The original image, or an image processed by any one of (i) to (v) other than the current processing.

In some embodiments of the present disclosure, step (b-2) is performed with a linear transformation algorithm or a centroid algorithm.

It is preferable that the aforementioned merged image has a double-sided image of the medicine bubble packaging.

According to some embodiments of the present disclosure, a machine learning algorithm is used to perform step (c).

Another aspect of the present disclosure relates to a computer-implemented method for identifying a drug via a drug bubble package. The method includes: (a) obtaining both the front and back images of the blister package of the drug; (b) regularizing the front and back images of step (a) to generate a candidate image; (c) comparing the candidate image with A reference image of a drug database created by the foregoing method; and (d) outputting the result of step (c).

According to some embodiments of the present disclosure, step (b) includes: (b-1) processing the front and back images separately to generate two first processed images each having a defined contour; (b-2) identifying The corners of the defined contours of the two first processed images to determine their coordinates; (b-3) based on the coordinates determined in step (b-2), rotate the two of step (b-1) The first processed image to generate two second processed images; and (b-4) combining the two second processed images to generate the combined image of step (b).

According to some embodiments of the present disclosure, the front and back images of step (b-1) are subjected to the following processes: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) a Edge recognition processing, (iv) a convex hull operation processing, and (v) a contour finding process.

In some embodiments, the aforementioned (i) to (v) processes can be independently performed in any order; and in the aforementioned (i) to (v) processes, the front and back images of step (b-1) can be used Or use the image processed by any one of (i) to (v) other than the current processing.

In some non-essential embodiments, step (b-2) is performed with a linear transformation algorithm or a centroid algorithm.

In some non-essential embodiments, a machine learning algorithm is used to perform step (d).

In some non-essential embodiments, the method further includes transmitting the candidate image to the drug database before step (c).

Yet another aspect of the present disclosure relates to a medicine management system, which includes an image capture device, an image processor, and a machine learning processor, to implement the aforementioned method.

Specifically, the image capturing device is configured to capture multiple images of a medicine package. The image processor is programmed with instructions to execute a method for generating a candidate image, the method includes: (1) processing the plurality of images of the drug package to generate a plurality of first images each having a defined contour Process the image; (2) Identify the corner points of the defined contours of the plurality of first processed images to determine their coordinates; (3) Rotate each of the steps of step (1) based on the coordinates determined in step (2) A processed image to generate a plurality of second processed images; and (4) normalizing two of the plurality of second processed images in step (3) to generate the candidate image, wherein the two second processed images are mutually Different. Furthermore, the machine learning processor executes a method through instruction programming, which is used to compare the candidate image with a reference image of the aforementioned drug database. The results generated by the machine learning processor can then be output to notify the operator who is dispensing.

In some embodiments of the present disclosure, the image capturing device includes a transparent plate, the medicine is placed on the transparent plate, and the two image capturing units are individually disposed above each side of the transparent plate.

In some embodiments of the present disclosure, each of the plurality of images in step (1) is subjected to the following processes: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) an edge Recognition processing, (iv) a convex hull arithmetic processing, and (v) a contour finding processing.

In some embodiments, the aforementioned (i) to (v) processes can be performed independently in any order; and in the aforementioned (i) to (v) processes, the plurality of images of step (1) or It is an image that has been processed by any of (i) to (v) other than the current processing.

Additionally or alternatively, the step (2) may be performed by a linear transformation algorithm or a centroid algorithm.

In some non-essential embodiments, the method for comparing the candidate image with the reference image of the aforementioned drug database is performed by a machine learning algorithm.

Through the above configuration, the medicine management method and system for classification and identification can be carried out in real-time and real-time, so that in the dispensing process, the processing time of the overall image recognition can be shortened regardless of the orientation of the medicine.

In addition, the accuracy of identifying the medicine through the appearance of the medicine and/or the bubble packaging can be improved, and the human error in the dispensing process can be reduced. Therefore, the safety of drug use can be improved.

After referring to the embodiments below, those with ordinary knowledge in the technical field to which the present invention belongs can easily understand the basic spirit of the present invention and other inventive objectives, as well as the technical means and implementation aspects adopted by the present invention.

In order to make the description of this disclosure more detailed and complete, the following provides an illustrative description of the implementation form and specific embodiments of the present invention; however, this is not the only form for implementing or using specific embodiments of the present invention. The embodiments cover the features of multiple specific embodiments, as well as the method steps and their sequence for constructing and operating these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and sequence of steps.

I. Definition

For ease of explanation, the specific terms described in the specification, examples, and appended patent applications are described here in a comprehensive manner. Unless otherwise defined in this specification, the meanings of scientific and technical terms used herein have the same meanings as those understood and used by those with ordinary knowledge in the technical field to which the present invention belongs. In addition, without conflicting with the context, the singular noun used in this specification covers the plural form of the noun; and the plural noun used also covers the singular form of the noun. Specifically, unless the context clearly indicates otherwise, the singular forms "a" (a and an) used in this and the appended patent applications include plural forms. In addition, in this specification and the scope of patent application, the expressions of "at least one" (at least one) and "one or more" (one or more) have the same meaning, and both represent one, two, three Or more.

Although the numerical ranges and parameters used to define the broader range of the present invention are approximate values, the relevant numerical values in the specific embodiments have been presented as accurately as possible. However, any numerical value inevitably contains standard deviations due to individual test methods. Here, "about" generally means that the actual value is within plus or minus 10%, 5%, 1%, or 0.5% of a specific value or range. Or, the term "about" means that the actual value falls within the acceptable standard error of the average value, depending on the consideration of those with ordinary knowledge in the technical field to which the present invention belongs. Except for experimental examples, or unless clearly stated otherwise, all ranges, quantities, values, and percentages used herein can be understood (for example, to describe the amount of materials, length of time, temperature, operating conditions, quantity ratio, and other similarities All) have been modified by "about". Therefore, unless stated to the contrary, the numerical parameters disclosed in this specification and the accompanying patent application are approximate values, and can be changed as required. At least these numerical parameters should be understood as the indicated significant digits and the values obtained by applying the general rounding method. Here, the numerical range is expressed from one end point to another segment point or between two end points; unless otherwise stated, the numerical range described herein includes end points.

As used herein, the term "blister pack" (blister pack or blister package) covers any type of layered packaging in which the product is contained between sheet materials; where these sheet materials can The method is well known to those skilled in the art. For example, these sheets can be bonded by heat and/or pressure activated glue. These are commercially available as sheet materials for individual sheets (for manual packaging) or as continuous sheets on a roll (for machine packaging). The main structure of the blister package is a cavity or bag made of a formable mesh (usually thermoplastic). The cavity or bag is large enough to contain the items in the blister pack. Depending on the application, the blister pack may have a thermoplastic backing. For the pharmaceutical field, blister packaging is often used as a single-dose packaging for tablets, and contains the drug information printed on the back of the blister packaging. In addition, these sheets are available in various thicknesses.

II. Embodiments of the invention

The most important thing for patient care is the safety of drug use. Correctly filling out prescription drugs and dispensing medicines is essential for patient care. Since there is still room for improvement in conventional automatic dispensing systems (such as ADCs), the present invention aims to provide an improved inductive deep learning method to solve the aforementioned problems. In addition, the present invention also aims to develop an automatic medication verification (AMV) device that uses a real-time operating system to reduce the workload of clinical staff.

Specifically, the so-called induced deep learning refers to a method of processing or highlighting features or images before inputting the features or images into the algorithm, thereby learning the feature information in a more emphasized manner. In order to better divide the identified targets and expose the inherent descriptive features, the feature processing is generally achieved through various image processing steps that integrate the two-sided cropped images of the appearance of the blister package into a fixed-size template , So as to facilitate the subsequent learning network classification process.

1. Method of establishing drug database

With reference to FIG. 1, the first aspect of the present disclosure relates to a method for establishing a drug database in a computer-readable storage medium.

As shown in FIG. 1, it illustrates a flowchart of a computer-implemented method 100 according to an embodiment of the present disclosure. The method includes at least the following steps: (S110) receiving a plurality of original images of a medicine package; (S120) arranging two of the plurality of original images to generate a combined image, wherein the two original images are different from each other (S130) processing the aforementioned merged image to generate a reference image; and (S140) using the reference image to create the drug database.

Before the method 100 is implemented, a plurality of original images of a pharmaceutical package (such as a bubble package) can be captured by any known method (such as an image capturing device, such as a video camera). Next, in step S110 of the method of the present invention, the captured image is automatically forwarded to the device and/or system in which the instruction for executing the method 100 of the present invention is embedded. In the following step S120, two of the plurality of original images received through the aforementioned device and/or system may be regularized with each other to generate a combined image. It is worth noting that the two original original images are different from each other. In an exemplary embodiment, the medicine is in the form of a blister package; therefore, the plurality of original images obtained by the image capturing device will include both sides of the blister package. That is, the two images showing the front and back sides of the blister package respectively will be regularized to produce a combined image. In order to improve the accuracy of subsequent machine learning programs in identifying the appearance, the original image is processed and the original image is highlighted to give a clean image with predetermined characteristics (for example, fixed at a predetermined pixel size, etc.). In a clean image, the background part of the non-body (ie, the image of the blister pack and/or label) is all removed.

In step S130, the aforementioned merged image is used to train a machine learning algorithm embedded in a computer (such as a processor) to generate a reference image. Steps S110 to S130 can be repeated multiple times, and the medicine package used each time is different from that used previously. Finally, a drug database can be created with reference to the reference image (step S140).

Returning to step S120, it usually includes the following steps: (S121) processing a plurality of original images separately to generate a plurality of first processed images each having a defined contour; (S122) identifying each of the defined first processed images The corner points of the contour to determine its coordinates; (S123) based on the coordinates determined in step S122, rotate each of the first processed images in step S121 to generate a plurality of second processed images; and (S124) combine the plurality of Any two of the second processed images to generate the merged image in step S120.

Step S121 is to use an image processor to process the plurality of original images respectively, and finally generate a plurality of first processed images each having a clearly defined contour. The image processor executes several algorithms to remove background noise from the image and extract target features. Specifically, each of the plurality of original images in step S121 is subjected to the following processes: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) an edge recognition process, (iv) a Convex hull arithmetic processing and (v) contour finding processing. It should be noted that the aforementioned processes (i) to (v) can be performed independently in any order.

Specifically, (i) use a color conversion algorithm to perform gray-scale conversion processing to convert BGR color (color) to gray-scale (S1211); (ii) use a filter algorithm to perform noise filtering processing to convert Minimize background noise (S1212); (iii) use the edge recognition algorithm to perform edge recognition processing to determine the coordinates of each edge of the bubble package in the image (S1213); (iv) use the convex hull algorithm to perform convexity Packet arithmetic processing to calculate the actual area of the medicine bubble packaging in the image (S1214); and (v) Perform the contour finding process using a contour definition algorithm to extract and highlight the main area of the bubble packaging (S1215).

According to some embodiments of the present disclosure, the original image in step S121 is subjected to all the aforementioned processes (i) to (v) before proceeding to step S122. As described above, these processes can be performed independently in any order, so in addition to the original image formed in step S121, the image after receiving any process may be the original image of the next process. For example, the original image in step S121 is first subjected to (i) processing or step S1211. In this step, the BGR color of the original image is converted into grayscale to generate a grayscale image. Then, by any processing from (ii) to (v), the gray-scale image from (i) processing is processed until all the processing from (i) to (v) is successfully applied to the image, thereby generating The first processed image of the defined contour.

Or, or unnecessarily, the original image in step S121 is first subjected to (iv) processing or step S1214. In this step, a convex hull algorithm is executed. Convex hull arithmetic is used to find the convex hull of a set of points in a plane or other low dimensions. In this disclosure, the convex hull is defined by calculating the actual area of the drug bubble packaging in the image. The image obtained from the (iv) process or step S1214 may then be subjected to any of (i), (ii), (iii) or (v) until all the processes from (i) to (v) are successfully applied at On this image, a first processed image with a defined contour is then generated.

It should be understood that the aforementioned processes (i) to (v) or steps S1211 to S1215 may be performed sequentially, randomly, and/or repeatedly (for example, the process (iii) may be repeated one or more times); preferably Is executed in the order of (i) to (v) or from steps S1211 to S1215.

By performing the aforementioned steps S1211 to S1215, a plurality of first processed images each having a defined contour are generated from the plurality of original images, wherein the background noise of the original image is eliminated, and the main part of the bubble packaging is extracted and eye-catching To facilitate subsequent image processing procedures.

Continue to step S122, which defines the corner point of each first processed image (Figure 1). Regardless of the outline shape of the bubble packaging, this step can determine the coordinates of at least one corner of each first processed image. Preferably, the coordinates of the four corner points of each first processed image acquired from step S121 are determined. The goal of this step is to predict at least three straight lines from the contour edge of the bubble package, and then use geometric reasoning to obtain the quadrilateral and the coordinates of the corner points that are closest to the edge of the bubble package. Since bubble packaging may have multiple shapes and/or contours, different algorithms may be used to identify corner points from the first processed image of the pharmaceutical packaging. For example, if the outline shape of the medicine package in the first processed image is a conventional quadrilateral (such as a rectangle), a straight line conversion algorithm can be used to identify the four corner points. On the other hand, if the outline shape of the medicine package in the first processed image is an unconventional quadrilateral, such as an atypical polygon with three straight edges and one curved edge, then the centroid algorithm is used for corner recognition. Through the aforementioned design, the coordinates of each corner point can be determined regardless of the direction and shape of the medicine package.

Recognizing the corners of each processed image in step S122 is not only to establish anchor points for the subsequent rotation step (S123) and merge step (S124), but also to ensure that the entire drug package is included in the first processed image for subsequent analysis. It is worth noting that the measured four corner points of the bubble packaging image must be arranged in a clockwise or counterclockwise manner, according to which the first processed image can be rotated to a predetermined position based on the determined coordinates of the four corner points (S123). The predetermined position may vary with actual implementation requirements. In some embodiments, the predetermined position refers to a position where the short side and the long side of the bubble packaging are parallel to the X-axis and Y-axis of the Cartesian coordinate system, respectively. The preferred orientation of the bubble packaging is to rotate the image only once to make the short and long sides parallel to the X-axis and Y-axis of the Cartesian coordinate system. In actual application, various perspective transformation algorithms can be used to rotate the first processed image, which is provided with a predetermined pixel size (for example, 448×224 pixels), so as to generate a second processed (rotated) image with a predetermined position.

Next, in step S124, any two second processed images of the medicine may be placed side by side (or aligned with each other) to generate a combined image. It should be noted that the merged images obtained from this step may contain various combinations, which is conducive to establishing the drug database data as much as possible. The two second processed images can be two opposite sides of the bubble packaging (whether upright or inverted). Preferably, the two second processed images in the same direction and respectively presenting both sides of the drug bubble package are regular to each other to generate a merged image containing the most characteristic information of the drug package.

In order to establish a drug database, the merged images are used to train the computer-embedded machine learning algorithm in steps S130 and S140 to generate reference images. The merged images containing drug information are classified with reference images stored in the drug database, and can then be extracted to identify candidate packages. In some embodiments, at least one merged image is input into the machine learning algorithm. In an exemplary embodiment, more than 10 to 20,000 merged images may be combined, such as 10, 100, 200, 300, 400, 500, 1000, 1,100, 1,200, 1,300, 1,400, 1,500, 2,000, 2,500, 3,000, 3,500, 4,000, 4,500, 5,000, 5,500, 10,000, 11,000, 12,000, 13,000, 14,000, 15,000, 16,000, 17,000, 18,000, 19,000 and 20,000 merged images are input to the machine learning algorithm to create a drug database. Each image can be used to train a machine learning system to convert the image information into reference information, which can then be stored in a database built into the device and/or system.

It should be noted that the machine learning program system applicable to the present disclosure may be any visual object detection model known in the art, and may or may not be optimized under certain criteria according to actual needs. These models include but are not limited to ：Formable parts model (DPM), regional convolutional neural network (R-CNN), fast regional convolutional neural network (Fast R-CNN), high-speed regional convolutional neural network Road (Faster R-CNN), mask area convolutional neural network (mask R-CNN) and YOLO. Preferably, the visual object detection model suitable for training the learning steps of the present disclosure is optimized, at least for the parameters (such as the number of input image pixels, bounding boxes and anchor boxes) And size) for optimization. According to the present disclosure, the merged image processed through the foregoing steps and subsequently input to the learning system should be a "full bleed image" that conforms to a predetermined size (for example, fixed pixels). Therefore, the number and size of the anchor frames can be minimized (for example, to only one anchor frame) to improve the calculation speed and performance. In other words, the machine learning system can run smoothly and quickly even when processing large amounts of data for large-scale pharmaceutical packaging through the above-mentioned steps of normalizing the two images into one. What's more, the overall processing time can be shortened, which greatly improves the efficiency of establishing a drug database.

By performing the aforementioned steps S130 and S140, when the drug database is implemented, the training result of each different drug can be output as a classification model.

2. Method of identifying drugs

With reference to FIG. 2, the second aspect of the present disclosure relates to a method for identifying the medicine via the medicine packaging (for example, bubble packaging).

Referring to FIG. 2, which illustrates a flowchart of the method 200 according to an embodiment of the present disclosure. The method 200 includes the following steps: (S210) Obtain the front and back images of the blister pack simultaneously; (S220) Regularize the front and back images of step S210 to generate a candidate image; (S230) Unnecessarily, send the Candidate images into the drug database; (S240) compare the candidate images with a reference image of the drug database created by the method described in the previous paragraph; and (S250) output the result of step S240.

Preferably, the method 200 of the present invention can be performed by a non-transitory processor-readable storage medium embedded with instructions for performing the steps of the present invention and a drug database. The drug database may be an originally stored or built-in drug database, or a drug database created by the aforementioned method 100.

In the image receiving step (S210), two original images of a medicine are simultaneously captured. The two original images are preferably on each side of the blister pack (ie, front and back images). The original images can be obtained by any known method, preferably through at least one image capture device (such as a camera) To get. In an exemplary embodiment, the two original images of the blister pack are simultaneously captured by two image capturing devices provided on both sides of the drug (eg, front and back of the drug); therefore, the two original images Covers information on drugs on both sides of blister packaging. In order to improve the accuracy of the subsequent machine learning program to recognize the appearance, the two original images obtained in step S210 are then aligned with each other to generate a candidate image (step S220). Similar to the aforementioned method 100, the goal is to generate a predetermined feature (for example, placing front and back images side by side, having a predetermined pixel size..., etc.), and without information other than the main body (ie, only Clean image of bubble packaging and/or label image).

Then, the candidate image generated in step S220 is compared with the reference image stored in the medicine database (S240), and the result is output to a user (S250). Step S220 may be performed by the same or different computing device as steps S210, S240, and S250. According to the present disclosure, step S220 is performed on a computing device, which is different from the computing devices that perform steps S210, S240, and S250. According to this, step S230 can be selected for execution. This step is to transfer the processed image or candidate image in step S220 to the computing device that executes steps S240 and S250.

Returning to step S220, which is similar to step S120 of method 100, and generally includes the following steps: (S221) processing the front and back images to generate two first processed images with defined contours; (S222) identifying the two The corner points of each defined contour of the first processed image to determine its coordinates; (S223) Rotate the two first processed images of step S221 based on the determined coordinates of step S222 to generate two second processed images An image; and (S224) combining the two second processed images of step S223 to generate the candidate image of the step.

In step S221, the image processor processes the front and back images separately to generate two first processed images, where each processed image has a clearly defined outline. To achieve this, the front and back images of step S221 are subjected to the following processes: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) an edge recognition process, (iv) a convex hull Operational processing and (v) contour finding processing. Note that the processes (i) to (v) can be performed independently in any order. In actual operation, a variety of known algorithms can be used to eliminate background noise of two original images and extract target features from these images.

Specifically, (i) use a color conversion algorithm to perform gray-scale conversion processing (S2211); (ii) use a filter algorithm to perform noise filtering processing to minimize background noise (S2212); (iii) Use the edge recognition algorithm to perform the edge recognition process to determine the coordinates of each edge of the bubble package in the image (S2213); (iv) use the convex hull algorithm to perform the convex hull operation process to calculate the image of the drug bubble package The actual area (S2214); and (v) The contour finding process is performed using a contour definition algorithm to extract and highlight the main area of the bubble packaging (S1215).

The actual application is that the original image in step S221 undergoes all the processes (i) to (v) before proceeding to step S222. As described above, these processes can be performed in any order, so the original image obtained in step S121 or the image after receiving any process can be used for the next process. For example, the original image of S221 is first subjected to (i) processing or step S2211, in which the BGR color of the original image is converted into grayscale, thereby generating a grayscale image. Then, by any processing from (ii) to (v), the gray-scale image from (i) processing is processed until all (i) to (v) processing is successfully applied to the image, thereby generating The first processed image that defines the outline.

Or, or unnecessarily, the original image in step S221 is first subjected to (iv) processing or step S1214. In this step, a convex hull algorithm is executed. Convex hull arithmetic is used to find the convex hull of a set of points in a plane or other low dimensions. In this disclosure, the convex hull is defined by calculating the actual area of the drug bubble packaging in the image. The image obtained from the (iv) process or step S2214 can then be subjected to any one of (i), (ii), (iii) or (v) until all the processes (i) to (v) are successfully applied at On this image, a first processed image with a defined contour is then generated.

In a preferred embodiment, the processing is performed in the order of step S2211 to step S2215. The strategies and algorithms used in steps S2211 to S2215 are the same as those described in step S121, so they are not repeated here for simplicity.

It should be noted that each of steps S2212 to S2215 can be executed by any alternative algorithm known in the art, as long as the algorithm can achieve the same result as described above.

By performing steps S2211 to S2215, two first processed images each having a defined contour are generated from two original images (ie, front and back images obtained from a blister package of a drug), wherein the background noise of the original image It is eliminated, and the main part of the bubble packaging is extracted and eye-catching to facilitate subsequent image processing procedures.

Then continue to perform steps S222 to S224. The purpose of these steps is the same as that described in steps S122-S124. Step S222 measures at least three straight lines of the blister package, and then uses geometric reasoning to obtain the quadrilateral closest to the edge of the blister package and the coordinates of each corner point. Once the step S222 of identifying corner points is completed, step S223 can be executed based on the determined coordinates to generate two processed, eye-catching images with fixed feature templates. In step S224, the two second processed images can be regularized side by side and merged into one image, thereby generating a candidate image containing information on both sides of the medicine package. The strategy used in step S220 is the same as the strategy used in the method 100, and will not be repeated here.

Next, in step S240, the merged image or candidate image is compared with the reference image stored in the drug database, so as to determine the identity of the drug. In some embodiments, the drug database may exist in a storage medium readable by a computing device or other processor having the same or different instructions as used to execute step S220. According to a non-essential embodiment of the present disclosure, the candidate image may be generated in the first processing unit embedded with the instruction for executing step S220, and then the processed image may be transferred to the drug database located in the second processing unit (for example, by borrowing The drug database established by the aforementioned method 100). Step S240 is executed under the machine learning instruction embedded in the storage medium, to compare the candidate image with the reference image in the drug database, and output the result (whether or not there is a match) to a user. If the comparison result indicates that the candidate image matches a specific reference image in the drug database or has a high degree of similarity, the drug information corresponding to the specific reference image is output. If the result of the comparison cannot produce a matching result between the candidate image and all reference images in the database, the conceptual representation of "no matching result" is output.

As mentioned above, the purpose of machine learning algorithms is to improve the visual recognition of drugs. In some embodiments, the exemplary algorithm may be a deep learning algorithm executed under any conventional visual object detection model, which may or may not be optimized with certain criteria according to actual needs. In an exemplary embodiment, deep learning is performed under an optimized detection model. In another exemplary embodiment, the candidate image processed by the method of the present invention should be a "full version image" having a predetermined pixel size (for example, 448×224 pixels); therefore, the operation parameters of machine learning can be set to the minimum value , To improve the calculation speed and performance.

It should be noted that by organizing two processed images into one "merged image", the machine learning system and/or method of the present disclosure can also run smoothly and quickly when processing large amounts of data for large-scale pharmaceutical packaging. In other words, the processed and eye-catching images actually increase the computing power and accuracy. Based on the above technical features, the purpose of the present disclosure is to identify the method of the drug 200 through the bubble packaging of the drug, which can improve the accuracy of drug identification and eliminate human errors during the dispensing process, thereby improving the safety of drug use and the quality of patient care .

The subject matter described herein can be implemented using non-transitory, tangible processor-readable storage media storing processor-readable instructions. When executed by a processor of a programmable device, the instructions can control the programmable device to execute the method according to the embodiments of the present disclosure. Exemplary processor-readable storage media suitable for implementing the subject matter described herein may include (but are not limited to) RAM, ROM, EPROM, EEPROM, flash memory, or other native memory technologies: CD-ROM, DVD, or Other optical storage, magnetic card, disk storage or other magnetic storage devices, and other media that can be used to store the required information and can be read by the processor. In addition, the processor-readable storage medium implementing the subject of the present invention may be located on a single device or computing platform, or may be distributed on multiple devices or computing platforms. In some embodiments, the computing platform is an embedded system with real-time computing constraints.

3. Drug Management System

Another aspect of the present invention is to provide a medicine management system. Referring to FIG. 3 of the drawing system 300, the system 300 includes an image capturing device 310, an image processor 320, and a machine learning processor 330, wherein the image capturing device 310 and the machine learning processor 330 are respectively coupled to the Image processor 320. The image capturing device 310 is configured to capture multiple images of a medicine package. In some embodiments, the structure of the image capturing device 310 includes a transparent plate 3101 and two image capturing units 3102 disposed on both sides of the transparent plate 3101 respectively. The transparent plate body 3101 may be made of glass or acrylic polymer. In implementation, the medicine is placed on the transparent plate 3101, and two images of the medicine package are simultaneously captured by two image capturing units 3102. For example, the two image capturing units 3102 are real-time digital cameras.

Unless otherwise stated, according to the present disclosure, the image processor 320 and the machine learning processor 330 each include a memory for storing a plurality of instructions that enable the processor to implement the method of the present invention. In some embodiments, the image processor 320 and the machine learning processor 330 are set as two independent devices; or the two may be set in the same hardware. In some embodiments, the image processor 320 and the machine learning processor 330 are communicatively connected to each other. Specifically, the image processor 320 is communicatively connected to the image capture device 310 to accept the image captured by the image capture device 310 and is configured to perform the image processing steps of the method of the present invention (such as step S120 And S220), generating candidate images that can be used for subsequent identification. The machine learning processor 330 is communicatively connected to the image processor 320, and is configured to implement the image comparison of the method of the present invention (for example, step S240) for drug identification. This step includes comparing candidate images with reference images in the drug database created by the method of the present invention. The drug database can be communicatively connected to the machine learning processor 330. In the exemplary embodiment described in FIG. 3, is the medicine database 3301 of the present invention stored in the machine learning processor 330 or is it optional, or the medicine database may be stored in a machine connected to the machine through a cable connection or a wireless network The learning processor 330 is connected to a storage device.

In some embodiments, the system 300 further includes a user interface (not shown) configured to output the drug recognition results, accept instructions from external users, and feed back user input to the image processor 320 and machine learning Processor 330.

Various techniques can be used to implement communication between the image capturing device 310, the image processor 320, and the image learning processor 330. For example, the system 300 of the present invention may include a network interface to allow the image capturing device 310, the image processor 320, and the machine learning processor 330 to pass through a network (such as a local area network (LAN), a wide area network Way (WAN), network or wireless network) to communicate. In other embodiments, the system may have a system bus that couples various system components (including the image capturing device 310) to the image processor 320.

The following provides a number of embodiments to illustrate some aspects of the present invention, so that those with ordinary knowledge in the technical field to which the present invention pertains can implement the present invention. These experimental examples should not be regarded as limiting the scope of the present invention. Without further explanation, it is believed that those with ordinary knowledge in the technical field to which they belong can make maximum use of the present invention according to the description herein. All publications cited herein are incorporated by reference in their entirety.

Example 1: Building a drug database

Prepare the corrected double-sided and straighten the image (Rectified Two-sided Images , RTIs)

Mackay Memorial Hospital (Taipei, Taiwan) Hospital Pharmacy collects more than 250 existing drugs on the market. In order to build the database of the drug database, as much as possible to get all the photos of the blister packaging. Using the development function library (Open Source Computer Vision 2, OpenCV 2) function programmed in the image processor, the image is cropped to minimize background noise and processed to produce multiple merged images of each drug ( Also known as "correcting double-sided and straighten images (RTIs)"). Each RTI can be fitted with a predetermined template (hereinafter referred to as a corrected two-sided template (RTT) and contains both sides of a drug bubble package. A total of 18,000 RTIs are obtained and CNN is used The model performs subsequent deep learning processing procedures.

Strategies for detecting irregular quadrilateral corners of drug packaging

In the case where the drug package to be tested has an irregular quadrilateral shape composed of three straight edges and one curved edge, the centroid algorithm (Table 1) is used to determine the curved edges and undefined corner points through geometric reasoning. Table 1: Centroid algorithm for detecting corners of irregular quadrilateral shapes

FIG. 4 exemplarily shows how to perform corner recognition on a medicine package having an irregular quadrilateral. As depicted in Figure 4, a blister package with three straight edges ( L ₁ , L ₂ , L ₃ ) and a curved edge (C1) presents a random direction, where the intersection of L ₁ and L ₂ is designated as P ₁ , And the intersection of L ₂ and L ₃ is designated as P ₄ . The goal is to determine the coordinates of P ₂ and P ₃ , according to the area surrounded by four points ( P ₁ , P ₂ , P ₃ and P ₄ ) and four edges ( L ₁ , L ₂ , L ₃ and C ₁ ) Will cover the image of the entire package. First determine the midpoint M on the edge L ₂ between P ₁ and P ₄ , then the centroid B of the bubble packaging area can be determined by the algorithm cv2.moments (OpenCV 2). The displacement vector v is calculated based on the coordinates of the midpoint M and the center of mass B. Finally, it can be determined that the positions of the displacement vectors twice from the intersection points P ₁ and P ₄ are the coordinates of P ₂ and P ₃ . Through the foregoing process, the four intersection points or corner points P ₁ , P ₂ , P ₃ and P ₄ can be automatically sorted in a clockwise or counterclockwise direction.

Optimize machine learning capabilities

In this embodiment, a convolutional neural network (Convolutional Neural Network, CNN) is used to achieve the purpose of machine learning. In order to achieve the aforementioned objective, a miniature version of YOLOv2 (also called Tiny YOLO) nerve executed by a graphics processing unit (GPU) is used to train the visual recognition of each RTI. The general concept of conventional YOLOv2 is to divide an image into multiple grids and use anchor boxes to predict bounding boxes. In short, for each RTI image with a size of 416×416 pixels in this embodiment, when input to the network, the Tiny YOLOv2 nerve will generate an odd number of grids in each image; accordingly, the There will only be one grid (ie, the center grid). Then, each grid and its neighbor grids are analyzed to determine whether these grids contain any feature information. For example, the input image can be divided into 13×13 grids, in which five size bounding boxes can be predicted, and the confidence score returned from each grid can be used to determine whether the grid contains the analysis Objects. In this embodiment, because the input image has been highlighted and outlined, the optimization of conventional Tiny YOLO can improve the recognition efficiency and reduce the operation cost. Since each RTI input image has been processed into a "full version image" and has been adapted to a predetermined pixel size, the number of operating anchor frames can be reduced to one, and the size of the anchor frame can also be set at 7×7 Grids. According to this, the machine learning network only needs to predict a bounding box with a full-size (full-size) input image. The specific parameters of optimized Tiny YOLO are listed in Table 2. Table 2: Parameters of optimized Tiny YOLO network

During execution, the optimized Tiny YOLO can be programmed on a machine learning processor with a high-resolution graphics card (GEFORCE ^® GTX 1080 (NVIDIA, USA)). The optimized Tiny YOLO training model is used to make the machine learning processor execute the deep learning program. Table 3 lists the optimized training guidelines for Tiny YOLO. Table 3: Optimized training guidelines for Tiny YOLO

Example 2: Evaluation of the classification efficiency of a drug management system that performs machine learning based on the pharmaceutical database of Example 1

Process and merge images

In order to evaluate the visual recognition performance of the RTIs of this disclosure, the original images of the drugs (unprocessed and unmerged) and RTIs (processed and merged) were input to the training model (optimized Tiny YOLO) for deep learning. Table 4 summarizes the training results. According to the data in Table 4, compared to the unprocessed original image, the "eye-catching" image of the present disclosure is highly effective in training visual recognition. The higher the trained F1-score, the better the deep learning network's effect on the visual recognition of packaging images. Compared with those unprocessed images, RTIs processed into RTT also significantly increase the training effect. Table 4: Comparison of unprocessed images and RTIs

Optimized learning model

Another comparison was made to evaluate the effectiveness of the training model of the present invention. Two conventional learning models are used: ResNet 101 and SE-ResNet 101 to build a training model of RTIs. Table 5 summarizes the comparison results. Table 5: Comparison between optimized system and conventional system

From the comparison results of Table 4 and Table 5, it can be seen that, especially when processing images with the optimized Tiny YOLO model, the RTIs processed by the method of the present invention to adapt to RTT can increase training performance and also significantly shorten training time. All 18,000 RTIs were input into the training model, thereby establishing the drug database of the present invention by executing the optimized Tiny YOLO model. The drug database and deep learning network can be further stored in an embedded system for subsequent applications.

Real-time drug identification application

During operation, a drug is randomly selected and placed in a special cabinet equipped with a transparent plate made of glass. Set up a light source around the transparent board to facilitate lighting, and set two BRIO network cameras (Logitech, USA) on both sides of the transparent board (at a distance from the board) to ensure the vision of the network camera Covers the entire board area. On the other hand, the established drug database is first stored in the real-time embedded computing device JETSON ^TM TX2 (NVIDIA, USA). JETSON ^TM TX2, called "developer kit", contains memory, CPU, GPU, USB port, mesh antenna and other computing components, so that the image processing steps and machine learning steps can be on the same device Executed within. In operation, the two webcams can be connected to JETSON ^TM TX2 via a USB cable, according to which the images of both sides of the selected drug can be transmitted to the processor simultaneously in real time. The two original images of the selected medicine bubble packaging are processed into one RTI, and then the RTI accepts the learning model Tiny YOLO programmed in JETSON ^TM TX2 for subsequent procedures. The visual recognition speed of the Tiny YOLO model can reach about 200 frames per second (FPS). The total process (ie, from capturing images to visual recognition) takes approximately 6.23 frames per second. The recognition result can be presented on an external display device in real time, such as a computer screen or mobile phone user interface. Therefore, the external user can obtain the identification result of the medicine at the same time as putting the medicine in the cabinet.

Furthermore, in order to evaluate whether the system of the present invention can effectively reduce the probability of human error in the dispensing process, another comparison is made again. For example, an anti-anxiety drug: Lorazepam (Lorazepam), its appearance is quite similar to other drugs, so it is the most misunderstood drug during the dispensing process. First, the RTI of Renepine was entered into the system of the present invention to be compared with all drugs in this database. After calculation, the learning model proposes several drug candidates that may be misidentified based on appearance similarity after analysis by the learning model. The candidate form can assist the clinical staff to check whether the selected drug is correct. Therefore, the drug management system of the present disclosure can improve the accuracy of dispensing, and can also minimize human error to ensure patient safety.

In summary, the method and system for drug management of the present invention can achieve the aforementioned objective by performing image processing steps to merge images into one, and executing a deep learning model in a real-time manner. The advantage of the present disclosure is not only that the original image can be effectively processed regardless of the orientation of the object (ie, medicine), but also the accuracy of medicine classification can be increased through the appearance of the medicine.

It should be understood that the foregoing description of the embodiments is given by way of examples only, and those with ordinary knowledge in the technical field to which the art belongs can make various modifications. The above specification, examples and experimental results provide a complete description of the structure and use of exemplary embodiments of the invention. Although the above embodiments disclose various specific examples of the present invention, they are not intended to limit the present invention. Those with ordinary knowledge in the technical field to which the present invention belongs, without departing from the principle and spirit of the present invention, Various changes and modifications can be made to it, so the scope of protection of the present invention shall be defined by the scope of the accompanying patent application.

3102 image capturing unit 320 image processor machine learning processor 330 3301 Drug Database S110-S140, S210-S250 step 300 method 100, 200, 310 image capturing apparatus 3101 system transparent plate

In order to make the above and other objects, features, advantages and embodiments of the present invention more comprehensible, the drawings are described below.

FIG. 1 is a flowchart of a method 100 according to an embodiment of the present disclosure.

FIG. 2 is a flowchart of the method 200 according to an embodiment of the present disclosure.

FIG. 3 illustrates a medicine management system 300 according to an embodiment of the present disclosure.

FIG. 4 shows an example of this disclosure to illustrate how to define the corners of the package through the centroid algorithm.

According to the usual working methods, the various elements and features in the drawings are not drawn to scale. The drawing method is to present the specific features and elements related to the present invention in an optimal manner. In addition, the same or similar element symbols are used to refer to similar elements/components between different drawings.

S110-S140: Steps

Claims (17)

A computer-implemented method for establishing a drug database includes: (a) receiving a plurality of original images of a drug package; (b) arranging two of the plurality of original images to generate a combined image, wherein the two The original images are different from each other; (c) processing the merged image to generate a reference image; and (d) using the reference image to create the drug database, wherein step (b) includes: (b-1) processing separately The plurality of original images to generate a plurality of first processed images each having a defined contour; (b-2) identifying the corner points of each defined contour of the plurality of first processed images to determine their coordinates; (b -3) Based on the coordinates determined in step (b-2), rotate the first processed images of step (b-1) to generate a plurality of second processed images; and (b-4) combine the plurality of Second, process any two of the images to generate the combined image of step (b). The computer-implemented method as described in claim 1, further comprising simultaneously capturing the plurality of original images of the medicine package before step (a). The computer-implemented method according to claim 1, wherein the medicine is in the form of a blister pack. The computer-implemented method according to claim 1, wherein each of the plurality of original images in step (b-1) is subjected to the following processing: (i) a gray-scale conversion process, (ii) a noise filtering process, (iii ) An edge recognition process, (iv) a convex hull operation process, and (v) a contour search process. The computer-implemented method according to claim 1, wherein step (b-2) is performed with a linear transformation algorithm or a centroid algorithm. The computer-implemented method of claim 1, wherein the combined image includes a double-sided image of the blister package of the drug. The computer-implemented method according to claim 1, wherein step (c) is performed with a machine learning algorithm. A computer-implemented method for identifying a medicine through a medicine bubble package, including: (a) acquiring both front and back images of the medicine bubble package; (b) regularizing the front and back images of step (a) To generate a candidate image; (c) compare the candidate image with the reference image of the drug database created as described in claim 1; and (d) output the result of step (c), wherein step (b) ) Includes: (b-1) separately processing the front and back images of step (a) to generate two first processed images each having a defined contour; (b-2) identifying the two first processed images The corner points of each defined contour to determine its coordinates; (b-3) Based on the coordinates determined in step (b-2), rotate the two first processed images of step (b-1) to generate two second processed images; and (b-4) combine The two second processed images of step (b-3) to generate the candidate image of step (b). The computer-implemented method according to claim 8, wherein the front and back images of step (b-1) are subjected to the following processes: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) ) An edge recognition process, (iv) a convex hull operation process, and (v) a contour search process. The computer-implemented method according to claim 8, wherein step (b-2) is performed with a linear transformation algorithm or a centroid algorithm. The computer-implemented method according to claim 8, wherein step (c) is performed with a machine learning algorithm. The computer-implemented method described in claim 8 further includes sending the candidate image to the drug database before step (c). A medicine management system includes: an image capture device for capturing a plurality of images of a medicine package; an image processor, programmed by instructions to execute a method for generating a candidate image, wherein the method includes: ( 1) Process the plurality of images of the drug package separately to generate a plurality of first processed images each having a defined contour; (2) Identify the corner points of each defined contour of the plurality of first processed images to determine Its coordinates; (3) Based on the coordinates determined in step (2), rotate the first processed images of step (1) to generate a plurality of second processed images; and (4) normalize the plurality of second processed images of step (3) Processing two of the images to generate the candidate image, wherein the two second processed images are different from each other; and a machine learning processor, through instruction programming, executes a method that is used to compare the candidate image with Reference image of the drug database created by the method described in claim 1. The system according to claim 13, wherein the image capturing device includes: a transparent plate body for placing the drug thereon; and two image capturing units, which are individually disposed above each side of the transparent plate body. The system according to claim 13, wherein each of the plurality of images in step (1) is subjected to the following processing: (i) a grayscale conversion process, (ii) a noise filtering process, (iii) an edge recognition process, (iv) A convex hull calculation process and (v) A contour finding process. The system according to claim 13, wherein step (2) is performed with a linear transformation algorithm or a centroid algorithm. The system of claim 13, wherein the method for comparing the candidate image with the reference image of the drug database is performed by a machine learning algorithm.

Images