JP2020018353A - Methods and system for sorting and identifying medication by its label and/or package - Google Patents

Abstract

To provide methods and a system for sorting and identifying a medication by its label and/or package.SOLUTION: Disclosed herein is an improved pharmaceutical management system and methods implemented by a system for sorting and identifying a medication by its label and/or package. The method comprises the steps of: (a) receiving a plurality of raw images of a package of a medication; (b) juxtaposing two of the plurality of raw images to produce a combined image, in which the two raw images are different from each other; (c) processing the combined image to produce a reference image; and (d) establishing a medication library using the reference image. The system comprises an image capturing device, an image processor, and a machine learning processor. The image processor is programmed with instructions to execute the method for producing a combined image.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates generally to the field of pharmacy management systems. More specifically, the present disclosure relates to methods and systems for sorting and identifying medications by drug label and / or packaging.

  Systems and / or methods for efficient storage and management of medical and pharmaceutical products are important requirements for providing smooth operation of hospital systems and high quality patient care. In such cases, the accuracy of the prescription is of paramount importance for all medical institutions. Although automated dispensing cabinets (ADCs) have been introduced to hospitals for over 20 years, dispensing errors still exist. Medication control by ADCs depends heavily on the clinical staff's correct adherence to standard protocols, and it is impossible to completely eliminate eventual human error. For example, a pharmacy may place the wrong drug in a given drug cabinet, or a clinician may pick a "similar" drug from the next drug drawer. Therefore, it is unreliable for a human eye to identify a medicine by its appearance, and it cannot be applied to a medicine management system.

  Deep learning techniques are one of a broad family of machine learning techniques based on multi-level learning data representations, each of which replaces one level representation with a higher, slightly abstracter level representation. It is obtained by constructing a simple but non-linear module that converts to A complex function, such as a classification task, can be learned with a sufficiently large transformation configuration. Thus, deep learning has made great strides in solving the problems that exist in the artificial intelligence community for many years and has been applied in many technical fields. For pharmacy management, the superior performance of deep learning in identifying the appearance of a subject appears to be a promising solution to the shortcomings of current drug dispensing technology. However, given the vast variety of drug packages with specific appearance similarities, simply performing deep learning did not produce the desired results.

  In view of the foregoing, there is a need in the related art for improved methods and systems for drug management (eg, distinguishing and identifying drugs by their labels and / or packaging).

  The following presents a brief summary of the disclosure in order to provide the reader with a basic understanding. This summary is not an extensive overview of the disclosure and it does not identify key / critical elements of the invention or delineate the scope of the invention. Its sole purpose is to present some concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

  As embodied and generally described in this overview, the objectives of the present disclosure are implemented by a system for identifying clinical drugs so that the efficiency and accuracy of drug dispensing is significantly improved. An improved dispensing management system and method.

  In one aspect, the present disclosure is directed to a computer-implemented method for constructing a drug library. In some embodiments, the method includes: (a) receiving a plurality of raw images of the package of the drug; and (b) juxtaposing two of the plurality of raw images to generate a combined image. A step, wherein the two raw images are different from each other; (c) processing the combined image to generate a reference image; and (d) establishing a drug library using the reference image. Step.

  In some optional embodiments, the method further comprises, prior to step (a), simultaneously capturing a plurality of raw images of the drug package.

  According to certain embodiments of the present disclosure, preferably the medicament is in blister packaged form.

  According to a particular embodiment of the present disclosure, the step (b) comprises: (b-1) processing each of the plurality of raw images to generate a plurality of first processed images each having a defined contour. (B-2) identifying the corners of each of the defined contours of the first processed image to determine the coordinate position of the corners; and (b-3) generating a plurality of second processed images. And (b-4) rotating each of the first processed images in step (b-1) based on the coordinate position calculated in step (b-2). Combining any two of the second processed images to generate a combined image.

  According to a particular embodiment of the present disclosure, each of the plurality of raw images in step (b-1) includes: (i) a grayscale conversion process, (ii) a noise reduction process, (iii) an edge identification process, (iv) And (v) contour formation processing.

  According to the embodiment of the present disclosure, each processing of (i) to (v) may be separately performed in an arbitrary order, and in each processing of (i) to (v), the processing of step (b-1) A raw image or an image processed by any of the processes (i) to (v) other than the current process is also used.

  In some embodiments of the present disclosure, step (b-2) is performed by a line transformation algorithm or a centroid algorithm.

  Preferably, the combination image comprises a double-sided image of a blister package of the medicament.

  According to some embodiments of the present disclosure, step (c) is performed by a machine learning algorithm.

  In another aspect, the present disclosure relates to a computer-implemented method for identifying a drug by a blister package. The method includes the steps of (a) simultaneously obtaining front and back images of a blister package of a drug; and (b) juxtaposing the front and back images from step (a) to generate a candidate image. And (c) comparing the candidate image with the reference image of the drug library established by the method described above, and (d) outputting the result of step (c).

  According to certain embodiments of the present disclosure, the step (b) comprises (b-1) processing the front and back images, respectively, to generate two first processed images, each having a defined contour. (B-2) identifying the corners of each of the defined contours of the two first processed images to determine the coordinate position of the corners; and (b-3) two second processes. Rotating each of the two first processed images of step (b-1) based on the calculated coordinate positions of step (b-2) to generate an image; Combining the two second processed images of step (b-3) to generate the candidate image of step (b).

  According to a particular embodiment of the present disclosure, the front and back images of step (b-1) are, respectively, (i) a grayscale conversion process, (ii) a noise reduction process, (iii) an edge identification process, iv) convex hull processing and (v) contour formation processing.

  In some embodiments, each of the processes (i) to (v) may be performed separately in any order, and in each of the processes (i) to (v), the front of the step (b-1) is performed. And images on the back side, or images processed by any of the processes (i) to (v) other than the current process.

  In some optional embodiments, step (b-2) is performed by a line transformation algorithm or a centroid algorithm.

  In some optional embodiments, step (d) is performed by a machine learning algorithm.

  In some optional embodiments, the method further comprises, prior to step (c) above, transferring the candidate image into a drug library.

  In yet another aspect, the present disclosure is directed to a pharmacy management system, the system including an image capture device, an image processor, and a machine learning processor configured to implement the method.

  More specifically, the image capture device is configured to capture a plurality of images of the medication package. The image processor is programmed with instructions for performing a method for generating a candidate image, the method comprising: (1) packaging a drug to generate a plurality of first processed images, each having a defined contour; (2) identifying the corners of each of the defined contours of the first processed image to determine the coordinate position of the corners; and (3) processing a plurality of second processes. Rotating each of the first processed images based on the coordinate position identified in step (2) to generate an image; and (4) generating a candidate image from two different second processed images. And juxtaposing two of the second processed images of step (3). Further, the machine learning processor is programmed with instructions for performing a method for comparing the candidate image with the reference image of the drug library described above. The results generated by the machine learning processor can subsequently be output to notify the person at the drug dispensing destination.

  In some embodiments of the present disclosure, the image capture device comprises a transparent board on which the medicament is placed, and two image capture units separately positioned above and below each side of the transparent board. Including.

  In some embodiments of the present disclosure, each of the plurality of images in step (1) includes (i) a grayscale conversion process, (ii) a noise reduction process, (iii) an edge identification process, and (iv) a convex hull. Processing and (v) contour forming processing.

  In some embodiments, each of the processes (i) to (v) may be performed separately in any order, and in each of the processes (i) to (v), the plurality of images of the above step (1) may be used. Or an image processed by any of the processes (i) to (v) other than the current process.

  Additionally or alternatively, step (2) may be performed by a line transformation algorithm or a centroid algorithm.

  In some optional embodiments, the method for comparing the candidate image with the reference image of the drug library is performed by a machine learning algorithm.

  With the above arrangement, the medicine management method and system for classification and identification can be performed in a real-time manner, so that during the medicine dispensing process, the whole processing of the image classification regardless of the medicine orientation is performed. Time is reduced.

  Furthermore, the appearance of the drug and / or the blister package can be used to improve the accuracy of its identification and greatly reduce human errors during the drug dispensing process. This could improve the safety of drug use.

  Many features and advantages of the present disclosure will be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings.

  The present specification will be better understood from the following detailed description read in light of the accompanying drawings.

4 is a flowchart illustrating a method 100 according to one embodiment of the present disclosure. 4 is a flowchart illustrating a method 200 according to one embodiment of the present disclosure. FIG. 6 is a diagram illustrating a dispensing management system 300 according to another embodiment of the present disclosure. FIG. 4 is a schematic diagram illustrating how corners of a package are identified by using a centroid algorithm, according to an example of the present disclosure.

  In accordance with common practice, the various features / elements described are not drawn to scale, but instead are drawn to best represent certain features / elements relevant to the invention. Also, like reference numerals and designations are used in the various drawings to indicate like elements / parts.

  The detailed description provided below in connection with the accompanying drawings is intended as an illustration of an example presentation, and is not intended to represent only the forms in which the examples are constructed or available. This description sets forth the functions of the example and the sequence of steps for constructing and operating the example. However, the same or equivalent functions and sequences can be achieved by different examples.

I. Definitions For convenience, certain terms employed in the specification, examples, and appended claims are collected in this paragraph. Unless defined otherwise herein, scientific and technical terms used in this disclosure shall have the meanings commonly understood and used by one of ordinary skill in the art. Unless otherwise required by context, singular terms shall include the plural and plural terms shall include the singular. Specifically, as used herein and in the claims, the singular forms "a" and "an" include plural referents unless the context clearly dictates otherwise. Further, the terms “at least one” and “one or more” used in the present specification and claims have the same meaning, and include one, two, three, or more.

  The numerical ranges and parameters describing the broad range of the present invention are approximate values, but the numerical values describing specific examples are described as accurately as possible. However, all numerical values inherently include certain errors necessarily resulting from the standard deviation present in each measurement. Also, as used herein, the term "about" generally means within 10%, 5%, 1%, or 0.5% of a given value or range. Alternatively, the term "about" means within an acceptable standard error of the mean, as one of ordinary skill would appreciate. All numerical ranges, amounts, etc. disclosed herein, such as amounts of materials, elapsed times, temperatures, operating conditions, ratios of amounts, and the like, are not examples of operation / work or unless explicitly specified otherwise. , Numbers, and percentages are to be understood as being modified in all instances by the term “about”. Thus, unless otherwise indicated, numerical parameters set forth in this disclosure and the appended claims are approximations that may vary as needed. At a minimum, each numerical parameter should be interpreted applying at least the number of significant digits described and applying common rounding techniques. As used herein, a range may be expressed as from one endpoint to another, or between two endpoints. All ranges disclosed herein include these endpoints unless otherwise indicated.

  As used herein, the term "blister pack" or "blister package" includes any type of layered package that includes a product encapsulated between sheets of material, where the sheets are, for example, heat and And / or bonded together in a manner known to those skilled in the art, such as by being adhesive activated by pressure. The sheet material is commercially available as individual sheets (for manual packaging) or, preferably, as continuous web sheets in roll form (for mechanical packaging). The main component of the blister package is a cavity or pocket, usually made of thermoformed plastic, made of molding wrap. This cavity or pocket is large enough to contain the goods stored in the blister package. For some applications, the blister package may be provided with a backing of thermoformed material. In the field of dispensing, a "blister package" is commonly used as a unit dose package of a pharmaceutical tablet and contains drug information printed on its back. Further, these sheets are available in various thicknesses.

II. DESCRIPTION OF THE INVENTION A primary concern in patient care is the safety of drug use. Correctly prescribing and dispensing medication is a paramount concern for patient care. Conventional automatic dispensing systems, such as ADCs, still have room for improvement, and the present invention seeks to provide an improved and highlighted deep learning method to address the aforementioned problems. It is another object of the present invention to develop an automatic medicine verification (AMV) device using a real-time operation system in order to reduce the workload of clinical staff.

  More specifically, a "highlighted" deep learning method refers to a method of processing and enhancing features or images so that they can be learned more intensified before they are input to the algorithm. Say. Generally, to better classify identified targets and extract unique descriptive features, this feature processing involves a variety of image processing that results in the joining of both cropped photos of the blister package appearance in certain signature templates Achieved by a step, thereby facilitating the subsequent classification process in the learning network.

1. Method of Building Drug Library A first aspect of the present disclosure is directed to a method for building a drug library implemented in a computer-readable storage medium. Please refer to FIG.

Referring to FIG. 1, this figure is a flowchart of a computer-implemented method 100 according to one embodiment of the present disclosure. The method comprises at least:
(S110) receiving a plurality of raw images of the medicine package;
(S120) juxtaposing two of the plurality of raw images to generate a combined image, wherein the two raw images are different from each other;
(S130) processing the combined image to generate a reference image;
(S140) establishing a drug library using the reference image;
including.

  Before the method 100 is performed, a plurality of raw images of a drug package (eg, a blister package) are captured by any well-known manner, such as an image capture device (eg, a video camera). Then, in step S110 of the method, the captured image is automatically transferred to a device and / or system (eg, a computer) having embedded instructions to perform the method 100. In a next step S120, two of the plurality of raw images received by the above device and / or system are juxtaposed to each other to generate a combined image. Note that these two side-by-side raw images are different from each other. In one exemplary embodiment, the medicament is in a blister package, so that the plurality of raw images taken by the image capture device can encompass both sides of the blister package. In other words, two images, each showing the front and back of the blister package, are juxtaposed to each other to generate a combined image. In order to improve the accuracy of appearance recognition for later machine learning processes, the raw image is highlighted and processed to obtain a suitable image with certain features (eg, certain pixel size, etc.). All background parts other than the subject (ie, the blister package and / or label image) are completely removed.

  In step S130, this combined image is then used to train a machine learning algorithm embedded in a computer (eg, a processor) to generate a reference image. Steps S110-S130 can be repeated any number of times, each time using a different drug package than the previous one. Finally, a drug library can be established using this reference image (step S140).

Returning to step S120, the step typically includes
(S121) processing each of the plurality of raw images to generate a plurality of first processed images each having a defined contour;
(S122) identifying the corners of each defined contour of the first processed image to determine the coordinate position of the corners;
(S123) rotating each of the first processed images in step S121 based on the coordinate positions calculated in step S122 to generate a plurality of second processed images;
(S124) combining any two of the second processed images to generate the combined image of step S120;
including.

  In step 121, the plurality of raw images are each processed by an image processor to ultimately generate a plurality of first processed images, each having a well-defined contour. The image processor uses several algorithms to remove background noise and extract target features from the image. Specifically, each of the plurality of raw images in step S121 includes (i) gray scale conversion processing, (ii) noise reduction processing, (iii) edge identification processing, (iv) convex hull processing, and (v) ) It is applied to the contour forming process. The processes (i) to (v) are individually performed in an arbitrary order.

  More specifically, (i) the grayscale conversion process is performed using a color conversion algorithm to change the BGR color to grayscale (S1211), and (ii) the noise reduction process minimizes background noise. (S1212), and (iii) an edge identification process is performed using an edge identification algorithm to calculate the coordinate position of each edge of the blister package in the image. S1213), (iv) convex hull processing is performed using a convex hull algorithm to calculate the actual area of the blister package of the drug in the image (S1214), and (v) contouring processing is performed by blister Performed using a contouring algorithm to extract and highlight key parts of the package S1215).

  According to a particular embodiment, the raw image in step S121 is subjected to all of the processes (i) to (v) before proceeding to step S122. As described above, these processes can be performed individually and in any order, and therefore, except for the raw image from step S121, the image obtained from one process is the raw image in the next process. It can be. For example, the raw image of step S121 is first subjected to the processing (i) or step S1211. In this step, the BGR color of the raw image is converted to grayscale, thereby generating a grayscale image. The grayscale image obtained from process (i) may be subsequently processed by any of processes (ii) to (v) until all processes (i) to (v) have been successfully applied; As a result, a first processed image having the defined contour is generated.

  Alternatively or optionally, the raw image of step S121 is first subjected to processing (iv) or step S1214, in which a convex hull algorithm is applied to the image. The convex hull algorithm is for finding the convex hull of a finite set of points in a plane, or other low-dimensional Euclidean space. In the present disclosure, the convex hull is defined by calculating the actual area of the blister package of the drug in the image. The image obtained from operation (iv) or step S1214 is then processed by (i), (ii), (iii) or (v) until all operations (i) to (v) have been successfully applied. , Which produces a first processed image having a defined contour.

  The above-described processes (i) to (v) or steps S1211 to 1215 can be performed sequentially, randomly, and / or repeatedly (for example, process (iii) is repeated one or more times) Can be performed in the order from (i) to (v) or from S1211 to S1215.

  By executing the above-described steps S1211 to S1215, a plurality of first processing images each having a defined contour are generated from the plurality of raw images, and in these images, background noise in the raw images is eliminated. The main part of the blister package has been extracted and highlighted for further image processing procedures.

  Proceeding to step S122, corners in each of the first processed images are identified (FIG. 1). In this step, the coordinate position of at least one corner of each of the first processed images is calculated, regardless of the profile shape of the blister package. Preferably, the coordinate positions of each of the four corners of the first processed image obtained from step S121 are identified. The goal of this step is to project at least three straight lines from the edges of the profile of the blister package, and then derive, by geometric estimation, the closest rectangle and its coordinate positions at its corners. The blister package may come in various shapes and / or profiles, and may use various algorithms to identify the corner (s) of the first processed image of the drug package. For example, when the outline shape of the medicine package in the first processing image is a square (for example, a rectangle), a line conversion algorithm is applied to identify four corners. On the other hand, if the medicine package in the first processed image has an irregular quadrangular shape such as an irregular polygon having three straight sides and one curved side, a centroid algorithm is applied for corner identification. With the above configuration, it is possible to calculate the coordinate position of each corner regardless of the orientation and shape of the medicine package.

  In step S122, identifying the corners of each processed image not only serves the purpose of anchor point setting for the subsequent rotation step (S123) and combination step (S124) execution, but also the package for further analysis. It also serves the purpose of ensuring that the whole is included in the first processed image. The four corners of the blister package image calculated above are clockwise or counterclockwise so that the first processed image can be rotated to a predetermined position based on the calculated coordinate positions of the four corners. (S123). This predetermined position may be changed based on practical needs. In some examples, this predetermined position refers to a position where the short and long sides of the blister package are parallel to the X and Y axes, respectively, in a Cartesian coordinate system. Preferably, the blister package is oriented in such a way that the image needs to be rotated only once, so that its short and long sides are parallel to the X and Y axes, respectively, in the Cartesian coordinate system. . In practice, several perspective transformation algorithms can be used to rotate the first processed image, which are set to have a predetermined pixel size (e.g., 448 x 224 pixels), whereby the predetermined A second processed (rotated) image having a position is generated.

  Next, in step S124, any two of the second processed images of the medicine are placed side by side (or placed side by side) to generate a combined image. It should be noted that various sorts can be included in the combined image obtained in this step, which is advantageous in constructing as many library databases as possible. Two of the second processed images need only be on both sides of the blister package, regardless of whether the blister package is facing front or back. Preferably, two of the second processed images, each representing a respective face of the drug package having the same orientation, are selected and juxtaposed to each other to produce a combined image containing the largest characteristic information in the package. Is done.

  In steps S130 and S140, this combined image is used to train a machine learning algorithm built into the computer to generate a reference image for establishing a drug library. This combined image containing the medication information is categorized by its features stored as reference information in the medication library and can be read out to identify candidate packages in the future. In some embodiments, at least one combination image is input to a machine learning algorithm. In an exemplary embodiment, 10, 100, 200, 300, 400, 500, 1000, 1,100, 1,200, 1,300, 1,400, 1,500, 2,000, 2,500, 3, 000, 3,500, 4,000, 4,500, 5,000, 5,500, 10,000, 11,000, 12,000, 13,000, 14,000, 15,000, 16,000 , 17,000, 18,000, 19,000, and 20,000 combined images are input to train machine learning algorithms to establish a drug library. Any image can be used to train a machine learning system to convert image information into reference information, which is subsequently stored in a built-in library in the device and / or system.

  It should be noted that a machine learning programming system suitable for use in the present disclosure may be any known visual object detection model, including, but not limited to, specific criteria depending on practical needs. Includes deformable parts model (DPM), region optimized neutral network (R-CNN), fast R-CNN, ultra-fast R-CNN, ultra-fast R-CNN, mask R-CNN, and YOLO, both optimized and unoptimized . Preferably, the visual object detection model suitable for training the learning step of the present disclosure is optimized at least for parameters such as pixels of the input image, number and size of bounding boxes, and anchor boxes. According to the present disclosure, the combined image processed by the foregoing steps and then input into the learning system must be a "full bleed image" and of a predetermined size (eg, with a certain number of pixels). This allows the number and size of anchor boxes for operations to be minimized (eg, to only one anchor box) in order to increase the speed and efficiency of computing. In other words, the aforementioned steps for juxtaposing two images together make the machine learning system smoother and faster, even when processing vast amounts of data for a large variety of drug packages. It is possible to execute with. Further, the overall processing time is reduced, which further enhances the utility in generating drug libraries.

  By executing steps S130 and S140 described above, it is possible to output a training result for each individual drug as a classification model when the drug library is implemented.

2. Method for identifying a drug A second aspect of the present disclosure is directed to a method for identifying a drug by a package of the drug (eg, a blister package). Please refer to FIG.

Referring to FIG. 2, this figure is a flowchart illustrating a method 200 according to one embodiment of the present disclosure. Method 200 includes:
(S210) simultaneously obtaining the front and back images of the blister package of the medicine;
(S220) juxtaposing the front and back images from step S210 to generate candidate images;
(S230) optionally transferring the candidate image into the drug library;
(S240) comparing the candidate image with a reference image of a drug library established by the method described above;
(S250) outputting the result of step S240;
including.

  Preferably, method 200 can be implemented by instructions for performing the steps of the method, and a processor-readable non-transitory storage medium that contains a drug library. The drug library can be initially stored and installed or established by the method 100 described above.

  In the image receiving step (S210), two raw images of the medicine are simultaneously captured. These two raw images are preferably each side of the blister package of the drug (i.e., front and back images). These raw images can be acquired in any known manner, preferably by at least one image capture device (eg a video camera). In one exemplary embodiment, the two raw images of these blister packages are simultaneously captured by two image capture devices located below and above both sides of the drug (eg, front and back of the drug), These two raw images will contain the medication information on both sides of the blister package. In order to improve the accuracy of appearance recognition for the later machine learning process, the two raw images obtained in step S210 are juxtaposed adjacent to each other to generate candidate images ( Step S220). As in method 100, the goal is to define certain features (e.g., only the blister package and / or label image) without information other than the main subject (e.g., an image of a predetermined pixel size on the front and back side by side). Etc.) to generate a suitable image.

  The candidate image generated in step S220 is then compared with the reference image stored in the medicine library (S240), and the result (ie, medicine information for a specific medicine) is output to the user (S250). Step S220 may be performed on the same or a different device as the computing device for performing steps S210, S240, and S250. According to an optional embodiment of the present disclosure, step S220 is performed on a different computing device than to perform steps S210, S240, and S250, so that optional step S230 is performed, In step, the processed image or candidate image generated in step 220 is transferred to a computing device where steps S240 and S250 are to be performed.

Returning to step S220, which is similar to step S120 of method 100, typically
(S221) processing the front and back images to generate two first processed images each having a defined contour;
(S222) identifying the corners of each defined contour of the two first processed images to determine a coordinate position of the corners;
(S223) rotating each of the two first processed images in step S221 based on the coordinate positions calculated in step S222 to generate two second processed images;
(S224) combining the two second processed images in step S223 to generate a candidate image;
including.

  In step S221, the front and back images are each processed by an image processor to generate a plurality of first processed images, each processed image having a well-defined contour. For this purpose, the front and rear images in step S221 are (i) grayscale conversion processing, (ii) noise reduction processing, (iii) edge identification processing, (iv) convex hull processing, and (v) contour formation, respectively. It is attached to the processing. The processes (i) to (v) are individually performed in an arbitrary order. In implementation, several well-known algorithms may be used to remove background noise from two raw images and extract target features.

  More specifically, (i) the grayscale conversion process is performed using a color conversion algorithm (S2211), and (ii) the noise reduction process uses a filter algorithm to minimize background noise. (S2212), and (iii) edge identification processing is performed using an edge identification algorithm to calculate the coordinate position of each edge of the blister package in the image (S2213), and (iv) convex hull processing is performed. The convex hull algorithm is used to calculate the actual area of the blister package of the drug in the image (S2214), and (v) the contouring process extracts and highlights the major parts of the blister package. In this case, the process is executed using a contour definition algorithm (S2215).

  In practice, the raw image in step S221 is subjected to all of the processes (i) to (v) before the method proceeds to step S222. As described above, these processes can be individually executed in any order, and thus the raw image obtained in step S221 or the image obtained from one process is used for the next process. Can be For example, the raw image of step S221 is first subjected to the processing (i) or step S2211, and in this step, the BGR color of the raw image is converted to grayscale, thereby generating a grayscale image. The grayscale image obtained from this process (i) may be subsequently processed by any of processes (ii) to (v) until all processes (i) to (v) have been successfully applied. Thus, a first processed image having a defined contour is generated.

  Alternatively or optionally, the raw image of step S221 is first subjected to processing (iv) or step S2214, for which a convex hull algorithm is applied. The convex hull algorithm is for finding the convex hull of a finite set of points in a plane, or other low-dimensional Euclidean space. In the present disclosure, the convex hull is defined by calculating the actual area of the blister package of the drug in the image. The image obtained from operation (iv) or step S2214 is then applied to (i), (ii), (iii) or (v) until all operations (i) to (v) have been successfully applied. , Which produces a first processed image having a defined contour.

  In one preferred embodiment, processing is performed sequentially from step S2211 to step S2215. The strategies and algorithms used in steps S2211 to S2215 are similar to those described in step S121, and therefore will not be repeated here for brevity.

  Of course, steps S2212 to S2215 may be performed by any other algorithm known in the art, as long as the algorithm achieves the same result as described above.

  By performing steps S2211 to S2215, from the two raw images (i.e., the front and back images taken from one blister package of one drug), two first processed images, each having a defined contour, are obtained. In the generated image, the background noise in the raw image has been removed and the main part of the blister package has been extracted and highlighted for further image processing procedures.

  Proceeding to steps S222 to S224, the targets of these steps are the same as the targets of steps S122 to S124. In step S222, at least three straight lines of the edges of the profile of the blister package are calculated, and then the geometric estimation derives the closest rectangle and the coordinate positions of its corners. Once step S222 for identifying a corner has been achieved, perform step S223 based on the calculated coordinate positions to generate two images that have been processed and highlighted in a signature template. Could be. In step S224, these two processed images are juxtaposed and combined into one image to generate a candidate image that contains information on both sides of the drug package. The strategy used in S220 is similar to that described in method 100, and thus will not be repeated here for brevity.

  In step S240, the combined image or candidate image is then compared to a reference image stored in a drug library to determine drug identification information. In some embodiments, the drug library is stored on the same computing device that has the instructions to perform step S220, or on another different processor-readable storage medium. According to an optional embodiment of the present disclosure, the candidate image is generated in a first processing unit incorporating instructions for performing step S220, and then the processed image is located in a second processing unit. Transferred to a drug library (e.g., a drug library constructed by the method 100 described above). In order to compare the candidate image with the reference image of the medicine library, step S240 is executed under a machine learning command built in the storage medium, and the result (match or mismatch) is output to the user. If the comparison indicates that the candidate image matches or has the highest similarity to a particular reference image in the drug library, the information of the drug corresponding to that particular reference image is Is output. If the comparison fails to produce a match between the candidate image and all of the reference images in the library, a view indicating "no match" is output.

  As mentioned above, the purpose of this machine learning algorithm is to improve drug visibility. In some embodiments, the exemplary algorithm is a deep learning performed under any well-known visual object detection model, some of which are optimized for specific criteria according to practical needs. It can be an algorithm. In one exemplary embodiment, this deep learning is performed under an optimized detection model. In another exemplary embodiment, the candidate images processed by the method must be “full bleed images” and of a predetermined pixel size (eg, 448 × 224 pixels), thereby providing a machine learning operation Can be kept to a minimum level, which increases computing speed and efficiency.

  It should be noted that by juxtaposing the two processed images into one image named “combined image”, the machine learning system and / or method processes a vast amount of data for a large variety of drug packages. Even at times, it can be executed smoothly and at a higher speed. In other words, the processed and highlighted image actually increases the efficiency and accuracy of computing. Thanks to the above technical features, the present method 200, intended to identify a drug by a blister package, improves the accuracy of drug identification and greatly eliminates human error in the drug dispensing process, thereby Improve safety of use and quality of patient care.

  The subject matter described herein could be implemented using a non-transitory, tangible, processor-readable storage medium having stored processor-readable instructions. These instructions, when executed by a processor of the programmable device, control the programmable device to perform a method according to embodiments of the present disclosure. Exemplary processor-readable storage media suitable for implementing the subject matter described herein include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM , DVD, or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, and any other that can be used to store desired information and accessible by the processor Including media. Further, a processor-readable storage medium implementing the subject matter described herein may be located on a single device or computing platform, or may be distributed across multiple devices or computing platforms. In some embodiments, the computing platform is a system with embedded real-time computing constraints.

3. Dispensing Management System In another aspect of the presently described subject matter, a dispensing management system is provided. Referring to FIG. 3, a system 300 is shown that includes an image capture device 310, an image processor 320, and a machine learning processor 330, where the image capture device 310 and the machine learning processor 330 , Each coupled to an image processor 320. Image capture device 310 is configured to capture a plurality of images of a package of medication. In some embodiments, the image capture device 310 includes, in its structure, a transparent board 3101 and two image capture units 3102 separately disposed above and below each side of the transparent board. The board can be made of glass or an acrylate-based polymer. In practice, the medication is placed on a transparent board 3101 and images on both sides of the medication package are captured simultaneously by the two image capture units 3102. Typically, these two image capture units 3102 are real-time digital cameras.

  According to the present disclosure, unless otherwise indicated, the image processor 320 and the machine learning processor 330 each include a memory for storing a plurality of instructions for causing the processors to perform the method. In some embodiments, image processor 320 and machine learning processor 330 are separately located as two separate devices, but alternatively they may be located in the same hardware. In some embodiments, the image processor 320 and the machine learning processor 330 are communicably connected to each other. More specifically, the image processor 320 is communicatively connected to the image capture device 310 to receive the image captured by the image capture device 310 and receives the image (eg, steps S120 and S220) of the method. It is configured to perform a processing step, thereby generating a candidate image for further identification. The machine learning processor 330 is communicatively connected to the image processor 320 and is configured to perform an image comparison (eg, step S240) of the method for drug identification. The above steps include comparing the candidate image to a reference image in a drug library established by the method. This drug library is communicably connected to the machine learning processor 330. In the exemplary embodiment shown in FIG. 3, the drug library 3301 is stored in the machine learning processor 330. Alternatively or optionally, the drug library may be stored in a storage device connected to machine learning processor 330 by a cable connection or a wireless network.

  In some embodiments, the system 300 is a user interface configured to output a drug identification result, receive instructions from an external user, and return user input to the image processor 320 and the machine learning processor 330. (Not shown).

  Communication between the image capture device 310, the image processor 320, and the machine learning processor 330 can be embodied using various techniques. For example, the system 300 may include an image capture device 310 and an image capture device 310 via a network (such as a local area network (LAN), a wide area network (WAN), the Internet, or a wireless network). A network interface may be included to enable communication between the processor 320 and the machine learning processor 330. In another example, the system may have a system bus that links various system components, including the image capture device 310 to the image processor 320.

  The following examples are provided to illustrate certain aspects of the present invention and to assist one of ordinary skill in practicing the invention. These examples should in no way be considered as limiting the scope of the invention in any way. Even if not detailed, it is believed that one skilled in the art can, based on the description herein, utilize the present invention to its fullest extent. All publications cited herein are incorporated by reference in their entirety.

Example 1 Building a Drug Library Creating a Modified Two-sided Image (RTI) Over 250 drugs currently on the market were collected from a hospital pharmacy at McKay Memorial Hospital (Taipei, Taiwan). To build a database of drug libraries, we took as many pictures of the blister package of all drugs as possible. The images are cropped to minimize background noise, and multiple combined images for each drug (OpenCV2: Open Source Computer Vision 2) function programmed into the image processor using the Open Source Computer Vision Library 2 (OpenCV2) function. (Also referred to as "modified double-sided image (RTI)"). Each RTI was matched to a predetermined template (hereinafter a modified two-sided template RTT), which included both sides of one drug blister package. A total of 18,000 RTIs were obtained for a further deep learning process using the CNN model.

Strategies for Detecting Corners of Irregular Rectangular Drug Package In the case where the drug package to be detected has an irregular rectangular shape composed of three straight sides and one curved side, the curved side is determined by geometric estimation. The centroid algorithm (Table 1) was used to calculate undefined and undefined corners.

Referring to FIG. 4, this figure exemplarily shows how corner identification is performed on a medicine package having an irregular square shape. As depicted in FIG. 4, a blister package having three straight sides (L ₁ , L ₂ , L ₃ ) and one curved side (C ₁ ) is in a random orientation and the L ₁ and L ₂ intersection is referred to as _{P 1,} an intersection between _{L 2} and _{L 3} is referred to as _{P 4.} The goal is that the area enclosed by four points (P ₁ , P ₂ , P ₃ , and P ₄ ) and four sides (L ₁ , L ₂ , L ₃ , and C ₁ ) represents an image of the entire package. so as to encompass, was to determine the coordinate position of P ₂ and P _3. First calculated on the sides _{L 2,} the center point M between _{P 1} and _{P 4,} the algorithm cv2. The centroid B of the area of the blister package was calculated by the moment (OpenCV2). Next, the displacement vector ν was calculated based on the coordinate positions of the center (mi-point) point M and the centroid B. Finally, the coordinate position of _{P 2} and _{P 3,} respectively, were calculated as two times the position of the _{P 1} and _{P 4} distance [nu. By the above procedure, four points or corners of _{P 1, P 2, P 3} , and P ₄ is a way around clockwise or counterclockwise, it would be automatically oriented.

Optimize Machine Learning Ability In this example, the goal of machine learning was achieved by a convolutional neural network (CNN). For this, a minimal configuration version YOLOv2 (also known as Tiny YOLO) neural implemented by a graphics processing unit (GPU) was used to train the visualization of each RTI. The general concept of conventional YOLOv2 is to divide one image into multiple grid cells and use anchor boxes to predict bounding boxes. Briefly, RTI images, each having a 416 × 416 pixel size, were input to a network where the Tiny YOLOv2 neural generated an odd number of grid cells in each image. This results in only one grid cell at the center (ie, the center grid cell). Each grid cell and its neighboring grid cells were then analyzed to determine if they contained any characteristic information. For example, the input image can be divided into 13x13 grid cells, which can predict a 5-size bounding box, and use the reliability score returned from each grid cell to analyze the grid cells. It is determined whether or not the subject has been included. In this example, the input image was previously highlighted and contoured, thus optimizing the conventional Tiny YOLO, which led to increased identification efficiency and reduced work costs. Specifically, since the input image of each RTI has been processed to be a “full bleed image” and a predetermined pixel size, the number of anchor boxes for work is minimized to one, and The size of the box was set to a 7 × 7 grid cell. Thus, the machine learning network only needed to predict one full size bounding box of the input image. Detailed parameters for optimizing Tiny YOLO are listed in Table 2.

  In practice, the optimized Tiny YOLO was programmed into a machine learning processor with a high resolution graphics card, GEFORCE® GTX1080 (NVIDIA, USA). This machine learning processor performed a deep learning process with the optimized Tiny YOLO training model. The training criteria for this optimized Tiny YOLO are listed in Table 3.

Example 2 Evaluating the drug classification efficiency of a dispensing management system that implements machine learning based on the drug library of Example 1 Processed and combined images In order to verify the visual recognition efficiency for the RTI of the present disclosure, (No processing Both the raw image of the drug (uncombined in) and the RTI (processed and combined) were input into a training model for deep learning (optimized Tiny YOLO). The results of the training are summarized in Table 4. According to the data in Table 4, the "highlighted" images of the present disclosure were very efficient in visual training compared to the unprocessed raw images. The higher the training F1 score, the higher the efficiency of the deep learning network in viewing package images. Also, RTIs processed by RTT have significantly increased training efficiency compared to unprocessed images.

Optimization learning model Another comparison was made to verify the effectiveness of this training model. An RTI training model was constructed using two conventional deep learning models, ResNet101 and SE-ResNet101. Table 5 summarizes the results of these comparisons.

  Comparison of the results in Tables 4 and 5 demonstrates that RTIs processed by the present method to match RTT would improve training efficiency, especially when images were processed by the optimized Tiny YOLO model. This model significantly reduced training time. A total of 18,000 RTIs were entered into this training model, with the drug library of the present disclosure being built by running an optimized Tiny YOLO model. The drug library and deep learning network were further stored in a built-in system for future applications.

Applications for Real-Time Drug Identification In practice, random drugs were selected and placed in a designed chamber with a clear board made of glass. For illumination, a light source is placed around the transparent board and two BRIO webcams (Logitech, USA) are positioned above and below the board to ensure that their field of view covers the entire area of the transparent board. Placed separately. On the other hand, the built drug library was pre-stored on a real-time embedded computing device, JETSON ™ TX2 (NVIDIA, USA). The JETSON (TM) TX2, referred to as a "Developer Kit", includes memory, CPU, GPU, USB port, web antenna, and some other computing elements, thereby enabling image processing in the same device. And machine learning steps could be performed. In operation, the webcam was connected to the JETSON ™ TX2 via a USB cable such that images of both sides of the selected drug were simultaneously transferred in real time to the processor. The two raw images of the blister package of the selected drug were processed into one RTI, which was then subjected to a learning model Tiny YOLO programmed in JETSON ™ TX2 with a speed of up to about 200 FPS. The time cost for the entire procedure (ie, from image capture to visual identification) is about 6.23 FPS. The result of this identification will be presented in real time on an external display device, such as a computer screen or mobile user interface. Thus, an external user can obtain an identification result almost at the same time as the medicine is put into the chamber.

  In addition, another comparison was made to verify that the system reduced the likelihood of human error during the drug dispensing process. For example, the anxiolytic lorazepam is the drug most frequently misidentified at the time of dispensing due to high appearance similarity to other drugs. Initially, the RTI of lorazepam was entered into the system for comparison with all drugs in the library. After computing, the learning model raised a number of candidate agents that could be mis-identified based on the appearance similarities analyzed by the learning model. This list of candidates can help clinical staff double check that the selected drug is correct. Thus, the drug management system of the present disclosure can improve the accuracy of drug dispensing and minimize human error to ensure patient safety.

  In summary, a method and system for drug management according to the present disclosure achieves the foregoing objectives by applying image processing steps and deep learning models to combine images in a real-time manner. Can be. An advantage of the present disclosure is that not only can the raw images be processed efficiently, regardless of the orientation of the subject (ie, the drug), but it also increases the accuracy for their classification through the appearance of the drug.

  Of course, the foregoing description of the embodiments is by way of example only, and those skilled in the art will be able to make various modifications. The foregoing specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the invention. While various embodiments of the invention have been described with some specificity or with reference to one or more individual embodiments, those skilled in the art will depart from the spirit and scope of the invention. Rather, many modifications may be made to the disclosed embodiments.

Claims (8)

A computer-implemented method for building a drug library, comprising:
(A) receiving a plurality of raw images of a package of a drug in the form of a blister package;
(B) juxtaposing two of the plurality of raw images to generate a combined image including images on both sides of the blister package;
(C) processing the combined image to generate a reference image;
(D) establishing the drug library using the reference image;
Including
The step (b) includes:
(B-1) processing each of the plurality of raw images to generate a plurality of first processed images each having a defined contour;
(B-2) identifying the corner to determine the coordinate position of the corner of each defined contour of the first processed image;
(B-3) each of the first processed images of the step (b-1) based on the coordinate positions calculated in the step (b-2) to generate a plurality of second processed images; Rotating the
(B-4) combining any two of the second processed images to generate the combined image of step (b);
including,
Computer implementation method.   Each of the plurality of raw images in step (b-1) includes (i) a gray scale conversion process, (ii) a noise reduction process, (iii) an edge identification process, (iv) a convex hull process, and (v) a contour. The computer-implemented method according to claim 1, wherein the method is applied to a forming process.   The computer-implemented method according to claim 1, wherein the step (b-2) is performed by a line transformation algorithm or a centroid algorithm, and the step (c) is performed by a machine learning algorithm. A computer-implemented method for identifying a drug by a blister package of the drug, comprising:
(A) simultaneously obtaining front and back images of the blister package of the drug;
(B) juxtaposing the front and back images from step (a) to generate a candidate image;
(C) comparing the candidate image to a reference image of a drug library established by the method of claim 1;
(D) outputting the result of the step (c);
Including
The step (b) includes:
(B-1) processing each of the front and back images of step (a) to generate two first processed images, each having a defined contour;
(B-2) identifying the corners of each of the defined contours of the two first processed images to determine a coordinate position of the corners;
(B-3) the two second processed images processed in step (b-1) based on the coordinate positions calculated in step (b-2) to generate two second processed images. Rotating each of the processed images;
(B-4) combining the two second processed images of step (b-3) to generate the candidate image of step (b);
including,
Computer implementation method.   The images of the front surface and the back surface in the step (b-1) are respectively (i) a gray scale conversion process, (ii) a noise reduction process, (iii) an edge identification process, (iv) a convex hull process, and (v) The computer-implemented method according to claim 4, wherein the computer-implemented method is subjected to a contour forming process.   5. The computer-implemented method of claim 4, further comprising, prior to step (c), transferring the candidate image into the drug library. An image capture device configured to capture a plurality of images of the medication package;
An image processor programmed with instructions for performing a method for generating a candidate image, the method comprising:
(1) processing each of the plurality of images of the package of the medication to generate a plurality of first processed images each having a defined contour;
(2) identifying the corner to determine the coordinate position of the corner of each defined contour of the first processed image;
(3) rotating each of the first processed images based on the coordinate positions identified in step (2) to generate a plurality of second processed images;
(4) juxtaposing two of the second processed images in the step (3) to generate the candidate image by two different second processed images;
Comprising the image processor;
A machine learning processor programmed with instructions for performing a method for comparing the candidate image with a reference image of a drug library established by the method of claim 1;
And a dispensing management system. The image capture device,
A transparent board on which the drug is placed,
Two image capture units separately disposed above and below each side of the transparent board;
The system of claim 7, comprising:

Images