WO2014092189A1

WO2014092189A1 - Image recognition device, image recognition method, and image recognition program

Info

Publication number: WO2014092189A1
Application number: PCT/JP2013/083524
Authority: WO
Inventors: 博史川口; 康毅斎藤
Original assignee: 株式会社メディポリ; チームラボ株式会社
Priority date: 2012-12-14
Filing date: 2013-12-13
Publication date: 2014-06-19
Also published as: JP5414879B1; JP2014117390A

Abstract

The present invention increases the recognition accuracy of an input image and rapidly obtains recognition results of a recognition subject. The present invention is characterized by containing: a ranking unit (112) that extracts a plurality of local feature quantities from an input image, references a registered image database (102) in which local feature quantities extracted from each of a plurality of recognition subject images that can be recognized are associated with each of a plurality of recognition subjects, and are registered, associates the plurality of local feature quantities extracted from the input image with the closest local feature quantities among the registered local feature quantities, and ranks the plurality of recognition subjects in accordance with the number of associated local feature quantities; a rotation processing unit (113) that, on the basis of each extracted local feature quantity, determines conversion information for converting the recognition subject images; and a verification processing unit (114) that performs a conversion using the determined conversion information and determines whether or not the recognition subject image and the input image are the same.

Description

Image recognition apparatus, image recognition method, and image recognition program

The present invention relates to an image recognition apparatus, an image recognition method, and an image recognition program, and more particularly, to an input image recognition process.

調 Dispensing pharmacies that provide drugs according to doctors' prescriptions handle a wide variety of drugs. Since drugs are life-threatening and there should be no medication errors, it is necessary to provide accurate drugs according to the doctor's prescription from a wide variety of drugs. Among various types of medicines, for example, many medicines packed in blister packs are similar in appearance. Therefore, the confirmation work of the medicine at the time of provision is a heavy burden on the pharmacist.

In order to reduce such a burden, there is a method for recognizing a drug name from an image generated by reading an image of a packaging material directly packaging a drug and confirming whether an accurate drug is selected. It has been proposed (see, for example, Patent Document 1).

JP 2004-167158 A

In the technique disclosed in Patent Document 1, it is assumed that a drug name image part capable of recognizing a drug name from an image of a packaging material is collated. However, since a special font may be used for the medicine name printed on the medicine, it may be difficult to perform highly accurate character recognition.

In addition to the drug name, the image displayed on the drug packaging material may contain information effective for drug recognition. However, in the technique disclosed in Patent Document 1, such information is used. Cannot be used effectively. In addition, such a subject is not limited to the medicine packaged by the blister pack, but can be a problem as long as the medicine displays information capable of identifying the medicine on the packaging material.

In addition to the recognition of drugs, the accuracy of recognition accuracy and the speed of recognition processing are important issues in general image recognition processing for recognizing an object displayed in an input image by image processing. It is.

The present invention has been made in consideration of the above-described circumstances, and an object thereof is to improve the recognition accuracy of a recognition target based on an input image and to quickly obtain a recognition result of the recognition target.

In order to solve the above problems, one embodiment of the present invention is a drug recognition device that recognizes a drug based on a read image generated by imaging a package of the drug, and acquires the read image. And a registered image database in which a plurality of local feature amounts are extracted from the read image, and local feature amounts extracted for each of a plurality of medicine images that can be recognized are registered in association with each of the plurality of medicines. Referring to and associating a plurality of local feature quantities extracted from the read image with the nearest local feature quantity among the local feature quantities registered in the registered image database, and the plurality of local feature quantities according to the number of associated local feature quantities An ordering unit for ranking medicines, and transforming one of the medicine image and the read image so that the medicine image and the read image are superimposed Conversion information acquisition unit that obtains conversion information for obtaining based on local feature amounts extracted from the image of the medicine and the read image, respectively, and the image of the medicine and the read image using the obtained conversion information And a verification processing unit that determines whether the image of the medicine and the read image are the same by converting one of the medicine image and comparing the image of the medicine and the read image. The unit obtains the conversion information in the order of the ranks of the plurality of medicines ranked, and the verification processing unit converts one of the medicine image and the read image using the conversion information obtained in order. Determining whether the image of the medicine and the read image are the same, and outputting the medicine corresponding to the image of the medicine determined to be the same as the recognition result of the medicine for the read image. When That.

According to another aspect of the present invention, there is provided a drug recognition method for recognizing a drug based on a read image generated by imaging a package of a drug, wherein the read image is acquired, and a plurality of read images are obtained from the read image. A local feature amount is extracted, and a local feature amount extracted for each of a plurality of medicine images that can be recognized is extracted from the read image with reference to a registered image database registered in association with each of the plurality of medicines. A plurality of local feature quantities are associated with the nearest local feature quantity among the local feature quantities registered in the registered image database, the plurality of drugs are ranked according to the number of associated local feature quantities, and the drugs Conversion information for converting one of the medicine image and the read image so that the read image and the read image are overlapped with each other. The drug is obtained based on the local feature amount extracted from the image of the medicine and the read image in the order of the medicine, and the medicine is used by using the conversion information obtained in the order of the ranking of the ranked medicines. One of the image and the read image is converted and the image of the drug and the read image are compared to determine whether the image of the drug and the read image are the same. The medicine corresponding to the determined medicine image is output as a medicine recognition result for the read image.

According to still another aspect of the present invention, there is provided a medicine recognition program for recognizing a medicine based on a read image generated by imaging a medicine packaging, the step of acquiring the read image, and the read image Extracting a plurality of local feature amounts from the registered image database in which local feature amounts extracted for each of a plurality of medicine images that can be recognized are registered in association with each of the plurality of medicines, and Associating a plurality of local feature quantities extracted from the read image with the nearest local feature quantity among the local feature quantities registered in the registered image database; and the plurality of drugs according to the number of associated local feature quantities The medicine image and the read image so that the medicine image and the read image are superimposed. A step of obtaining conversion information for converting a method based on local features extracted from the image of the medicine and the read image in the order of the ranking of the plurality of medicines, respectively. Converting one of the medicine image and the read image using the conversion information obtained in the order of the plurality of medicines, and comparing the medicine image and the read image with the medicine image The information processing apparatus includes a step of determining whether or not the read image is the same, and a step of outputting a drug corresponding to the drug image determined to be the same as a drug recognition result for the read image. It is made to perform.

According to the present invention, the recognition accuracy of the recognition target based on the input image can be improved, and the recognition result of the recognition target can be obtained quickly.

It is a figure which shows the external appearance and internal structure of the chemical | medical agent recognition apparatus of this invention. It is a block diagram which shows the hardware constitutions of the chemical | medical agent recognition apparatus which concerns on embodiment of this invention. It is a figure which shows the function structure of the chemical | medical agent recognition apparatus which concerns on embodiment of this invention. It is a figure. It is a figure which shows the example of the registration image database which concerns on embodiment of this invention. It is a figure which shows the example of extraction of the registration image, reference image, and feature point which concern on embodiment of this invention. It is a figure which shows the example of the read image which concerns on embodiment of this invention. It is a figure which shows the example of matching of the feature point which concerns on embodiment of this invention. It is a figure which shows the example of matching of the feature point which concerns on embodiment of this invention. It is a figure which shows the example of the ranking result which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the rotation process part and verification process part which concern on embodiment of this invention. It is a flowchart which shows operation | movement of the rotation process part which concerns on embodiment of this invention. It is a figure which shows notionally the projection to the read image of the reference image which concerns on embodiment of this invention. It is a figure which shows the example of the read image which concerns on embodiment of this invention. It is a figure which shows the example of the similar image which concerns on embodiment of this invention. It is a figure which shows the example of the similar image database which concerns on embodiment of this invention. It is a figure which shows the example of the verification area | region coordinate which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, as an example of an image recognition device, a drug recognition device that uses an image obtained by imaging a medicine packaging as an input image, recognizes the type of the drug based on the image, and notifies the user of the input image. This will be described as an example.

FIG. 1 is a perspective view showing an appearance and an internal configuration of a medicine recognition apparatus 1 according to the present embodiment. As shown in FIG. 1, the drug recognition device 1 according to the present embodiment is configured by installing a touch panel 3 on a box-shaped housing 2. A part of the upper surface of the housing 2 is formed of a transparent plate, and the transparent portion serves as an imaging stand 4 on which a medicine to be imaged is placed.

A camera 6 for imaging a medicine placed on the imaging table 4 is installed inside the housing 2, and a ball-type illumination 5 is provided so as to face the imaging table 4 from the periphery of the camera 6. ing. Due to the effect of the ball-type illumination 5, the medicine placed on the imaging table 4 is irradiated with light from multiple directions, and an image of the medicine from which the shadow is eliminated is taken by the camera 6.

A controller device 7 is provided inside the housing 2, and an image captured and generated by the camera 6 is input to the controller device 7. The controller device 7 processes the medicine image acquired from the camera 6 to perform the medicine recognition process, and displays the recognition result on the touch panel 3.

FIG. 2 is a block diagram showing a hardware configuration of the medicine recognition apparatus 1 according to the present embodiment. As shown in FIG. 1, the medicine recognition apparatus 1 according to the present embodiment includes the above-described camera 6 in addition to the same configuration as an information processing terminal such as a general server or a PC (Personal Computer). That is, the drug recognition apparatus 1 according to the present embodiment includes a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, an HDD (Hard Disk Drive) 13 and an I / F 14. 17 is connected. The I / F 14 is connected to an image camera 6, an LCD (Liquid Crystal Display) 15, and an operation unit 16.

The CPU 10 is a calculation means and controls the operation of the entire medicine recognition apparatus 1. The RAM 11 is a volatile storage medium capable of reading and writing information at high speed, and is used as a work area when the CPU 10 processes information. The ROM 12 is a read-only nonvolatile storage medium, and stores programs such as firmware. The HDD 13 is a non-volatile storage medium that can read and write information, and stores an OS (Operating System), various control programs, application programs, and the like. The I / F 14 connects and controls the bus 18 and various hardware and networks.

The LCD 15 is a visual user interface for the operator of the apparatus to confirm the state of the medicine recognition apparatus 1. The operation unit 16 is a user interface for an operator to input information to the medicine recognition device 1. In the present embodiment, the LCD 15 and the operation unit 16 constitute the touch panel 3 shown in FIG.

In such a hardware configuration, a program stored in a recording medium such as the ROM 12, the HDD 14, or an optical disk (not shown) is read into the RAM 11, and the CPU 10 performs an operation according to the program, thereby configuring a software control unit. . A functional block that realizes the function of the medicine recognition apparatus 1 according to the present embodiment is configured by a combination of the software control unit configured as described above and hardware.

Next, the functional configuration of the controller device 7 of the drug recognition device 1 according to the present embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing a functional configuration of the controller device 7 according to the present embodiment. As shown in FIG. 3, the controller device 7 according to the present embodiment includes a registered image database 1-2 and an image processing unit 110 in addition to the camera driver 101 that drives the camera 6 and the display driver 103 that drives the LCD 15. Including.

Note that, as described above, the camera driver 101, the display driver 103, and the image processing unit 110 have the software control unit and hardware realized by the CPU 10 performing calculations according to the program read into the RAM 11. It works by. The image processing unit 110 acquires an image of the medicine placed on the imaging stand 4 taken by the camera 6 via the camera driver 101, and performs medicine recognition processing based on the image.

The registered image database 102 is an information storage unit that stores information related to an image of a medicine that can be placed on the imaging table 4 and can be recognized (hereinafter referred to as “registered image”). FIG. 4 is a diagram illustrating an example of information stored in the registered image database 102. As illustrated in FIG. 4, the registered image database 102 according to the present embodiment includes a “medicine ID” that identifies a drug that can be recognized, a “medicine name” that indicates the name of a drug that can be recognized, and a drug that can be recognized. This is information associated with “data path” information indicating a storage area in which an image of the package is stored.

As shown in FIG. 4, the “medicine name” according to the present embodiment includes a part of the drug name itself such as “ABC tablet” and a part of the amount of the drug such as “250 mg”. In the present embodiment, the “data path” is a data path indicating a storage area in the HDD 13 described with reference to FIG. 2, for example, but may be a storage area outside the medicine recognition apparatus 1 such as a network drive.

FIG. 5A is a diagram illustrating an example of a registered image according to the present embodiment. As shown in FIG. 5A, in a general medicine packaging, the name and quantity of the medicine are repeatedly displayed on one side. In the medicine recognition device 1 according to the present embodiment, an image of a surface on which such names and amounts of medicines are repeatedly displayed is registered in advance.

Fig. 5 (b) is an image obtained by extracting one portion of the display range of the drug name and quantity shown in Fig. 5 (a). In the medicine recognition apparatus 1 according to the present embodiment, in addition to the image of the whole medicine packaging shown in FIG. 5A, an image of a portion that can be a feature in medicine recognition as shown in FIG. Reference image ”is registered in advance.

As shown in FIG. 3, the image processing unit 110 according to the present embodiment includes an image acquisition unit 111, a ranking unit 112, a rotation processing unit 113, and a verification processing unit 114. The image acquisition unit 111 acquires an image (hereinafter, referred to as “read image”) that is captured and generated by the camera 6 via the camera driver 101. The ranking unit 112 performs a comparison process between the read image acquired by the image acquisition unit 111 and the registered image whose “data path” is registered in the registered image database 102, and ranks the registered images in an order similar to the read image. I do.

The rotation processing unit 113 considers that the medicine is read while being tilted or rotated in the read image, and aligns the direction of the registered image with the portion serving as a key for image comparison included in the read image. Performs image rotation processing. That is, the rotation processing unit 113 functions as a conversion information acquisition unit that obtains conversion information for converting either one of the read image and the registered image so as to overlap each other. The verification processing unit 114 compares the key portion of the read image rotated by the rotation processing unit 113 and the registered image in the order of the ranks generated by the ranking unit 112, and determines whether or not they are the same image. And outputs information to display the verification result.

In such a configuration, the gist of the present embodiment is to perform each process with high accuracy and high speed by using parameters related to the processes in the ranking unit 112 and the rotation processing unit 113. Hereinafter, processing according to the gist of the present embodiment will be described. In the operation of the medicine recognition apparatus 1 according to this embodiment, first, as described above, the operator places the medicine on the imaging table 4 and the medicine 6 is imaged by the camera 6. When the imaging by the camera 6 is executed, the camera 6 may detect and execute in real time that the medicine is placed on the imaging table 4, or may be executed by the operator operating the touch panel 3. .

FIG. 6A is a diagram illustrating an example of a read image generated by imaging a medicine placed on the imaging stand 4. The read image generated by the imaging by the camera 6 is in a state where the medicine is tilted as shown in FIG. 6A according to how the medicine is placed on the imaging stand 4 by the operator. The image acquisition unit 111 acquires the read image generated in this way via the camera driver 101.

When the image acquisition unit 111 acquires read images, the ranking unit 112 registers information in the registered image database 102 in the order of similarity to the read images based on the read images acquired by the image acquisition unit 111. Ranking registered images. The details of the processing by the ranking unit 112 is one of the gist according to the present embodiment. The ranking unit 112 according to the present embodiment uses the read image and the registered image as input, and performs ranking by nearest neighbor search processing based on local feature amounts.

The local feature amount extraction processing includes processing for extracting key points effective for recognition in an image and processing for generating feature amounts for each of the extracted key points. FIG. 6B is a diagram illustrating an example of a result of key point extraction based on the read image illustrated in FIG. Such key point extraction processing can be realized, for example, by extracting corner pixels using a simple corner detection filter. In that case, a predetermined scale can be used for the local scale. It is also possible to perform key point extraction using Fast-Hessian Detector.

When the key point extraction as shown in FIG. 6B is completed, the ranking unit 112 performs feature amount extraction based on the extracted image around the key point. In this feature amount extraction processing, SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), FRAK (Fast Retina Keypoint), or the like can be used. By this processing, feature amounts are extracted for each extracted key point as shown in FIG.

The content of the feature amount extracted by such processing differs depending on whether one of the above-described algorithms is used, but all is information calculated or extracted according to the content of the image. For example, when SIFT is used, a 128-dimensional feature amount can be extracted.

Note that the ranking unit 112 according to the present embodiment executes the above-described key point extraction and feature amount extraction processing in advance for an image whose information is registered in the registered image database 102. FIG. 5C is a diagram illustrating an example of the processing executed in advance. As shown in FIG. 5C, the ranking unit 112 according to the present embodiment extracts key points for the reference image shown in FIG. 5B among the images whose information is registered in the registered image database 102. The feature amount extraction is executed in advance, and the result is stored in the registered image database 102.

The local feature value f _q of the read image and the local feature value f _t of the reference image obtained in this way are _expressed by, for example, the following equations (1) and (2).

Here, p _q and p _t shown in equations (1) and (2) are the positions of the feature points, and are indicated by the coordinates of the pixels in the image, for example. Also, σ _q and σ _t are feature point scales, respectively. Further, d _q and d _t are descriptors indicating the features of the feature points, respectively.

When the local feature amounts are extracted from the read image and the reference image in this way, the ranking unit 112 uses the feature points extracted from the read image by the nearest neighbor search process as keys, and features corresponding to the feature points. Search for points in the reference image. In the nearest neighbor search, the ranking unit 112 sequentially refers to the feature points extracted from the read image, and selects d _t closest to d _q of the feature points from the information registered in the registered image database. Associate.

In FIG. 7, the feature points in the read image and the feature points in the reference image that are associated with each other are connected by a broken line. In FIG. 7, for ease of illustration, a broken line associated with a part of the feature point in the read image, specifically, one display part of the medicine name as one unit is shown. However, in practice, all the feature points included in the read image are associated with the feature points of any registered image whose information is registered in the registered image database by the nearest neighbor search.

That is, in the example of FIG. 7, each of the feature points of the part of the drug name displayed in the read image is associated with the feature point in the reference image, so that the plurality of feature points in the read image are referred to. It can happen that it is associated with one feature point in the image redundantly.

FIG. 7 shows a case where the drug in the read image is the same as the drug in the reference image. FIG. 8 is a diagram illustrating an example of feature points associated with different reference images, that is, different drugs, for the same read image as FIG. 7. As shown in FIG. 8, when the drug in the read image and the drug in the reference image are different, the feature amounts are naturally different, and the number of feature points is small even if they are associated with each other.

By such processing, the ranking unit 112 associates feature points by the nearest neighbor search with respect to one registered image for all registered images whose information is registered in the registered image database 102. In the nearest neighbor search, it is preferable to perform approximation for speeding up the processing. As a specific process of the approximate nearest neighbor search, for example, Hierarchical K-Means Tree or ANN (Approximate Nearest Neighbor) is used. Can do.

When the nearest neighbor search based on the read image is completed in this way, the ranking unit 112 counts the number of feature points associated with the feature points in the read image for each reference image as the number of votes, The number of votes for each reference image is obtained. In other words, the ranking unit 112 counts how many votes out of n votes each reference image obtains with the number n of feature points extracted from the read image as the total number of votes.

According to such a process, if the medicine is the same as that of the read image, it occupies a large number of votes, and if the image is similar to the read image, a certain number of votes is obtained. Become. The ranking unit 112 that performed such vote count processing performs ranking of reference images based on the number of votes counted for each reference image. FIG. 9 shows an example of the results of counting and ranking the number of votes.

In the present embodiment, the case where the feature points extracted from the read image are associated with the feature points extracted from the reference image will be described as an example. However, the ranking by the ranking unit 112 is not a reference image. The feature points extracted from the read image may be associated with the feature points extracted from the registered image shown in FIG.

In many cases, the medicine corresponding to the reference image that has obtained the first place by the processing of the ranking unit 112 represents an accurate recognition result. In the example of FIG. 9, the “ABC tablet whose medicine ID is“ 0001 ”is also used. “250 mg”, the correct recognition result has won first place. However, depending on the presence of a similar image, there may be a case where a feature point in an image of a different drug is associated by a nearest neighbor search, and an accurate recognition result is not ranked first.

Therefore, in the drug recognition device 1 according to the present embodiment, the ranking result verification process is executed by the rotation processing unit 113 and the verification processing unit 114. Therefore, the ranking unit 112 inputs the ranking result illustrated in FIG. 9, the read image and local feature amount extraction result, and the feature point association result to the rotation processing unit 113.

FIG. 10 is a flowchart showing the order of processing executed by the rotation processing unit 113 and the verification processing unit 114. As illustrated in FIG. 10, the rotation processing unit 113 that has acquired the above-described information from the ranking unit 112 selects reference images in the order of the ranking results illustrated in FIG. 9, and selects the selected reference images as corresponding feature points. A conversion matrix H for matching and projecting on the read image is obtained (S1001). The transformation matrix H can be obtained using, for example, RANSAC (RANdom Sampl Consensus). Processing for obtaining the transformation matrix H using RANSAC will be described below with reference to FIG.

As shown in FIG. 11, in the calculation process of the transformation matrix H, the rotation processing unit 113 first associates the feature points included in the reference image with the feature points included in the read image (S1101). In the processing of S1101, the rotation processing unit 113, using the _{f q-i} and _{f t-j} described above, the following equation (3), the _{f q-i} and _{f t-j} that satisfy the constraints shown in (4) Corresponding points C = {p _q−i , p _t−j } are obtained by associating the sets as corresponding points.

Here, the expression (3) is an expression for calculating the Hamming distance in the case of FREEK, and is an expression for calculating the Euclidean distance in the case of SIFT and SURF. Therefore, “T _d ” is set with respect to the Hamming distance or the Euclidean distance. That is, the threshold value is used to determine that the feature amount of the feature point in the reference image is close to the feature amount of the feature point in the read image. Further, since the reference image is stored at the same resolution as that of the image picked up by the camera 6, for example, “1” can be used as T _s indicating the threshold of the scale difference.

According to such an association process, as shown in FIG. 6, in the case of a read image in which a plurality of images identical to the reference image are displayed, a plurality of feature points in the read image are compared with one feature point in the reference image. It will be associated with the feature point. Even in such a case, the corresponding point C = {p _q−i , p _t−j } is obtained as a separate corresponding point. The processing for selecting one corresponding point from such corresponding points and obtaining the transformation matrix H is the subsequent processing.

It should be noted that the feature point association processing by the above formulas (3) and (4) can also be used in the feature point association by the ranking unit 112. In that case, the result of association by the ranking unit 112 can be adopted as the result of association in S1101. In particular, for a reference image that is ranked first, it is predicted that many feature points are associated with each other. Therefore, feature points are associated again based on the above formulas (3) and (4). Even if it is performed, it is predicted that many feature points have the same correspondence result. Therefore, by adopting the feature point association result by the ranking unit 112 as the association result in S1101, it is possible to reduce the amount of processing and obtain a result quickly without degrading the processing accuracy.

The rotation processing unit 113 that has completed the processing of S1101 next randomly selects one corresponding point associated in S1101 (S1102), and is within a predetermined range on the read image side with the selected corresponding point as the center. Two other corresponding points are randomly selected from the reference image side (S1103). This predetermined range is, for example, a range in which the diagonal line of the reference image shown in FIG.

When a total of three corresponding points are acquired from the predetermined range on the read image side, the rotation processing unit 113 performs the reference image side according to the affine transformation based on the positions {p _q−i , p _t−j } of the corresponding points. A transformation matrix H for projecting the feature points to the read image side is obtained (S1104). Since the number of parameters in the affine transformation is 6, it is possible to obtain the transformation matrix H according to the affine transformation using two parameters included in each of the three feature points.

When the transformation matrix H is calculated, the rotation processing unit 113 projects the feature points in the reference image into the read image using the calculated H (S1105), and the feature points of the reference image projected in the read image; The position difference from the corresponding feature point of the read image, that is, the number of corresponding points whose distance between corresponding feature points is within a predetermined threshold value is counted as Inlier (S1106). The predetermined threshold value can be set by the number of pixels, for example, and a relatively small value such as a few pixels to a dozen pixels is set according to the resolution of the reference image.

The rotation processing unit 113 repeatedly executes the processing from S1102 in various corresponding point selection states (S1107 / NO). When the specified number of repetitions is completed (S1107 / YES), each corresponding point is selected. The count number of Inlier counted in the state is compared, and the conversion matrix H obtained when the count number is the largest is determined as the final conversion matrix H (S1108), and the process is terminated. In the process of S1108, the Inlier count may be the same in the selection state of a plurality of corresponding points. In such a case, any one may be selected.

If the drug in the read image is different from the drug in the reference image, it may be impossible to calculate the transformation matrix H that matches the affine transformation conditions in S1104. In that case, the rotation processing unit 113 determines that any of the corresponding corresponding points being selected is incorrect, and returns to S1102 to select another feature point.

Further, in addition to the case where the conversion matrix H obtained when the number of counts is the largest is determined as the final conversion matrix H, based on the positional relationship between corresponding points of the Inlier when the number of counts is the largest, The transformation matrix H that minimizes the difference in position for each corresponding point may be recalculated. Such a method is called DLT (Direct Linear Transform).

Further, in S1104 of FIG. 11, the case where affine transformation is used has been described as an example. However, as shown in FIG. 1, in the medicine recognition device 1 according to the present embodiment, the medicine placed on the imaging stand 4 is imaged. Therefore, image distortion that requires affine transformation hardly occurs. Therefore, it is possible to use Euclidean transformation instead of affine transformation. In this case, the Euclidean transformation can be calculated by selecting two feature points instead of three points, and the processing load can be reduced and the time required to obtain a drug recognition result can be shortened compared to the case of affine transformation. .

10, when the rotation processing unit 113 obtains the transformation matrix H by such processing, the rotation processing unit 113 inputs the obtained transformation matrix H to the verification processing unit 114. The verification processing unit 114 projects the selected reference image onto the read image based on the conversion matrix H acquired from the rotation processing unit 113 in this way. By projecting the reference image onto the read image, a range corresponding to the reference image in the read image can be extracted based on the outer frame of the reference image, as shown in FIG.

The verification processing unit 114 performs an image in a range corresponding to the reference image in the read image, that is, an image of a characteristic part for identifying a drug such as “ABC tablet 250 mg” (hereinafter, “ Whether or not the two images are the same by comparing the extracted medicine display unit image with the reference image converted by the conversion matrix H. That is, the accuracy of the reference image ranked higher by the ranking unit 112 as being similar to the read image is verified (S1002).

The comparison processing of the image by the verification processing unit 114 is performed by, for example, calculating the similarity of the shape by normalized correlation or calculating the similarity by comparing the color histograms generated by the HSV (Hue, Saturation, Value) system. This can be realized by making a threshold judgment on the determined value. In addition, verification accuracy can be improved by using a combination of the above-described threshold determination for the similarity of the shape and the similarity of the color.

The rotation processing unit 113 and the verification processing unit 114 according to the present embodiment perform conversion matrix H calculation processing and verification processing in the order of the generated order as shown in FIG. When the verification processing unit 114 determines that the reference image being determined is a read image, that is, when the verification is passed (S1003 / YES), the determination process is terminated at that time, and the determination result is displayed. By generating information and outputting it to the display driver 103, the judgment result, that is, the recognition result of the medicine placed on the imaging stand 4 is displayed on the LCD 15.

On the other hand, if the verification processing unit 114 determines that the reference image being determined is not a read image (S1003 / NO), the verification processing unit 114 notifies the rotation processing unit 113 of the determination result. As a result, the rotation processing unit 113 executes the process described with reference to FIG. 11 for the reference image having the next highest order in the order shown in FIG. As a result, the verification processing unit 114 performs the verification process on the next highest-reference image. By repeating such processing in the order of ranking as shown in FIG. 9, it is possible to obtain accurate drug recognition results.

In the medicine recognition apparatus 1 according to the present embodiment, after ranking the reference images based on the local feature amounts as described above, a detailed comparison inspection is performed by the verification processing unit 114 according to the ranking. Therefore, it is possible to improve the accuracy of drug recognition through detailed comparison tests, and since ranking is performed using local features in advance, comparisons are made in order from the reference images that are most likely to be accurate. Since the drug recognition result is confirmed when the accuracy is confirmed, it is not necessary to carry out detailed comparison inspections for many reference images, and the recognition result of the drug can be obtained quickly by reducing the processing load. It is possible.

Further, in the medicine recognition apparatus 1 according to the present embodiment, in order to enable a high-precision comparison inspection by the verification processing unit 114, the conversion matrix H is calculated by the rotation processing unit 113 to correct the inclination of the read image. Thus, a portion corresponding to the reference image in the read image, that is, a portion to be subjected to the high-precision comparison inspection is extracted from the read image.

In the calculation process of the transformation matrix H by the rotation processing unit 113 and the extraction process of the comparison target from the read image, the matching result of the local feature amount obtained by the ranking unit 112 is used. It is possible to link the processing of the rotation processing unit 113 with each other, realize efficient processing, and contribute to the acquisition of the rapid drug recognition result described above.

As described above, according to the medicine recognition device 1 according to the present embodiment, it is possible to improve the medicine recognition accuracy based on an image obtained by imaging the packaging material and to quickly obtain the medicine recognition result. . Moreover, in the said embodiment, although the case where the image displayed on the packaging of a medicine was character information like "ABC tablet 250 mg" was demonstrated as an example, the method which concerns on this embodiment is not restricted to character information. Applicable and widely applicable to drugs in various packaging forms.

Furthermore, in this embodiment, although the apparatus which recognizes the medicine image | photographed based on the image obtained by imaging the packaging of a medicine was demonstrated as an example, it is not restricted to such an aspect, The input image Can be widely used as a technique for recognizing a recognition object displayed in an image.

In the above embodiment, the display on the LCD 15 has been described as an example as a result of the drug recognition process. However, this is only an example, and for example, the drug name of the recognized drug may be read out by voice. If there is information indicating a medicine to be provided, such as prescription information, whether the medicine is correctly selected by comparing the recognition result by the verification processing unit 114 with the information indicating the medicine to be provided. It is also possible to notify the pharmacist by judging.

In the process described with reference to FIG. 11, in S1108, since the transformation matrix H when the Inlier count is the highest is adopted, a plurality of medicines displayed in the read image as shown in FIG. The case where one of the display unit images is extracted and compared is described as an example. However, in S1102, if the corresponding points are selected evenly from the entire read image, a conversion matrix H corresponding to all of the plurality of medicine display unit images displayed in the read image as shown in FIG. Can be obtained.

If the conversion matrix H corresponding to all of the plurality of medicine display unit images displayed in the read image can be obtained, the number of medicine display unit images included in the read image is determined based on the number of the conversion matrix H. can do. Depending on the medicine, since the number of medicine display unit images displayed on one packaging material is determined in advance, the number of medicine display unit images determined in this way can be used in medicine recognition. Such a case can be realized by registering the number of medicine display unit images for each medicine in the registered image database 102 described in FIG.

Further, depending on the prescription of the medicine, as shown in FIG. 13, the medicine may be prescribed in a divided state instead of one package of the blister pack. In such a case, it is also possible to determine the amount of the prescribed medicine by judging the number of medicine display unit images included in the read image as described above. This makes it possible to determine whether or not the amount of medicine actually provided matches the doctor's prescription.

Note that the number of medicine display unit images displayed in the medicine packaging does not always correspond to the amount of medicine, for example, the number of tablets. In that case, the number of medicine display unit images included in one package of the blister pack is registered in the registered image database 102 for each medicine, and based on the ratio to the number of medicine display unit images determined from the read image. The amount of medicine corresponding to the number of recognized medicine display unit images may be registered in the registered image database 102 for each medicine.

In the above-described embodiment, the case where the verification is terminated by the process such as the normalized correlation in the verification processing unit 114 has been described as an example. In addition, a combination of medicines whose reference images are very similar may be stored in a database in advance, and when a medicine registered in the database is recognized, further detailed verification may be performed.

As shown in FIG. 14, the type of medicine may be the same and the amount may be different. In the case of the example in FIG. 14, only the “2” portion of “250 mg” and the “1” portion of “150 mg” are different as images. Even if the read image is “250 mg”, If “150 mg” is ranked first by the ranking unit 112, the verification processing by the verification processing unit 114 may pass verification.

FIG. 15 is a diagram showing an example of a database (hereinafter referred to as “similar drug database”) in which similar drugs are registered. As shown in FIG. 15, in the similar drug database, for example, drug IDs of drugs with similar reference images are registered in association with each other, and the reference image of the drug ID is verified by the verification processing unit 114. Is passed, the information of coordinates indicating the area in the reference image to be further verified is associated as “verification area coordinates”.

FIG. 16 is a diagram illustrating an example of the coordinate range specified by the verification area coordinates. As indicated by a broken line in FIG. 16, a different part in a set of similar images is specified as a verification region. Thereby, when the medicine ID corresponding to the reference image that has passed the verification pass in S1003 in FIG. 10 is registered in the similar medicine database, the verification processing unit 114 performs the verification process again on the verification region coordinates associated with the medicine ID. That is, the shape similarity is calculated by the above-described normal correlation, and the similarity is calculated by comparing color histograms generated by the HSV system. Thereby, it is possible to improve the recognition accuracy for similar images.

Further, when a verification error occurs in the re-verification of such a similar image, the verification is not repeated by returning to S1001 of FIG. 10, but the drug ID associated with the similar drug as shown in FIG. By recognizing the corresponding drug, it is possible to output the recognition result quickly without performing another verification process.

In addition, even when the recognition process is performed again, it is considered that the transformation matrix H obtained from the reference image that has been verified as a similar image can be used as it is. Therefore, the process of S1001 in FIG. 11 may be omitted and only the verification process using normalized correlation or the like may be performed. When the transformation matrix H obtained in the reference image in error is used as it is, there is a possibility that a positional deviation occurs between the read image and the reference image in the verification process with the similar image. This misregistration can be absorbed in the normalized correlation processing, and the amount of processing can be reduced by such processing, and the recognition result can be obtained quickly.

In the above embodiment, the case has been described as an example where, after ranking by the ranking unit 112, the verification processing by the verification processing unit 114 is always performed to ensure the accuracy of the recognition result. However, in the result of ranking as shown in FIG. 9, if it can be clearly determined that the first place is correct based on the difference between the first place and the second place, the rotation processing unit 113 and the verification processing unit 114. The first result shown in FIG. 9 may be output as the recognition result.

In such a case, the ranking unit 112 determines that the medicine corresponding to the first medicine ID is correct if, for example, the number of votes in the second place is 1% or less of the number of votes in the first place as a result of the ranking shown in FIG. By determining that the result is a recognition result and generating information for displaying the recognition result in place of the verification processing unit 114 and outputting the information to the display driver 103, the determination result on the LCD 15, that is, the image is placed on the imaging stand 4. The recognition result of the selected drug is displayed.

In the present embodiment, the case where information for superimposing the reference image on the read image is calculated as the transformation matrix H has been described as an example. This is an example, and information for superimposing the read image on the reference image may be calculated.

DESCRIPTION OF SYMBOLS 1 Drug | medical agent recognition apparatus 2 Case 3 Touch panel 4 Imaging stand 5 Ball type illumination 6 Camera 7 Controller apparatus 10 CPU
11 RAM
12 ROM
13 HDD
14 I / F
15 LCD
16 Operation Unit 17 Bus 101 Camera Driver 102 Registered Image Database 103 Display Driver 110 Image Processing Unit 111 Image Acquisition Unit 112 Ranking Unit 113 Rotation Processing Unit 114 Verification Processing Unit

Claims

An image recognition device for recognizing a recognition target displayed in an image based on an input image,
An image acquisition unit for acquiring the input image;
A registered image database in which a plurality of local feature amounts are extracted from the input image, and local feature amounts extracted for each of a plurality of recognition target images that are recognized are associated with each of the plurality of recognition target images and registered. , The plurality of local feature quantities extracted from the input image are associated with the nearest local feature quantity among the local feature quantities registered in the registered image database, and the local feature quantity is associated with the number of associated local feature quantities. A ranking unit for ranking a plurality of recognition target images;
Conversion information for converting one of the recognition target image and the input image is extracted from the recognition target image and the input image so that the recognition target image and the input image are superimposed. Conversion information acquisition unit to be obtained based on the local feature amount,
One of the recognition target image and the input image is converted using the obtained conversion information, and the recognition target image and the input image are compared by comparing the recognition target image and the input image. A verification processing unit that determines whether or not they are the same,
The conversion information acquisition unit obtains the conversion information in the order of the ranks of the plurality of recognition targets that are ranked,
The verification processing unit converts one of the recognition target image and the input image using conversion information obtained in order, and determines whether the recognition target image and the input image are the same. And outputting a recognition target corresponding to the recognition target image determined to be the same as a recognition target recognition result for the input image.
The image recognition apparatus according to claim 1, wherein the conversion information acquisition unit obtains the conversion information based on a result of association of the local feature amounts by the ranking unit.
The registered image database stores, for each of the plurality of recognition targets, an image of a portion that can be a characteristic in recognition of the recognition target among the recognition target images in association with each other.
The conversion information acquisition unit obtains the conversion information so that an image of a part that can be the feature is superimposed on a corresponding part in the input image,
The verification processing unit converts an image of a portion that can be a feature using the obtained conversion information, and becomes an image of a portion of the input image in which the image of a portion that can be the feature is superimposed and the feature. The image recognition apparatus according to claim 1, wherein an image of a portion to be obtained is compared.
The conversion information acquiring unit, when a plurality of images of the portion that can be the feature are included in the input image, the image of the portion of the image that can be the feature included in the input image based on the number of conversion information obtained. Determine the number,
The said verification process part determines the quantity of the recognition target imaged at the time of the production | generation of the said input image based on the number of the images of the part which can become the said characteristic contained in the said input image. Image recognition device.
The registered image database stores, in association with each of the plurality of recognition objects, the number of images of the portion that can be the feature.
The conversion information acquiring unit, when a plurality of images of the portion that can be the feature are included in the input image, the image of the portion of the image that can be the feature included in the input image based on the number of conversion information obtained. Determine the number,
The verification processing unit compares the number of images of the portion that can be the feature included in the input image with the number of images of the portion that can be the feature stored in the registered image database, and The image recognition apparatus according to claim 3, wherein it is determined whether an image to be recognized and the input image are the same.
The conversion information acquisition unit associates and associates a local feature amount extracted from the recognition target image with a local feature amount whose difference is within a predetermined range among the local feature amounts extracted from the input image. The image according to claim 1, wherein a plurality of local feature values are selected, and conversion information is calculated so that the selected local feature values are projected from one of the recognition target image and the input image to the other. Recognition device.
The conversion information acquisition unit calculates conversion information such that a selected local feature is projected from one of the recognition target image and the input image to the other according to affine transformation or Euclidean transformation. 6. The image recognition apparatus according to 6.
The verification processing unit refers to a similar image database in which a set of similar images is registered among a plurality of recognition target images, and the recognition target images determined to be the same are registered. The image recognition apparatus according to claim 1, wherein reverification is performed in the case where it has been performed.
The similar image database stores a set of similar images in association with information indicating portions of different images in the set of similar images,
When the recognition target images determined to be the same are registered in the similar image database, the verification processing unit compares the recognition target image and the input image for the different image portions. The image recognition apparatus according to claim 8, wherein re-verification is performed.
An image recognition method for recognizing a recognition target displayed in an image based on an input image,
Obtaining the input image;
Extracting a plurality of local features from the input image;
A plurality of local features extracted from the input image with reference to a registered image database in which local feature amounts extracted for each of a plurality of recognition target images that can be recognized are associated with each of the plurality of recognition targets. Associating the amount with the closest local feature amount among the local feature amounts registered in the registered image database;
Ranking the plurality of recognition objects according to the number of associated local features,
Conversion information for converting one of the recognition target image and the input image so that the recognition target image and the input image are overlaid in order of the ranking of the plurality of recognition targets. , Obtained based on local features extracted from the recognition target image and the input image, respectively,
One of the recognition target image and the input image is converted using the conversion information obtained in the order of the ranking of the plurality of recognized recognition targets, and the image using the recognition field is compared with the input image. To determine whether the image to be recognized and the input image are the same,
An image recognition method comprising: outputting a recognition target corresponding to the recognition target image determined to be the same as a recognition target recognition result for the input image.
An image recognition program for recognizing a recognition target displayed in an image based on an input image,
Obtaining the input image;
Extracting a plurality of local features from the input image;
A plurality of local features extracted from the input image with reference to a registered image database in which local feature amounts extracted for each of a plurality of recognition target images that can be recognized are associated with each of the plurality of recognition targets. Associating a quantity with the closest local feature quantity among the local feature quantities registered in the registered image database;
Ranking the plurality of recognition objects according to the number of associated local feature quantities;
Conversion information for converting one of the recognition target image and the input image so that the recognition target image and the input image are overlapped with each other in the order of the ranking of the plurality of recognition targets. Determining based on local features extracted from the image to be recognized and the input image, respectively,
One of the recognition target image and the input image is converted using the conversion information obtained in the order of the ranking of the plurality of recognized recognition targets, and the recognition target image and the input image are compared. Determining whether the recognition target image and the input image are the same,
An image recognition program that causes an information processing apparatus to execute a step of outputting a recognition target corresponding to the recognition target image determined to be the same as a recognition target recognition result for the input image.