JP7170602B2 - LEARNING DATA CREATION DEVICE AND METHOD, AND MACHINE LEARNING DEVICE AND METHOD - Google Patents

Description

The present invention relates to a learning data creation device and method, and a machine learning device and method, and more particularly to a technology for efficiently creating learning data for machine learning of a learner.

2. Description of the Related Art Conventionally, as an image output device for creating learning data, there is one disclosed in Japanese Unexamined Patent Application Publication No. 2002-200013.

The image output device described in Patent Document 1 acquires a first image of an object photographed from a first direction and a second image of an object photographed from a second direction, and a second image in one image in a predetermined positional relationship.

The image output device described in Patent Document 1 captures an object from multiple directions by using the synthesized image generated as described above as input data for a learner that can be machine-learned or obtained as a learning result. It enables machine learning using the information of the obtained image, or obtains the evaluation result of the target object.

JP 2018-132962 A

The image output device in Patent Document 1 creates a synthetic image as input data to a learning device, but does not create "output correct data used for learning" when performing "supervised learning".

When performing "supervised learning", it is necessary to prepare learning data consisting of a set of "input data used for learning" and "output correct data used for learning". A method for creating correct data is not described. In general, when creating learning data, it takes time and effort to create correct data, but the invention described in Patent Document 1 does not solve such a problem.

The present invention has been made in view of such circumstances. It is an object of the present invention to provide a learning device and method.

In order to achieve the above object, a learning data creation apparatus according to one aspect of the present invention includes: a second image obtaining unit for obtaining a second image by photographing with a second photographing method different from the first image and the second image; and a control unit.

According to one aspect of the present invention, for example, the first image acquired by the first image acquisition unit is used as input data to the learning device, and the second image acquired by the second image acquisition unit is used as correct data for the learning device. can do. Thereby, a set of the first image and the second image can be efficiently created.

In the learning data creation device according to another aspect of the present invention, the second image acquisition unit includes an image processing unit that processes the image captured by the second image capturing method to generate the second image, and the storage control unit Preferably, the first image is used as input data for the learning device, and the second image is stored as correct data for the learning device in the storage unit.

In the learning data creation device according to still another aspect of the present invention, the images captured by the second capturing method are a plurality of images with different incident directions of illumination on the subject, and the image processing unit converts the plurality of images into It is preferable to generate a second image in which the stamp or print added to the subject is emphasized. It is preferable to generate a plurality of edge images in which the engraving or print is emphasized, and generate a composite image as the second image by synthesizing the generated plurality of edge images.

In the learning data creation device according to still another aspect of the present invention, the image processing unit uses a plurality of images as input images, and outputs an image in which the stamp or print added to the subject is emphasized as a recognition result. and inputting a plurality of images to the recognizer and generating the recognition result of the recognizer as the second image.

In the learning data creation device according to still another aspect of the present invention, it is preferable that the subject is a drug to which a stamp or print is added.

In the learning data creation device according to still another aspect of the present invention, it is preferable that the first imaging method and the second imaging method use different illumination methods for the subject.

In the learning data creation device according to still another aspect of the present invention, the lighting method of the first photographing method causes light to enter the subject from all directions, and the lighting method of the second photographing method causes light to enter the subject from all directions. It is preferable to sequentially enter a plurality of different lights.

In the learning data creation device according to still another aspect of the present invention, the first image acquisition unit and the second image acquisition unit control lighting of the plurality of illumination light sources arranged around the subject and the illumination of the plurality of illumination light sources. Preferably, the illumination control unit simultaneously turns on the plurality of illumination light sources as the illumination method of the first imaging method, and sequentially turns on the plurality of illumination light sources as the illumination method of the second imaging method.

In the learning data creation device according to still another aspect of the present invention, it is preferable that the plurality of illumination light sources are arranged at least two of the front, rear, right and left sides of the subject.

In the learning data creation device according to still another aspect of the present invention, the first image acquisition unit and the second image acquisition unit include a camera unit that captures an image of a subject, a camera control unit that controls the camera unit, and an image processing unit. The camera control unit causes the camera unit to photograph the subject when the subject is illuminated by the illumination method of the first photographing method, and sequentially illuminates the subject when the subject is sequentially illuminated by the illumination method of the second photographing method. Preferably, the subject is photographed each time the subject is illuminated, and the image processing section processes a plurality of images photographed by the camera section each time the illumination is sequentially performed to generate the second image.

In the learning data creation device according to still another aspect of the present invention, the subject is stamped or printed, and the image processing unit performs image processing for emphasizing the stamp or print added to the subject. It is preferable to generate a second image that emphasizes the .

In the learning data creation device according to still another aspect of the present invention, the first image acquisition unit and the second image acquisition unit are a plurality of illumination light sources arranged around the subject, each having a plurality of different wavelength bands. A first photographing method includes a plurality of illumination light sources for irradiating a subject with light, and a camera section having a plurality of types of pixels each having sensitivity to a plurality of lights having different wavelength bands. In the second imaging method, the subject illuminated by a plurality of illumination light sources is photographed by the camera section, and a plurality of images corresponding to a plurality of types of pixels are obtained. It is preferable that the second image acquisition unit acquires the second image by performing image processing on the plurality of images.

In the learning data creation device according to still another aspect of the present invention, the subject is stamped, and the second image acquisition section includes a three-dimensional information acquisition section for acquiring three-dimensional information of the subject, and an image processing section. Preferably, the image processing section extracts the marking of the subject based on the three-dimensional information of the subject obtained by the three-dimensional information obtaining section, and generates a second image in which the marking of the subject is emphasized.

In the learning data creation device according to still another aspect of the present invention, it is preferable that the first imaging method and the second imaging method are imaging methods for imaging a subject using different imaging modalities.

In the learning data creation device according to still another aspect of the present invention, one of the first imaging method and the second imaging method is an imaging method for imaging a subject using a CT apparatus, and the first The other imaging method of the imaging method and the second imaging method is preferably an imaging method of imaging a subject using an MRI apparatus.

A machine learning device according to still another aspect of the present invention uses a learning device and learning data stored in a storage unit of the learning data generation device to machine-learn the first images forming a set. a learning control unit that causes the learning device to perform machine learning using the second image that constitutes the set as the input image in the case of causing the learning device to perform machine learning.

A machine learning device according to still another aspect of the present invention includes a first image acquiring unit that acquires a first image by photographing a subject with a first photographing method, and a second photographing method that differs from the first photographing method. , a learning device, and a learning control unit that causes the learning device to perform machine learning on the basis of a set of the first image and the second image.

A learning data creation method according to still another aspect of the present invention includes a step in which a first image obtaining unit obtains a first image by photographing a subject using a first photographing method; acquiring a second image by photographing with a second photographing method different from the first photographing method; and storing the first image and the second image as a set by the memory control unit as learning data for the learning device. and storing in the unit.

In the learning data creation method according to still another aspect of the present invention, it is preferable that the subject is a drug with a stamp or print added.

In the learning data creation method according to still another aspect of the present invention, it is preferable that the first imaging method and the second imaging method use different lighting methods for the subject.

In a machine learning method according to still another aspect of the present invention, the learning control unit uses the learning data created by the learning data creating method and stored in the storage unit to learn the first images forming the set. a step of machine-learning the learner by using the second image forming the pair as the correct data, with the second image as the input image for machine learning of the learner.

A machine learning method according to still another aspect of the present invention includes a step in which a first image obtaining section obtains a first image by photographing a subject with a first photographing method; a step of acquiring a second image by photographing with a second photographing method different from the first photographing method; include.

According to the present invention, it is possible to efficiently and automatically create learning data for “supervised learning” of a learning device.

FIG. 1 is a block diagram showing a schematic configuration of a machine learning device according to the present invention. FIG. 2 is a block diagram of main parts of the first embodiment of the learning data creation device shown in FIG. FIG. 3 is a side view showing how an image of the sachet is acquired using the camera section and the illumination light source. FIG. 4 is a plan view showing how an image of the sachet is acquired using the camera section and the illumination light source. FIG. 5 is a diagram showing five images of one medicine in a sachet with different lighting directions. FIG. 6 is a block diagram mainly showing a specific configuration of the image processing section shown in FIG. FIG. 7 is a diagram showing an example of a drug image obtained by cutting out one drug region from the image of the sachet. FIG. 8 is a schematic diagram of the cross-sectional structure of the drug taken along the xy plane passing through the center of the drug. FIG. 9 is a diagram showing an example of a Sobel filter used for edge extraction in the edge image generator. FIG. 10 is an image diagram of a plurality of sets of learning data sets of the first image and the second image stored in the storage unit. FIG. 11 is a diagram showing the main part of the second embodiment of the learning data creation device shown in FIG. FIG. 12 is a diagram showing the spectral sensitivity of the color image sensor of the camera section used in the second embodiment of the learning data creation device shown in FIG. FIG. 13 is a diagram showing the main part of the third embodiment of the learning data creation device shown in FIG. FIG. 14 is a flow chart illustrating an embodiment of a machine learning method according to the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a learning data creation device and method and a machine learning device and method according to the present invention will be described below with reference to the accompanying drawings.

[Configuration of machine learning device]
FIG. 1 is a block diagram showing a schematic configuration of a machine learning device according to the present invention.

The machine learning device 10 shown in FIG. 1 includes a learning data creation device 10A and a learning device 50. The learning device 50 performs machine learning using a learning data set created by the learning data creation device 10A.

The learning data generation device 10A is composed of an image acquisition section 20 including a first image acquisition section 20A and a second image acquisition section 20B, and a storage control section 40. FIG.

The first image acquisition section 20A is a section that acquires a first image 25 by photographing a subject with a first photographing method, and the second image acquisition section 20B photographs the same subject with a second photographing method and obtains a second image. This is the part where the image 27 is acquired. Details of the first imaging method by the first image acquisition unit 20A and the second imaging method by the second image acquisition unit 20B will be described later.

The first image 25 acquired by the first image acquisition section 20A and the second image 27 acquired by the second image acquisition section 20B are output to the storage control section 40, respectively.

The storage control unit 40 sets the input first image 25 and the second image 27 as a set and causes the storage unit 42 to store the set. The storage unit 42 may be a medium such as a hard disk device built into the learning data creation device 10A, or may be a removable medium that is detachable from the learning data creation device 10A.

Here, the set of the first image 25 and the second image 27 is one of the first image 25 and the second image 27 (in this example, the first image 25 ) can be used as input data used for learning by the learner 50 , and the other image (in this example, the second image 27 ) can be used as correct data used for learning by the learner 50 .

The storage control unit 40 acquires the first image 25 and the second image 27 for each subject of the plurality of subjects, and stores a plurality of sets of learning data sets of the first image 25 and the second image 27. stored in the unit 42;

The learning control unit 52 of the learning device 50 can cause the learning device 50 to perform machine learning based on the learning data set stored in the storage unit 42 .

As the learning device 50 that uses the first image as an input image and outputs the second image as a recognition result, a convolutional neural network (CNN: Convolutional Neural Network) can be considered, which is a typical learning model. VGG16, AlexNet, etc. can be applied.

Since the method of using the learning data set and causing the learning controller 52 to perform “supervised learning” of the learning device 50 is well known, the description of the learning method of the learning device 50 is omitted here.

When an image of an arbitrary subject (for example, an image corresponding to the first image 25) is input, the learned learning device (recognizing device) 50 acquires a second image obtained by photographing the arbitrary subject with the second photographing method. An image equivalent to the image 27 is output as a recognition result.

Accordingly, an image (recognition result) equivalent to the second image 27 obtained by photographing an arbitrary subject by the second photographing method can be obtained without photographing the arbitrary subject by the second photographing method. .

<First Embodiment of Learning Data Creation Device>
FIG. 2 is a block diagram of the essential parts of the first embodiment of the learning data creation device 10A shown in FIG.

The subject in this example is a drug with a stamp or print added, and the image acquisition unit 20 shown in FIG. It is composed of a control section 26 that controls the light source 24 and an image processing section 30 .

3 and 4 are a side view and a plan view showing how images of the sachet TP are acquired using the camera units 22 (22A, 22B) and the illumination light sources 24 (24F, 24B, 24R, 24L). It is a diagram.

A medicine bandage PB in which sachets TP are continuously formed is placed on a stage 21 and conveyed in sequence. The stage 21 is a plate-like member having a mounting surface and a back surface parallel to the xy plane (horizontal plane). The stage 21 is made of a material having optical transparency. Here, the stage 21 has a size of 130 mm in the x-axis direction and 80 mm in the y-axis direction.

The sachet TP contains a plurality of individually-prepared medicines to be taken at one time. A plurality of illumination light sources 24F, 24B, 24R, and 24L of the illumination light source 24 are arranged on the upper side (the +z direction side in FIG. 3) and the lower side (the −z direction side in FIG. 3) of the medical bandage PB, respectively. Four left and right illumination sources 24F, 24B, 24R and 24L are arranged. 3 shows illumination light sources 24R and 24L on the right and left sides of the medical bandage PB, and FIG. 4 shows four illumination light sources 24F, 24B, 24R, and 24L on the lower side of the medical bandage PB. are omitted respectively.

The four illumination light sources 24F, 24B, 24R, and 24L on the upper side of the medical bandage PB emit light obliquely upward in the -x direction, +x direction, +y direction, and -y direction of FIG. 4 in the xy plan view. Irradiate. That is, the illumination direction of the illumination light source 24F is a direction facing the illumination direction of the illumination light source 24B in xy plan view, and the illumination direction of the illumination light source 24R is orthogonal to the illumination directions of the illumination light sources 24F and 24B in xy plan view. direction, which is the direction opposite to the illumination direction of the illumination light source 24L.

The illumination sources 24F, 24B, 24R, and 24L on the lower side of the medical bandage PB are similarly arranged. Thereby, the illumination light source 24 irradiates light on the front side and the back side of (the drug packaged in) the sachet TP.

The camera section 22 (two camera sections 22A and 22B) is composed of a digital camera. As shown in FIG. 3, one camera section 22A is arranged above the medical bandage PB, and the other camera section 22B is arranged below the medical bandage PB. The camera section 22A and the camera section 22B photograph the front side and the back side of (the drug packaged in) the sachet TP.

The sachet TP (medicine bandage PB) is conveyed in the +x direction (longitudinal direction of the medicine bandage PB) in FIG. 4 by a conveying mechanism (not shown). When photographing, the upper side of the sachet TP is illuminated by the illumination light sources 24F, 24B, 24R, and 24L on the upper side of the sachet TP. illuminates the lower side of the sachet TP from four directions, front, rear, left, and right, respectively. It is preferable that the sachet TP is not irradiated with light other than the light emitted from the illumination light source 24 during photographing.

As shown in FIG. 4, the distances (d1, d2, d3, d4) between each of the illumination light sources 24F, 24B, 24R, and 24L on the upper side of the sachet TP and the photographing optical axis PA of the camera section 22A are the same. is. That is, the plurality of illumination light sources 24F, 24B, 24R, and 24L and the photographing optical axis PA are equidistant (d1=d2=d3=d4). The illumination light sources 24F, 24B, 24R, and 24L on the lower side of the packet TP and the camera section 22B are similarly arranged.

Returning to FIG. 2, the control unit 26 includes a camera control unit that controls the camera units 22A and 22B, and an illumination control unit that controls the illumination light source 24. When photographing the upper side of the sachet TP, the illumination control The unit sequentially lights the illumination light sources 24F, 24B, 24R, and 24L above the sachet TP, and sequentially illuminates the sachet TP from different directions. The camera control section causes the camera section 22A to photograph the sachets TP illuminated from each lighting direction each time they are sequentially illuminated.

In addition, the illumination control unit of the control unit 26 turns on the upper illumination light sources 24F, 24B, 24R, and 24L at the same time to uniformly illuminate (from all directions), and the camera control unit uniformly (from all directions) The illuminated sachet TP is photographed by the camera section 22A.

When photographing the lower side of the sachet TP, the control section 26 controls the illumination light sources 24F, 24B, 24R, 24L on the lower side of the sachet TP and the camera section 22B on the lower side in the same manner as described above.

FIG. 5 is a diagram showing five images in which one medicine (an example of medicine) in the sachet TP is illuminated in different directions.

In FIG. 5, four images _GF , GB, _GR , and _GL are images captured by the camera unit 22A by sequentially turning on the illumination light sources 24F, _24B , 24R, and 24L above the drug. , and an image _GA are images captured by the camera unit 22A with the upper illumination light sources 24F, 24B, 24R, and 24L turned on simultaneously.

In the four images _GF , _GB , _GR , and _GL shown in FIG. 5, luminance unevenness occurs along with the illumination direction. Also, " _A " on _{each image shown} in FIG. As shown, the appearance of the shadow of the stamp S differs depending on the lighting direction.

On the other hand, the image _GA photographed with the four illumination light sources 24F, 24B, 24R, and 24L turned on at the same time does not have uneven luminance, but the marking S is less clear because the shadow of the marking S is less likely to appear. It should be noted that FIG. 5 does not show how the shadow of the marking S appears or whether the marking S is clear or unclear.

FIG. 6 is a block diagram mainly showing a specific configuration of the image processing section 30 shown in FIG.

The image processing section 30 shown in FIG. 6 is composed of an image clipping section 32 and an edge image generating section 34 .

Images of the sachet TP acquired by the camera units 22A and 22B (in this example, 5 images of the front side and 5 images of the back side, a total of 10 images) are sent to the image clipping unit 32. Added.

The image clipping unit 32 clips regions of a plurality of medicines packaged in the sachet TP from the image of the sachet TP to generate a plurality of medicine images.

FIG. 7 is a diagram showing an example of a drug image obtained by cutting out one drug region from the image of the sachet TP.

It is preferable that the medicine image is cut out according to the outline of the medicine after detecting the outline of the medicine. In the example shown in FIG. 7, a rectangular area in which the outer shape of the drug is inscribed is cut out from the image of the sachet TP. In this way, 10 drug images are cut out for one drug.

Note that the medicine image shown in FIG. 7 is an image without luminance unevenness, which is cut out from the image of the sachet TP photographed with light incident from all directions.

Hereinafter, five drug images taken by the camera unit 22A above the drug among the ten drug images are defined as images _GF , _GB , _GR , _GL and _GA shown in FIG. Image processing for these images will be described.

The image G _A cut out by the image cut-out unit 32 shown in FIG. 6 is an image without brightness unevenness, which is obtained by photographing the subject (medicine) with light incident from all directions. The image _GA is output as the first image 25 photographed by the first photographing method. That is, the first photographing method is a photographing method of photographing with a _lighting method of uniformly illuminating the medicine (illuminating the medicine from all directions). do.

Further, among the five images _GF , _GB , _GR , _GL , and GA cut out by the image cutout unit 32, four images _G in which the appearance of the shadow of the stamp S differs depending on the lighting direction. _F , _GB , _GR and _GL are applied to the edge image generator 34 .

The edge image generation unit 34 uses an edge extraction filter (for example, a Sobel filter) in a direction corresponding to the illumination direction from the four images G _F , GB, _GR , and _{G L} to generate four edge images. to generate

FIG. 8 is a schematic diagram of the cross-sectional structure of the drug T cut along the xy plane passing through the center of the drug T, showing the profile of a line for one pixel.

The medicine T shown in FIG. 8 has a diameter D, and a mark S, which is a secant line formed by a groove having a V-shaped cross section, is formed on the surface. The width of the groove of the mark S is W. The width of the groove of the marking S is the distance from one end to the other end of the groove in the direction orthogonal to the extending direction of the groove, and is the distance on the surface of the medicine T.

Here, only the illumination light source 24F of the illumination light source 24 is lit and the drug T is illuminated with the illumination light _LF , and only the illumination light source _24B of the illumination light source 24 is lit and the drug T is illuminated with the illumination light LB. The appearance of the shadow of the stamp S differs depending on the case.

That is, when the medicine T is illuminated with the illumination light _LF , the left side (rear side) surface SB of the marking S is illuminated with the illumination light _LF , but the right _side (front side) surface SF of the marking S is irradiated with the illumination light _LF . is no longer irradiated with the illumination light _LF , and a shadow is generated on the right side surface _SF of the stamp S. Similarly, when the drug _T is illuminated with the illumination light LB in the opposite direction to the illumination light _LF , the right surface _SF of the marking S is illuminated with the illumination light _LB , but the left surface S of the marking S is irradiated with the illumination light LB. The illumination light L _B is no longer applied to _B , and a shadow is generated on the left surface S _B of the stamp S.

FIG. 9 is a diagram showing an example of a Sobel filter used for edge extraction in the edge image generator 34. As shown in FIG.

The Sobel filter _FF is used for edge extraction from the image _GF of the drug _T irradiated with illumination light _LF from the right direction (front) in FIG. This is used for edge extraction from the image GB of the drug _T irradiated with the illumination light _LB from the upper left direction (rear).

As for the kernel size of the Sobel filters F _F and F _B shown in FIG. 9, it is preferable to use a Sobel filter with a size larger than half the width W of the stamp S (the number of pixels of the width W). For example, if the number of pixels in the width of the groove of the marking S is 4 pixels, a Sobel filter with a size larger than the half of 2 pixels (3 pixels in the x-axis direction×3 pixels in the y-axis direction, etc.) is used. In the present embodiment, each illumination light produces a shadow in an area half the width of the groove. Therefore, by using an edge extraction filter having a size that takes into consideration the number of pixels from the edge, the groove can be extracted with high accuracy and the groove It is possible to reduce information other than imprints such as surface patterns and scratches smaller than the width of the .

The edge image generator 34 uses Sobel filters _FF and _FB for the images _GF and _GB , respectively, to generate edge images corresponding to the images _GF and _GB . In addition, the edge image generation unit 34 also generates the image G _R of the drug T irradiated with illumination light from the right direction and the image G _L of the drug T irradiated with illumination light from the left direction as described above. Similarly, an edge image is generated by using a Sobel filter corresponding to the direction of illumination light.

Note that the filter used for the edge extraction filtering process in the edge image generation unit 34 is not limited to the Sobel filter, and a Laplacian filter, a Canny filter, or the like can be used.

The edge image generator 34 generates edge images for each of the four images _GF , _GB , _GR , and _GL , and generates a second image 27 by combining the generated four edge images. output as This second image 27 is an image in which the stamp S added to the medicine is emphasized.

In this way, the first image acquisition unit 20A of the image acquisition unit 20 uses a method (first imaging method) of photographing an illuminated drug by a lighting method of uniformly illuminating the drug (illuminating the drug from all directions). 1 image 25 (image G _A ) is acquired.

On the other hand, the second image acquisition section 20B of the image acquisition section 20 acquires a second image captured by a second imaging method different from the first imaging method. That is, the second image acquisition unit 20B captures the four images G _F , G _B , G _L , G _R and G by sequentially capturing a plurality of light beams (front, back, left, and right) incident on the drug in different directions. _A is acquired, image processing is performed to emphasize the stamp added to the medicine, and a second image 27 in which the stamp is emphasized is acquired.

Needless to say, the image processing unit 30 acquires the first image 25 and the second image 27 based on the five drug images captured by the camera unit 22B below the drug among the ten drug images. stomach.

The first image 25 and the second image 27 output from the image processing section 30 are applied to the storage control section 40 shown in FIG.

The memory control unit 40 sets the first image 25 and the second image 27 as a set as described above, and stores this set in the storage unit 42 as learning data for the learning device 50 .

The illumination light source 24 in this example has four illumination light sources 24F, 24B, 24R, and 24L, which are arranged in front, rear, right, and left of the subject. It may be arranged.

Also, a learned recognizer may be applied instead of the image processing unit 30 (especially the edge image generating unit 34) shown in FIG.

The trained recognizer in this case uses a set of multiple images (in this example, four images _GF , _GB , _GL , _GR and _GA ) with different illumination directions to the subject as input images. , a stamp added to the subject or an image in which the print is emphasized is output as a recognition result. As input data to this recognizer, when a set of multiple images with different illumination directions on the subject (medicine) is input to the recognizer, the recognizer outputs an image in which the marking or printing is emphasized, which is the recognition result. It can be generated (output) as a second image 27 .

FIG. 10 is an image diagram of a plurality of sets of learning data sets of the first image 25 and the second image 27 stored in the storage unit 42. As shown in FIG.

The storage unit 42 stores a plurality of sets of the first image 25 and the second image 27 of each of a plurality of drugs with different stamp shapes and the like as one set.

As a result, the learning data creation device 10A efficiently creates a learning data set (a plurality of sets of the first image 25 and the second image 27) for “supervised learning” of the learning device 50, and stores the learning data set in the storage unit 42. can be made

Returning to FIG. 1, the learning control unit 52 of the learning device 50 uses the learning data set created by the learning data creation device 10A (the learning data set stored in the storage unit 42), and the learning device 50 performs machine learning. Let For example, learning is performed to optimize the parameters of the learning device 50 (each parameter of a learning model such as CNN).

The learning device 50, which has undergone machine learning using the learning data set of this example, outputs an edge image that emphasizes the engraving or printing added to the medicine as a recognition result when a single image of an arbitrary medicine is input. This recognition result can be used to identify or audit any drug, and by combining the edge image with the image of any drug, a composite image can be generated in which the inscription of the drug is easy to see. can be used for

<Second Embodiment of Learning Data Creation Device>
FIG. 11 is a diagram showing the main part of the second embodiment of the learning data creation device 10A shown in FIG. 1, and particularly shows the illumination light source.

The illumination light source 60 shown in FIG. 11 is composed of a plurality of illumination light sources 60A, 60B, and 60C arranged around the drug T, and the plurality of illumination light sources 60A, 60B, and 60C each emit light with different wavelength bands. , to irradiate the drug T from three directions. In this example, the illumination light sources 60A, 60B, and 60C respectively emit red (R) light in the wavelength band λa, green (G) light in the wavelength band λb, and blue (B) light in the wavelength band λc.

Also, the camera section (not shown) has a known color image sensor having a plurality of types of pixels each having sensitivity to a plurality of lights (RGB lights) having different wavelength bands. A plurality of types of pixels (R pixels, G pixels, and B pixels) of the color image sensor are provided with RGB color filters, respectively. It has sensitivities corresponding to the bands λa, λb, and λc.

In the second embodiment, when photographing a subject (medicine T), a plurality of illumination light sources 60A, 60B, and 60C are simultaneously turned on to irradiate the drug T with RGB light.

A first photographing method for obtaining a first image of the drug T is to photograph the drug T illuminated by the illumination light sources 60A, 60B, and 60C with a camera unit equipped with a color image sensor, and obtain a color image of the drug T (first image).

A second imaging method for obtaining a second image of the drug T is to photograph the drug T illuminated by the illumination light sources 60A, 60B, and 60C by a camera unit equipped with a color image sensor in the same manner as the first imaging method. A plurality of images (R image, G image, B image) are obtained corresponding to the types of pixels (RGB pixels). Then, the second image acquiring section performs image processing on each of the R image, the G image, and the B image to acquire the second image.

Image processing for the R, G, and B images uses edge extraction filters corresponding to the illumination directions of the R, G, and B lights of the illumination light sources 60A, 60B, and 60C to generate three edge images. A synthesized image obtained by synthesizing the edge images is generated as a second image.

According to the second embodiment, it is possible to acquire the first image and the second image that form a set by one shot.

<Third Embodiment of Learning Data Creation Device>
FIG. 13 is a diagram showing the main part of the third embodiment of the learning data creation device 10A shown in FIG. 1, and particularly shows the second image acquisition part.

The second image acquiring section 20B shown in FIG. 13 is a section that acquires the second image 27 of the subject, and is composed of the three-dimensional measuring device 70 and the image processing section 72. As shown in FIG. Note that the first image acquisition unit 20A that acquires the first image of the subject can use, for example, that of the first embodiment or the second embodiment.

A three-dimensional measuring device (three-dimensional information acquisition unit) 70 shown in FIG. 13 acquires three-dimensional information of a subject (medicine) having a three-dimensional shape. The three-dimensional measuring device 70 may be one that scans a laser beam to obtain three-dimensional information on a drug, one that obtains three-dimensional information on a drug by a stereo camera, or the like.

The image processing unit 72 extracts the marking of the medicine based on the three-dimensional information of the medicine acquired by the three-dimensional measuring device 70, generates an image emphasizing the marking of the medicine, and uses the generated image as the second image 27. Output.

<Fourth Embodiment of Learning Data Creation Device>
The first image acquisition unit and the second image acquisition unit of the fourth embodiment capture images of the same subject using imaging modalities with different imaging methods, and acquire a set of first and second images.

A CT (Computed Tomography) apparatus and an MRI (Magnetic Resonance Imaging) apparatus are conceivable as imaging modalities that use different imaging methods.

An MRI apparatus can acquire volume data of a subject. In addition, the CT apparatus captures a plurality of CT images at a predetermined pitch, and can obtain volume data in which the plurality of CT images are stacked.

Therefore, images at the same cross-sectional position of each volume data obtained by photographing the same subject by the CT apparatus and the MRI apparatus can be used as the first image and the second image, respectively.

The image obtained by the CT apparatus may be the first image and the image obtained by the MRI apparatus may be the second image, or the image obtained by the MRI apparatus may be the first image and the image obtained by the CT apparatus may be the second image. good too.

<Machine learning method>
FIG. 14 is a flow chart illustrating an embodiment of a machine learning method according to the invention.

Note that the machine learning method shown in FIG. 14 shows the processing contents of the machine learning method including the learning data creation method in the learning data creation device of the first embodiment.

14, the control unit 26 shown in FIG. 2 turns on all the illumination light sources 24F, 24B, 24R, and 24L of the illumination light source 24 at the same time (step S10), and the subject (medicine) illuminated from all directions is viewed by the camera unit. 22A. Thereby, the first image acquisition section 20A of the image acquisition section 20 having the camera section 22A, the illumination light source 24, and the control section 26 acquires the first image 25 (step S12).

Subsequently, the control unit 26 turns on only the illumination light source 24F (step S14), causes the camera unit 22A to photograph the drug illuminated by the illumination light source 24F, and acquires an image _GF of the drug (step S16).

Similarly to steps S16 and S18, the control unit 26 sequentially turns on the illumination light sources _24B , 24R, and 24L, causes the camera unit 22A to photograph the drug in synchronization with the sequential lighting, and produces images GB, _GR , . _GL is acquired (steps S20 to S28).

Subsequently, the edge image generator 34 applies the corresponding Sobel filters to the four images _GF , _GB , _GR , and _GL obtained in steps S16, S20, S24, and S28, respectively. 4 edge images are generated by using the edge images, and an edge image obtained by synthesizing the generated 4 edge images is obtained as the second image 27 (step S30). This second image 27 is an image in which the stamp or print added to the medicine is emphasized.

The storage control unit 40 sets the first image 25 obtained in step S12 and the second image 27 obtained in step S30 as a set, and stores the set in the storage unit 42 (step S32).

In the steps S10 to S32 described above, a set of the first image 25 and the second image 27 is acquired based on a plurality of images of the drug taken by the camera section 22A above the drug and stored in the storage section 42. The camera section 22</b>B below the drug also photographs the drug, similarly acquires a set of the first image 25 and the second image 27 and stores it in the storage section 42 . Further, the processing of steps S10 to S32 described above is also performed for other drugs (a plurality of drugs) of different types, and as a result, a plurality of first images 25 and second images 27 corresponding to the plurality of drugs are displayed. A set is stored in the storage unit 42 .

When the learning device 50 undergoes machine learning, the learning control unit 52 uses the learning data set created by the learning data creation method and stored in the storage unit 42 to cause the learning device 50 to undergo machine learning (step S34).

[others]
It goes without saying that the present invention is not limited to the machine learning device and method of this embodiment, but also includes a learning data creation device and method included in the machine learning device and method.

The learning data created by the learning data creation device is stored in the storage unit. may contain.

Furthermore, the present invention is not limited to the embodiments described above, and it goes without saying that various modifications are possible without departing from the spirit of the present invention.

10 Machine learning device 10A Learning data creation device 20 Image acquisition unit 20A First image acquisition unit 20B Second image acquisition unit 21 Stage 22 Camera unit 22A Camera unit 22B Camera units 24, 24B, 24F, 24L, 24R Illumination light source 25 First Image 26 Control unit 27 Second image 30 Image processing unit 32 Image clipping unit 34 Edge image generation unit 40 Memory control unit 42 Storage unit 50 Learning device 52 Learning control unit 60, 60A, 60B, 60C Illumination light source 70 Three-dimensional measuring device 72 Image processing unit FB, _FF Sobel filter _GA , GB, _GF , _GL , _GR Image _LB , _{LF F illumination} light PA Photographing optical axis PB Medicine bandage S Engraving S10 to S34 Step _T Medicine TP Minutes Wrapper λa, λb, λc Wavelength band

Claims (13)

a first image obtaining unit that obtains a first image by photographing a medicine to which a stamp or print is added using a first photographing method;
a second image acquisition unit configured to acquire a second image by capturing an image of the drug using a second image capturing method different from the first image capturing method;
a storage control unit that sets the first image and the second image and stores the set in a storage unit as learning data for a learning device;
The second image acquisition unit emphasizes the engraving or printing added to the drug from a plurality of images captured by the second imaging method, in which directions of illumination to the drug are different. an image processing unit that generates the second image obtained by
The memory control unit stores the first image as input data for the learning device and the second image as correct data for the learning device in the storage unit.
Learning data creation device. The image processing unit generates, from the plurality of images, a plurality of edge images in which stamps or prints added to the medicine are emphasized, and generates a synthesized image obtained by synthesizing the generated plurality of edge images as the second image. 2. The learning data creation device according to claim 1 , wherein the learning data creation device generates as The image processing unit includes a trained recognizer that uses the plurality of images as input images and outputs an image in which the stamp or print added to the medicine is emphasized as a recognition result, and the plurality of images are processed by the recognizer. 2. The learning data creation device according to claim 1 , wherein the recognition result of said recognizer is input to said second image. 2. The learning data creation device according to claim 1 , wherein the first imaging method and the second imaging method use different lighting methods for the drug . The illumination method of the first imaging method includes making light incident on the drug from all directions,
5. The learning data creation apparatus according to claim 4 , wherein the illumination method of the second imaging method sequentially causes a plurality of lights having different incident directions with respect to the drug . The first image acquisition unit and the second image acquisition unit include a plurality of illumination light sources arranged around the drug , and an illumination control unit that controls lighting of the plurality of illumination light sources,
6. The learning according to claim 5 , wherein the illumination control unit simultaneously turns on the plurality of illumination light sources as the illumination method for the first photographing method, and sequentially lights the plurality of illumination light sources as the illumination method for the second photographing method. Data production device. 7. The learning data creation device according to claim 6 , wherein the plurality of illumination light sources are arranged at least two of front, rear, right and left sides of the drug . The first image acquisition unit and the second image acquisition unit include a camera unit that captures the drug , a camera control unit that controls the camera unit, and an image processing unit,
The camera control unit causes the camera unit to photograph the medicine when the medicine is illuminated by the illumination method of the first imaging method, and sequentially illuminates the medicine by the illumination method of the second imaging method. , causing the drug to be photographed each time the sequential illumination is performed;
The learning data according to any one of claims 5 to 7 , wherein the image processing unit processes a plurality of images captured by the camera unit each time the illumination is performed sequentially to generate the second image. creation device. The first image acquisition unit and the second image acquisition unit are a plurality of illumination light sources arranged around the drug , and a plurality of illumination light sources for irradiating the drug with a plurality of lights having different wavelength bands. , a camera unit comprising a plurality of types of pixels each having sensitivity to a plurality of lights with different wavelength bands,
The first image acquisition unit captures the drug illuminated by the plurality of illumination light sources with the camera unit to acquire the first image,
The second image acquisition unit captures the drug illuminated by the plurality of illumination light sources with the camera unit, and acquires a plurality of images corresponding to the plurality of types of pixels.
The learning data creation device according to claim 1 . a first image obtaining unit that obtains a first image by photographing the medicine to which the stamp is added using a first photographing method;
a second image acquisition unit configured to acquire a second image by capturing an image of the drug using a second image capturing method different from the first image capturing method;
a storage control unit that sets the first image and the second image and stores the set in a storage unit as learning data for a learning device;
The second image acquisition unit includes a three-dimensional information acquisition unit that acquires three-dimensional information of the drug , and an image processing unit,
The image processing unit extracts the stamp of the drug based on the three-dimensional information of the drug acquired by the three-dimensional information acquisition unit, and generates the second image in which the stamp of the drug is emphasized ,
The memory control unit stores the first image as input data for the learning device and the second image as correct data for the learning device in the storage unit.
Learning data creation device. a learner;
11. Input when the learning data stored in the storage unit of the learning data creation device according to any one of claims 1 to 10 is used, and the first image constituting the set is machine-learned by the learning device a learning control unit that causes the learner to perform machine learning using a second image that constitutes the set as an image, and a second image that constitutes the set as correct data;
A machine learning device with a step in which the first image acquisition unit acquires a first image by photographing the medicine to which a stamp or print is added using a first photographing method;
a step in which a second image obtaining unit obtains a second image by photographing the medicine with a second photographing method different from the first photographing method;
a step in which the memory control unit sets the first image and the second image as a set and stores the set in a storage unit as learning data for a learning device;
The step of acquiring the second image includes a plurality of images photographed by the second photographing method, wherein a plurality of images in which the direction of incidence of illumination on the medicine differs, and a stamp or print added to the medicine. generating the second image in which the
The step of storing in the storage unit stores the first image as input data for the learning device and the second image as correct data for the learning device in the storage unit.
How to create learning data. The learning control unit uses the learning data created by the learning data creation method according to claim 12 and stored in the storage unit, and the input when the learning device performs machine learning on the first images that make up the set. A machine learning method comprising the step of causing a learner to perform machine learning using the second images constituting the set as correct data.

Images