JP2021033737A - Learning data creation device and method, machine learning device and method - Google Patents

Abstract

To provide a learning data creation device and method, a machine learning device and method which efficiently create learning data that makes a learning machine learn with a teacher, and make the learning machine perform machine learning.SOLUTION: A machine learning device 10 comprises a learning data creation device 10A and a learning machine 50. The learning data creation device 10A comprises: a first image acquisition unit 20A which acquires a first image 25 by imaging a subject in a first imaging method; a second image acquisition unit 20B which acquires a second image 27 by imaging the subject in a second imaging method different from the first imaging method; and a storage control unit 40 which pairs the first image 25 and the second image 27 to store the pair in a storage unit 42 as learning data of the learning machine 50. A learning control unit 52 of the learning machine 50 uses the learning data stored in the storage unit 42, makes the first image 25 constituting the pair the input image in a case where the learning machine 50 is made to perform machine learning, and makes the learning machine 50 perform machine learning with the second image 27 constituting the pair as correct data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning data creation device and method, and a machine learning device and method, and particularly relates to a technique for efficiently creating learning data for machine learning of a learning device.

Conventionally, as an image output device for creating learning data, there is one described in Patent Document 1.

The image output device described in Patent Document 1 acquires a first image of an object taken from a first direction and a second image of an object taken from a second direction, and the first of these. A composite image is generated in which the first image and the second image are included in one image in a predetermined positional relationship.

The image output device described in Patent Document 1 captures an object from a plurality of directions by using the composite image generated as described above as input data of a learner capable of machine learning or obtained as a learning result. It enables machine learning using the information of the image, or makes it possible to obtain the evaluation result of the object.

Japanese Unexamined Patent Publication No. 2018-132962

In Patent Document 1, the image output device creates a composite image as input data to the learner, but does not create "correct answer data of the output used for learning" in the case of "supervised learning".

When performing "supervised learning", it is necessary to prepare learning data consisting of a set of "input data used for learning" and "correct answer data of output used for learning". There is no description on how to create correct answer data. Generally, when creating learning data, it takes time and effort to create correct answer data, but the invention described in Patent Document 1 does not solve such a problem.

The present invention has been made in view of such circumstances, and is a learning data creation device and method capable of efficiently creating learning data for "supervised learning" of a learning device and machine learning of the learning device, and a machine. It is an object of the present invention to provide a learning device and a method.

In order to achieve the above object, the learning data creation device according to one aspect of the present invention includes a first image acquisition unit that captures a subject by a first imaging method and acquires a first image, and a first imaging method for the subject. A storage unit in which a second image acquisition unit that acquires a second image by photographing with a second imaging method different from the above, and a first image and a second image are paired, and the pair is stored in the storage unit as learning data of the learner. It includes a control unit.

According to one aspect of the present invention, for example, the first image acquired by the first image acquisition unit is used as input data to the learning device, and the second image acquired by the second image acquisition unit is used as correct answer data for the learning device. can do. As a result, a pair of the first image and the second image can be efficiently created.

In the learning data creation device according to another aspect of the present invention, the second image acquisition unit includes an image processing unit that processes an image captured by the second imaging method to generate a second image, and is a storage control unit. It is preferable that the first image is used as the input data of the learning device and the second image is stored in the storage unit as the correct answer data of the learning device.

In the learning data creating apparatus according to still another aspect of the present invention, the images taken by the second shooting method are a plurality of images in which the incident directions of the illuminations on the subject are different, and the image processing unit is composed of the plurality of images. In the learning data creation device according to still another aspect of the present invention, it is preferable to generate a second image in which the marking or printing added to the subject is emphasized, the image processing unit is added to the subject from a plurality of images. It is preferable to generate a plurality of edge images in which engraving or printing is emphasized, and to generate a composite image in which the generated plurality of edge images are combined as a second image.

In the learning data creation device according to still another aspect of the present invention, the image processing unit uses a plurality of images as input images and outputs an image in which the marking or printing emphasized added to the subject is emphasized as a recognition result. It is preferable that a plurality of images including a device are input to the recognizer and the recognition result of the recognizer is generated as a second image.

In the learning data creating apparatus according to still another aspect of the present invention, the subject is preferably a drug to which a stamp or a print is added.

In the learning data creating device according to still another aspect of the present invention, it is preferable that the first photographing method and the second photographing method have different illumination methods for the subject.

In the learning data creation device according to still another aspect of the present invention, the lighting method of the first photographing method causes light to be incident on the subject from all directions, and the illumination method of the second photographing method has the incident direction to the subject. It is preferable to sequentially inject a plurality of different lights.

In the learning data creation device according to still another aspect of the present invention, the first image acquisition unit and the second image acquisition unit control a plurality of illumination light sources arranged around the subject and lighting of the plurality of illumination light sources. It is preferable that the illumination control unit includes the illumination control unit, and the illumination control unit lights a plurality of illumination light sources at the same time as the illumination method of the first imaging method, and sequentially lights the plurality of illumination light sources as the illumination method of the second imaging method.

In the learning data creating device according to still another aspect of the present invention, it is preferable that the plurality of illumination light sources are arranged at at least two of the front, the rear, the right and the left of the subject.

In the learning data creation device according to still another aspect of the present invention, the first image acquisition unit and the second image acquisition unit include a camera unit that captures a subject, a camera control unit that controls the camera unit, and an image processing unit. When the subject is illuminated by the lighting method of the first shooting method, the camera control unit causes the camera unit to shoot the subject, and when the subject is sequentially illuminated by the lighting method of the second shooting method, the subject is sequentially illuminated. It is preferable that the subject is photographed each time the subject is photographed, and the image processing unit performs image processing on a plurality of images captured by the camera unit each time the image is sequentially illuminated to generate a second image.

In the learning data creating device according to still another aspect of the present invention, the subject is engraved or printed, and the image processing unit performs image processing for emphasizing the engraving or printing added to the subject, and the engraving or printing is performed. It is preferable to generate a second image in which the above is emphasized.

In the learning data creation device according to still another aspect of the present invention, the first image acquisition unit and the second image acquisition unit are a plurality of illumination light sources arranged around the subject, each having a plurality of different wavelength bands. The first shooting method includes a plurality of illumination light sources that irradiate a subject with light and a camera unit having a plurality of types of pixels having sensitivity to a plurality of lights having different wavelength bands, and the first shooting method is illuminated by a plurality of illumination light sources. The first image is acquired by shooting the subject with the camera unit, and the second shooting method is to shoot the subject illuminated by a plurality of illumination light sources with the camera unit and capture a plurality of images corresponding to a plurality of types of pixels. It is preferable that the second image acquisition unit acquires the second image by performing image processing on a plurality of images.

In the learning data creating device according to still another aspect of the present invention, the subject is engraved, and the second image acquisition unit includes a three-dimensional information acquisition unit that acquires three-dimensional information of the subject and an image processing unit. Including, it is preferable that the image processing unit extracts the marking of the subject based on the three-dimensional information of the subject acquired by the three-dimensional information acquisition unit and generates a second image in which the marking of the subject is emphasized.

In the learning data creating device according to still another aspect of the present invention, it is preferable that the first imaging method and the second imaging method are imaging methods for photographing a subject using different imaging modality.

In the learning data creating apparatus according to still another aspect of the present invention, one of the first imaging method and the second imaging method is an imaging method for photographing a subject using a CT apparatus, and the first imaging method. The other imaging method of the imaging method and the second imaging method is preferably an imaging method in which a subject is photographed using an MRI apparatus.

The machine learning device according to still another aspect of the present invention uses the learning device and the learning data stored in the storage unit of the above-mentioned learning data creating device, and machine-learns the first image forming the set from the learning device. It is provided with a learning control unit for machine learning the learner using the second image constituting the set as the correct answer data as the input image for the case of making the data.

The machine learning device according to still another aspect of the present invention includes a first image acquisition unit that captures a subject by the first imaging method and acquires a first image, and a second imaging method that captures the subject differently from the first imaging method. It is provided with a second image acquisition unit for acquiring a second image by taking a picture with the image, a learning device, and a learning control unit for machine learning the learning device by combining the first image and the second image.

In the learning data creation method according to still another aspect of the present invention, the first image acquisition unit captures the subject by the first imaging method and acquires the first image, and the second image acquisition unit captures the subject. The step of acquiring a second image by photographing with a second imaging method different from the first imaging method, and the memory control unit sets the first image and the second image as a set and stores the set as learning data of the learner. Includes steps to be stored in the department.

In the learning data creation method according to still another aspect of the present invention, the subject is preferably a drug to which a stamp or a print is added.

In the learning data creation method according to still another aspect of the present invention, it is preferable that the first photographing method and the second photographing method have different illumination methods for the subject.

In the machine learning method according to still another aspect of the present invention, the learning control unit learns the first image constituting the set by using the learning data created by the above-mentioned learning data creation method and stored in the storage unit. It includes a step of making the learner machine-learn by using the second image constituting the set as the correct answer data as the input image when the instrument is machine-learned.

In the machine learning method according to still another aspect of the present invention, the first image acquisition unit takes a picture of the subject by the first shooting method and acquires the first image, and the second image acquisition unit takes the subject. A step of acquiring a second image by shooting with a second shooting method different from the first shooting method, and a step of the learning control unit machine learning the learner by combining the first image and the second image. Including.

According to the present invention, learning data for "supervised learning" of a learning device can be efficiently and automatically created.

FIG. 1 is a block diagram showing a schematic configuration of a machine learning device according to the present invention. FIG. 2 is a block diagram of a main part of the first embodiment of the learning data creating apparatus shown in FIG. FIG. 3 is a side view showing how an image of the packaging bag is acquired by using the camera unit and the illumination light source. FIG. 4 is a plan view showing how an image of the packaging bag is acquired by using the camera unit and the illumination light source. FIG. 5 is a diagram showing five images in which the illumination direction of one drug in the sachet is different. FIG. 6 is a block diagram mainly showing a specific configuration of the image processing unit shown in FIG. FIG. 7 is a diagram showing an example of a drug image obtained by cutting out a region of one drug from the image of the sachet. FIG. 8 is a schematic view of the cross-sectional structure of the drug cut in the xy plane passing through the center of the drug. FIG. 9 is a diagram showing an example of a sobel filter used for edge extraction in the edge image generation unit. FIG. 10 is an image diagram of a plurality of sets of training data sets of the first image and the second image stored in the storage unit. FIG. 11 is a diagram showing a main part of the second embodiment of the learning data creating apparatus shown in FIG. FIG. 12 is a diagram showing the spectral sensitivity of the color image sensor of the camera unit used in the second embodiment of the learning data creating apparatus shown in FIG. FIG. 13 is a diagram showing a main part of the third embodiment of the learning data creating apparatus shown in FIG. FIG. 14 is a flowchart showing an embodiment of the machine learning method according to the present invention.

Hereinafter, preferred embodiments of the learning data creation device and method and the machine learning device and method according to the present invention will be described with reference to the accompanying drawings.

[Configuration of machine learning device]
FIG. 1 is a block diagram showing a schematic configuration of a machine learning device according to the present invention.

The machine learning device 10 shown in FIG. 1 includes a learning data creating device 10A and a learning device 50, and the learning device 50 performs machine learning using a learning data set created by the learning data creating device 10A.

The learning data creation device 10A includes an image acquisition unit 20 including a first image acquisition unit 20A and a second image acquisition unit 20B, and a storage control unit 40.

The first image acquisition unit 20A is a portion that captures the subject by the first imaging method and acquires the first image 25, and the second image acquisition unit 20B captures the same subject by the second imaging method and second. This is the part for acquiring the image 27. The details of the first photographing method by the first image acquisition unit 20A and the second photographing method by the second image acquisition unit 20B will be described later.

The first image 25 acquired by the first image acquisition unit 20A and the second image 27 acquired by the second image acquisition unit 20B are output to the storage control unit 40, respectively.

The storage control unit 40 sets the input first image 25 and the second image 27 as a set and stores them in the storage unit 42. The storage unit 42 may be a medium such as a hard disk device built in the learning data creation device 10A, or a removable medium that can be attached to and detached from the learning data creation device 10A.

Here, the pair of the first image 25 and the second image 27 is an image of either one of the first image 25 and the second image 27 during machine learning by the learner 50 (in this example, the first image 25). ) Can be input data used for learning of the learning device 50, and the other image (second image 27 in this example) can be used as correct answer data used for learning of the learning device 50.

The storage control unit 40 acquires the first image 25 and the second image 27 for each subject of the plurality of subjects, and stores a plurality of sets of learning data sets of the first image 25 and the second image 27, respectively. It is stored in the part 42.

The learning control unit 52 of the learning device 50 can machine-learn the learning device 50 based on the learning data set stored in the storage unit 42.

As the learner 50 that uses the first image as an input image and outputs the second image as a recognition result, it is conceivable that the learner 50 is composed of a convolutional neural network (CNN), which is a typical learning model. VGG16, AlexNet, etc. can be applied.

Since the method of making the learning device 50 "supervised learning" by the learning control unit 52 using the learning data set is well known, the description of the learning method of the learning device 50 is omitted here.

When the trained learner (recognition device) 50 inputs an image of an arbitrary subject (for example, an image corresponding to the first image 25), the second image acquired by photographing the arbitrary subject by the second photographing method. An image equivalent to the image 27 is output as a recognition result.

As a result, an image (recognition result) equivalent to the second image 27 acquired by photographing an arbitrary subject by the second photographing method can be acquired without photographing an arbitrary subject by the second photographing method. ..

<First Embodiment of Learning Data Creation Device>
FIG. 2 is a block diagram of a main part of the first embodiment of the learning data creating device 10A shown in FIG. 1, and is particularly shown with respect to the image acquisition unit 20.

The subject of this example is a drug to which a stamp or print is added, and the image acquisition unit 20 shown in FIG. 2 includes a camera unit 22 for photographing the drug, an illumination light source 24 for illuminating the drug, a camera unit 22, and illumination. It is composed of a control unit 26 that controls the light source 24 and an image processing unit 30.

3 and 4 are side views and a plan view showing how the camera unit 22 (22A, 22B) and the illumination light source 24 (24F, 24B, 24R, 24L) are used to acquire an image of the packaging bag TP. It is a figure.

The drug bandage PB in which the sachet TP is continuously formed is placed on the stage 21 and sequentially transported. The stage 21 is a plate-shaped member having a mounting surface and a back surface parallel to the xy plane (horizontal plane). The stage 21 is made of a light-transmitting material. Here, the stage 21 has a size of 130 mm in the x-axis direction and 80 mm in the y-axis direction.

A plurality of drugs to be taken at one time are packaged in the packaged bag TP. The plurality of illumination light sources 24F, 24B, 24R, 24L of the illumination light source 24 are located on the upper side (+ z direction side in FIG. 3) and the lower side (−z direction side in FIG. 3) of the medicine bandage PB, respectively, forward, backward, and right. Four illumination light sources 24F, 24B, 24R, and 24L on the left side and the left side are arranged. In FIG. 3, the right and left illumination light sources 24R and 24L of the medicine bandage PB are shown, and in FIG. 4, the four illumination light sources 24F, 24B, 24R, and 24L below the medicine bandage PB are shown. Are omitted respectively.

The four illumination light sources 24F, 24B, 24R, and 24L on the upper side of the medicine bandage PB emit light from diagonally above in the −x direction, + x direction, + y direction, and −y direction in FIG. 4, respectively, in the xy plan view. Irradiate. That is, the illumination direction of the illumination light source 24F is a direction facing the illumination direction of the illumination light source 24B in the xy plan view, and the illumination direction of the illumination light source 24R is orthogonal to the illumination direction of the illumination light sources 24F and 24B in the xy plan view. It is a direction, which is a direction facing the illumination direction of the illumination light source 24L.

The illumination light sources 24F, 24B, 24R, and 24L below the medicine bandage PB are also arranged in the same manner. As a result, the illumination light source 24 irradiates the front side and the back side of the packaging bag TP (the drug packaged in the packaging bag TP) with light.

The camera unit 22 (two camera units 22A and 22B) is composed of a digital camera. As shown in FIG. 3, one camera unit 22A is arranged above the medicine bandage PB, and the other camera unit 22B is arranged below the medicine bandage PB. The camera unit 22A and the camera unit 22B photograph the front side and the back side of the packaging bag TP (the drug packaged in the packaging bag TP).

The sachet TP (medicine bandage PB) is transported in the + x direction (longitudinal direction of the drug bandage PB) in FIG. 4 by a transport mechanism (not shown). At the time of shooting, the illumination light sources 24F, 24B, 24R, and 24L on the upper side of the packaging bag TP allow the upper side of the packaging bag TP to be the illumination light sources 24F, 24B, 24R, and 24L on the lower side of the packaging bag TP. The lower side of the packaging bag TP is illuminated from four directions, front, back, left and right, respectively. It is preferable that the packaging bag TP is not irradiated with light other than the light emitted from the illumination light source 24 at the time of photographing.

As shown in FIG. 4, the distances (d1, d2, d3, d4) between the illumination light sources 24F, 24B, 24R, and 24L on the upper side of the packaging bag TP and the photographing optical axis PA of the camera unit 22A are the same. Is. That is, the plurality of illumination light sources 24F, 24B, 24R, and 24L and the photographing optical axis PA are at equal intervals (d1 = d2 = d3 = d4). The illumination light sources 24F, 24B, 24R, and 24L below the packaging bag TP and the camera unit 22B are also arranged in the same manner.

Returning to FIG. 2, the control unit 26 includes a camera control unit that controls the camera units 22A and 22B and a lighting control unit that controls the illumination light source 24, and when photographing the upper side of the packaging bag TP, the illumination control The unit sequentially lights the illumination light sources 24F, 24B, 24R, and 24L on the upper side of the packaging bag TP, and sequentially illuminates the packaging bag TP from different directions. The camera control unit causes the camera unit 22A to take a picture of the packaging bag TP illuminated from each illumination direction each time it is sequentially illuminated.

Further, the illumination control unit of the control unit 26 simultaneously lights the upper illumination light sources 24F, 24B, 24R, and 24L to illuminate them uniformly (from all directions), and the camera control unit uniformly (from all directions) lights them. The illuminated sachet TP is photographed by the camera unit 22A.

When photographing the lower side of the packaging bag TP, the control unit 26 controls the illumination light sources 24F, 24B, 24R, 24L on the lower side of the packaging bag TP and the camera unit 22B on the lower side in the same manner as described above.

FIG. 5 is a diagram showing five images in which the illumination direction of one drug (an example of the drug) in the sachet TP is different.

5, four images _{G F, G B, G R} and _{G L,} the agent of the upper illumination source 24F, 24B, 24R, and 24L sequentially lighting the, be a captured image from the camera section 22A the image _{G a,} the upper of the illumination light source 24F, 24B, 24R, and at the same time turns on the 24L, an image taken from the camera unit 22A.

Four images _G F shown in FIG. _5, G B, the _{G R} and _{G L,} luminance unevenness with the illumination directions is generated. Further, "A" on the images shown in FIG. 5 shows a marking S, the image G _F, G _B, stamped S of G _L and G _R are irregularities on the surface of the drug, later As a result, the appearance of the shadow of the marking S differs depending on the illumination direction.

On the other hand, the four illumination sources 24F, 24B, 24R, the luminance unevenness in image G _A which is simultaneously photographed by lighting a 24L is not generated, since it is difficult out shadow marking S, engraved S becomes unclear. It should be noted that in FIG. 5, the appearance of the shadow of the marking S and whether the marking S is clear or unclear are not shown.

FIG. 6 is a block diagram mainly showing a specific configuration of the image processing unit 30 shown in FIG.

The image processing unit 30 shown in FIG. 6 is composed of an image cutting unit 32 and an edge image generating unit 34.

The images of the packaging bag TP acquired by the camera units 22A and 22B (in this example, the five images on the front side and the five images on the back side, for a total of 10 images) are attached to the image cropping unit 32. Is added.

The image cutting unit 32 cuts out the regions of the plurality of drugs packaged in the package bag TP from the image of the package bag TP, and generates a plurality of drug images.

FIG. 7 is a diagram showing an example of a drug image obtained by cutting out a region of one drug from the image of the sachet TP.

It is preferable to detect the outer shape of the drug and cut out the drug image according to the outer shape of the drug. In the example shown in FIG. 7, a rectangular region inscribed with the outer shape of the drug is cut out from the image of the sachet TP. In this way, 10 drug images are cut out for one drug.

The drug image shown in FIG. 7 is an image with no uneven brightness cut out from the image of the sachet TP taken by injecting light from all directions.

Hereinafter, the five drug image captured by the upper camera portion 22A of the drug out of the ten drug image, the image _G F shown in FIG. _5, as the _{G B,} G _R, G _L and _{G A,} Image processing for these images will be described.

Image G _A cut out by the image cutting unit 32 shown in FIG. 6, were taken the drug by the incidence of light from all directions with respect to the subject (agent), an image with no luminance unevenness, in this example, this image G _A is output as a first image 25 captured by the first imaging method. That is, the first imaging method, agent uniformly illuminating a photographing method for photographing at illumination method (from all directions illuminate the drug) to an image G _A taken by the first imaging method and the first image 25 To do.

Further, five images _G F cut out by the image cutting unit _32, G _B, G R, _{G L} and _G image G shadow attitude of four different engraved S with the illumination direction of _{A F, G B, G R,} G L is applied to the edge image generation unit 34.

Edge image generating unit 34, four images _{G F, G B, G R} , an edge extraction filter in the direction corresponding to the respective illumination direction from G _L (e.g., Sobel filter) using a four edge images To generate.

FIG. 8 is a schematic view of the cross-sectional structure of the drug T cut along the xy plane passing through the center of the drug T, and shows the profile of the line for one pixel.

The drug T on FIG. 8 has a diameter of D, and a marking S, which is a score line having a V-shaped groove in a cross section, is formed on the surface of the drug T. The width of the groove of the engraved S is W. The width of the groove of the marking S is the distance from one end of the groove to the other end in the direction orthogonal to the extending direction of the groove, and refers to the distance on the surface of the drug T.

Here, lights only illumination light source 24F of the illumination light source 24, in the case of illuminating the drug T illumination light L _F, lit only illumination light source 24B of the illumination light source 24 illuminates the drug T illumination light L _B Depending on the case, the appearance of the shadow of the engraved S is different.

That is, the illumination light L _F is illuminated drug T, although the surface S _B illumination light L _F on the left side of the stamp S (rear side) is irradiated to the surface S _F of the right stamp S (front side) the illuminating light L _F is no longer illuminated, shadows are generated in the right side S _F engraved S. Similarly, the illumination light L If the _F illuminates the drug T illumination light L _B in the opposite direction, but the right side S _F of the stamp S is illumination light L _B is irradiated, the surface of the left marking S S illumination light L _B is no longer irradiated to _B, a shadow is generated on the surface S _B of the left marking S.

FIG. 9 is a diagram showing an example of a sobel filter used for edge extraction in the edge image generation unit 34.

Sobel filter F _F is used to edge extraction from the image G _F drugs T illumination light L _F is irradiated from the right direction in FIG. 8 (front), the Sobel filter F _B, 8 illumination light L _B from the left side of the upper (rear) is used when edge extraction from the image G _B drugs T irradiated.

Sobel filter F _F shown in FIG. _9, the kernel size of the F _B, it is preferable to use a larger size Sobel filter than half the width W of the engraved S (number of pixels). For example, if the number of pixels in the width of the groove of the marking S is 4, a sobel filter having a size larger than half that of 2 pixels (3 pixels in the x-axis direction x 3 pixels in the y-axis direction, etc.) is used. In the present embodiment, each illumination light causes a shadow in an area half the width of the groove. Therefore, by using an edge extraction filter having a size considering the number of pixels from the edge, the groove can be extracted accurately and the groove can be extracted. Information other than markings such as surface patterns and scratches smaller than the width of can be reduced.

Edge image generating unit 34, an image _G F, respectively Sobel filter to _{G B F} F, using _{F B,} the image _G F, to generate an edge image corresponding to the _{G B.} Moreover, the edge image generation unit 34, an image G _R drug T in which illumination light from the right direction is _irradiated, and also for the image G _L agents T which illumination light is irradiated from the left, and the Similarly, an edge image is generated by using a sobel filter according to the direction of the illumination light.

The filter used for the edge extraction filter processing in the edge image generation unit 34 is not limited to the Sobel filter, and a Laplacian filter, a canny filter, or the like can be used.

Edge image generating unit 34, four images _{G F, G B, G R} , and respectively generates the edge image with respect to G _L, the synthetic composite image generation four edge image the second image 27 Output as. The second image 27 is an image emphasizing the marking S added to the drug.

In this way, the first image acquisition unit 20A of the image acquisition unit 20 is the first by a method (first imaging method) of photographing the agent illuminated by the illumination method of uniformly illuminating the agent (illuminating the agent from all directions). obtaining one image 25 (image _{G a).}

On the other hand, the second image acquisition unit 20B of the image acquisition unit 20 acquires the second image captured by the second imaging method different from the first imaging method. That is, the second image acquisition unit 20B, a plurality of light incident direction is different (front, rear, right and left of light) to the drug by sequentially incident shot, four images _{G F,} G _B, G L, _{G R} and G _A is acquired, image processing for emphasizing the engraving added to the drug is performed, and a second image 27 emphasizing the engraving is acquired.

Needless to say, the image processing unit 30 acquires the first image 25 and the second image 27 based on the five drug images taken by the camera unit 22B below the drug out of the ten drug images. No.

The first image 25 and the second image 27 output from the image processing unit 30 are added to the storage control unit 40 shown in FIG.

As described above, the storage control unit 40 sets the first image 25 and the second image 27 as a set, and stores this set as learning data of the learning device 50 in the storage unit 42.

The illumination light source 24 of this example has four illumination light sources 24F, 24B, 24R, and 24L, and is arranged at four locations in front of, behind, to the right, and to the left of the subject, but at least at two locations. It may be arranged.

Further, the trained recognizer may be applied instead of the image processing unit 30 (particularly the edge image generation unit 34) shown in FIG.

Trained recognizer in this case, (in this example, four images _{G F, G B, G L} , G R and G _A) a plurality of images set of incident direction is different from the illumination on the subject as an input image , The image is machine-learned so as to output an image in which the marking or printing added to the subject is emphasized as a recognition result. When a plurality of image sets having different incident directions of illumination on the subject (drug) are input to the recognizer as input data to the recognizer, the recognizer outputs an image in which the marking or printing which is the recognition result is emphasized. It can be generated (output) as the second image 27.

FIG. 10 is an image diagram of a plurality of sets of learning data sets of the first image 25 and the second image 27 stored in the storage unit 42.

In the storage unit 42, a plurality of sets of the first image 25 and the second image 27 of each drug of the plurality of drugs having different marking shapes and the like are stored as one set.

As a result, the learning data creation device 10A efficiently creates a learning data set (a plurality of sets of the first image 25 and the second image 27) for causing the learning device 50 to "learn with supervised learning" and stores it in the storage unit 42. Can be made to.

Returning to FIG. 1, the learning control unit 52 of the learning device 50 uses the learning data set (learning data set stored in the storage unit 42) created by the learning data creation device 10A to machine learn the learning device 50. Let me. For example, learning that optimizes the parameters of the learner 50 (each parameter of a learning model such as CNN) is performed.

When the learning device 50 machine-learned by the learning data set of this example inputs one image of an arbitrary drug, it outputs an edge image emphasizing the marking or printing added to the drug as a recognition result. This recognition result can be used to discriminate or audit an arbitrary drug, and by synthesizing an edge image with an image of an arbitrary drug, a composite image in which the marking of the drug is easily visible can be generated. Can be used for.

<Second Embodiment of Learning Data Creation Device>
FIG. 11 is a diagram showing a main part of the second embodiment of the learning data creating device 10A shown in FIG. 1, and is particularly shown with respect to an illumination light source.

The illumination light source 60 shown in FIG. 11 is composed of a plurality of illumination light sources 60A, 60B, 60C arranged around the agent T, and the plurality of illumination light sources 60A, 60B, 60C emit light having different wavelength bands. The drug T is irradiated from three directions. In this example, the illumination light sources 60A, 60B, and 60C are assumed to emit red (R) in the wavelength band λa, green (G) in the wavelength band λb, and blue (B) in the wavelength band λc, respectively.

Further, the camera unit (not shown) has a known color image sensor having a plurality of types of pixels having sensitivity to a plurality of light (RGB light) having different wavelength bands. Each of the plurality of types of pixels (R pixel, G pixel, B pixel) of the color image sensor is provided with an RGB color filter, and each of the R pixel, G pixel, and B pixel has a wavelength as shown in FIG. It has a sensitivity corresponding to the bands λa, λb, and λc.

In the second embodiment, when the subject (drug T) is photographed, a plurality of illumination light sources 60A, 60B, and 60C are turned on at the same time to irradiate the drug T with RGB light.

In the first imaging method of acquiring the first image of the drug T, the drug T illuminated by the illumination light sources 60A, 60B, 60C is photographed by a camera unit equipped with a color image sensor, and a color image of the drug T (first). Image) is acquired.

In the second photographing method for acquiring the second image of the drug T, the drug T illuminated by the illumination light sources 60A, 60B, 60C is photographed by a camera unit equipped with a color image sensor in the same manner as in the first photographing method. A plurality of images (R image, G image, B image) are acquired corresponding to the types of pixels (RGB pixels). Then, the second image acquisition unit acquires the second image by performing image processing on each of the R image, the G image, and the B image.

For image processing on the R image, G image, and B image, three edge images are generated using an edge extraction filter according to the illumination direction of the R, G, and B lights of the illumination light sources 60A, 60B, and 60C, and 3 A composite image obtained by synthesizing the edge images of the sheets is generated as the second image.

According to the second embodiment, the first image and the second image constituting the set can be acquired by one shooting.

<Third Embodiment of the learning data creation device>
FIG. 13 is a diagram showing a main part of the third embodiment of the learning data creating device 10A shown in FIG. 1, and particularly shows the second image acquisition unit.

The second image acquisition unit 20B shown in FIG. 13 is a portion for acquiring the second image 27 of the subject, and is composed of a three-dimensional measuring device 70 and an image processing unit 72. As the first image acquisition unit 20A for acquiring the first image of the subject, for example, the one of the first embodiment or the second embodiment can be used.

The coordinate measuring device (three-dimensional information acquisition unit) 70 shown in FIG. 13 acquires three-dimensional information of a subject (drug) having a three-dimensional shape. The three-dimensional measuring device 70 may be one that scans a laser beam to acquire three-dimensional information of a drug, one that acquires three-dimensional information of a drug by a stereo camera, or the like.

The image processing unit 72 extracts the marking of the medicine based on the three-dimensional information of the medicine acquired by the coordinate measuring device 70, generates an image emphasizing the marking of the medicine, and uses the generated image as the second image 27. Output.

<Fourth Embodiment of the learning data creation device>
The first image acquisition unit and the second image acquisition unit of the fourth embodiment shoot the same subject using shooting modality with different shooting methods, and acquire a set of the first image and the second image.

As an imaging modality with a different imaging method, a CT (Computed Tomography) apparatus and an MRI (Magnetic Resonance Imaging) apparatus can be considered.

The MRI apparatus can acquire the volume data of the subject. In addition, the CT apparatus can capture a plurality of CT images at a predetermined pitch and acquire the plurality of CT images as stacked volume data.

Therefore, the images at the same cross-sectional position of each volume data obtained by photographing the same subject with the CT device and the MRI device can be used as the first image and the second image, respectively.

The image acquired by the CT device may be used as the first image, the image acquired by the MRI device may be used as the second image, the image acquired by the MRI device may be used as the first image, and the image acquired by the CT device may be used as the second image. May be good.

<Machine learning method>
FIG. 14 is a flowchart showing an embodiment of the machine learning method according to the present invention.

The machine learning method shown in FIG. 14 shows the processing contents in the machine learning method including the learning data creation method in the learning data creation device of the first embodiment.

In FIG. 14, the control unit 26 shown in FIG. 2 simultaneously lights all the illumination light sources 24F, 24B, 24R, and 24L of the illumination light source 24 (step S10), and captures the subject (drug) illuminated from all directions as a camera unit. Let's take a picture with 22A. As a result, the first image acquisition unit 20A of the image acquisition units 20 having the camera unit 22A, the illumination light source 24, and the control unit 26 acquires the first image 25 (step S12).

Subsequently, the control unit 26, the illumination light source 24F only is lit (step S14), and the illuminated drug by shooting by the camera unit 22A acquires the image _{G F} drugs by the illumination light source 24F (step S16).

Control unit 26, as in step S16, S18, illumination sources 24B, 24R, sequentially lit 24L, to shoot the drug by the camera unit 22A in synchronization sequential lighting, drug image _{G B, G} R, acquires G _L (step S20~S28).

Then, the edge image generation unit 34, step S16, S20, S24, S28 4 images _G F obtained, _{respectively,} G B, with respect to _{G R,} and _{G L,} and apply the corresponding Sobel filter 4 edge images are generated, and the edge image obtained by synthesizing the generated 4 edge images is acquired as the second image 27 (step S30). The second image 27 is an image emphasizing the marking or printing added to the drug.

The storage control unit 40 sets the first image 25 acquired in step S12 and the second image 27 acquired in step S30 as a set and stores them in the storage unit 42 (step S32).

In steps S10 to S32 described above, a set of the first image 25 and the second image 27 is acquired based on a plurality of images taken by the camera unit 22A above the drug and stored in the storage unit 42. The drug is also photographed by the camera unit 22B below the drug, and similarly, a set of the first image 25 and the second image 27 is acquired and stored in the storage unit 42. Further, the processing of steps S10 to S32 is also performed on other drugs (plurality of drugs) of different types, and as a result, a plurality of first image 25 and second image 27 corresponding to the plurality of drugs. The set will be stored in the storage unit 42.

In the phase of machine learning the learning device 50, the learning control unit 52 machine-learns the learning device 50 using the learning data set created by the learning data creation method and stored in the storage unit 42 (step S34).

[Other]
Needless to say, the present invention is not limited to the machine learning device and method of the present embodiment, but also includes a learning data creating device and method included in the machine learning device and method.

Further, the learning data created by the learning data creation device is stored in the storage unit, but the learning data stored in the storage unit is a set of a first image and a second image created by a method other than the present invention. It may be included.

Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

10 Machine learning device 10A Learning data creation device 20 Image acquisition unit 20A First image acquisition unit 20B Second image acquisition unit 21 Stage 22 Camera unit 22A Camera unit 22B Camera unit 24, 24B, 24F, 24L, 24R Illumination light source 25 First Image 26 Control unit 27 Second image 30 Image processing unit 32 Image cropping unit 34 Edge image generation unit 40 Storage control unit 42 Storage unit 50 Learner 52 Learning control unit 60, 60A, 60B, 60C Illumination light source 70 Three-dimensional measuring device 72 The image processing unit _F B, _{F F} Sobel filter _{G A, G B, G F} , G L, G R image _L B, _{L F} illumination PA photographing optical axis PB agents bandage S engraved S10~S34 step T agent TP min Packaging λa, λb, λc Wave band

Claims (23)

The first image acquisition unit that captures the subject by the first imaging method and acquires the first image,
A second image acquisition unit that acquires a second image by photographing the subject by a second imaging method different from the first imaging method.
A storage control unit in which the first image and the second image are paired and the set is stored in the storage unit as learning data of the learning device.
A learning data creation device equipped with. The second image acquisition unit includes an image processing unit that processes an image captured by the second imaging method to generate the second image.
The learning data creation device according to claim 1, wherein the storage control unit stores the first image as input data of the learning device and stores the second image as correct answer data of the learning device in the storage unit. The images taken by the second shooting method are a plurality of images in which the incident directions of the illuminations on the subject are different.
The learning data creation device according to claim 2, wherein the image processing unit generates the second image in which the marking or printing added to the subject is emphasized from the plurality of images. The image processing unit generates a plurality of edge images in which the markings or prints added to the subject are emphasized from the plurality of images, and the second image is a composite image obtained by synthesizing the generated plurality of edge images. The learning data creation device according to claim 3, which is generated as. The image processing unit includes a trained recognizer that takes the plurality of images as input images and outputs an image in which the marking or printing emphasized added to the subject is emphasized as a recognition result, and outputs the plurality of images to the recognizer. The learning data creation device according to claim 3, wherein the recognition result of the recognizer is generated as the second image. The learning data creation device according to claim 1 or 2, wherein the subject is a drug to which a stamp or print is added. The learning data creating device according to claim 6, wherein the first shooting method and the second shooting method have different lighting methods for the subject. In the illumination method of the first photographing method, light is incident on the subject from all directions.
The learning data creation device according to claim 7, wherein the illumination method of the second photographing method sequentially injects a plurality of lights having different incident directions with respect to the subject. The first image acquisition unit and the second image acquisition unit include a plurality of illumination light sources arranged around the subject and an illumination control unit that controls lighting of the plurality of illumination light sources.
The learning according to claim 8, wherein the illumination control unit simultaneously lights the plurality of illumination light sources as the illumination method of the first imaging method, and sequentially lights the plurality of illumination light sources as the illumination method of the second imaging method. Data creation device. The learning data creation device according to claim 9, wherein the plurality of illumination light sources are arranged at at least two locations in front of, behind, to the right, and to the left of the subject. The first image acquisition unit and the second image acquisition unit include a camera unit for photographing the subject, a camera control unit for controlling the camera unit, and an image processing unit.
When the subject is illuminated by the lighting method of the first shooting method, the camera control unit causes the camera unit to shoot the subject, and the subject is sequentially illuminated by the lighting method of the second shooting method. , The subject is photographed each time the lighting is sequentially performed.
The learning data according to any one of claims 8 to 10, wherein the image processing unit performs image processing on a plurality of images taken by the camera unit each time the images are sequentially illuminated to generate the second image. Creation device. The subject is engraved or printed.
The learning data creation device according to claim 11, wherein the image processing unit performs image processing for emphasizing the marking or printing added to the subject, and generates the second image in which the marking or printing is emphasized. The first image acquisition unit and the second image acquisition unit are a plurality of illumination light sources arranged around the subject, and are a plurality of illumination light sources that irradiate the subject with a plurality of lights having different wavelength bands. A camera unit including a plurality of types of pixels each having sensitivity to a plurality of lights having different wavelength bands.
In the first photographing method, the subject illuminated by the plurality of illumination light sources is photographed by the camera unit to acquire the first image.
In the second shooting method, the subject illuminated by the plurality of illumination light sources is photographed by the camera unit, and a plurality of images are acquired corresponding to the plurality of types of pixels.
The learning data creation device according to claim 1 or 2, wherein the second image acquisition unit acquires the second image by image processing the plurality of images. The subject is engraved
The second image acquisition unit includes a three-dimensional information acquisition unit that acquires three-dimensional information of the subject and an image processing unit.
The image processing unit extracts the marking of the subject based on the three-dimensional information of the subject acquired by the three-dimensional information acquisition unit, and generates the second image in which the marking of the subject is emphasized. The learning data creation device described in. The learning data creation device according to claim 1, wherein the first shooting method and the second shooting method are shooting methods for shooting the subject using different shooting modality. One of the first imaging method and the second imaging method is an imaging method for photographing the subject using a CT device, and is one of the first imaging method and the second imaging method. The learning data creation device according to claim 10, wherein the other shooting method is a shooting method for shooting the subject using an MRI apparatus. With a learner
Input when the learning device is machine-learned from the first image constituting the set by using the learning data stored in the storage unit of the learning data creating device according to any one of claims 1 to 16. A learning control unit that makes the learner machine-learn by using an image and a second image constituting the set as correct answer data.
Machine learning device equipped with. The first image acquisition unit that captures the subject by the first imaging method and acquires the first image,
A second image acquisition unit that acquires a second image by photographing the subject by a second imaging method different from the first imaging method.
With a learner
A learning control unit that causes the learner to perform machine learning by combining the first image and the second image.
Machine learning device equipped with. The step that the first image acquisition unit captures the subject by the first imaging method and acquires the first image,
A step in which the second image acquisition unit acquires the second image by photographing the subject by a second imaging method different from the first imaging method.
A step in which the storage control unit sets the first image and the second image as a set and stores the set as learning data of the learning device in the storage unit.
How to create learning data including. The learning data creation method according to claim 19, wherein the subject is a drug to which a stamp or print is added. The learning data creation method according to claim 19 or 20, wherein the first shooting method and the second shooting method have different lighting methods for the subject. The learning control unit is created by the learning data creation method according to any one of claims 19 to 21, and uses the learning data stored in the storage unit to obtain a first image constituting the set as a learning device. A machine learning method including a step of machine learning a learner using the second image constituting the set as correct answer data as an input image for machine learning. The step that the first image acquisition unit captures the subject by the first imaging method and acquires the first image,
A step in which the second image acquisition unit acquires the second image by photographing the subject by a second imaging method different from the first imaging method.
A step in which the learning control unit causes the learner to perform machine learning by combining the first image and the second image.
Machine learning methods including.