JP7219023B2 - Information processing device and object determination program - Google Patents

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus and the like for determining a classification category of an object from an image of the object.

2. Description of the Related Art Techniques for detecting objects from captured images of the objects and techniques for classifying the detected objects are conventionally known. For example, in Patent Document 1 below, by recognizing the type of product and the information on the discount sticker attached to the product from an image of the product, it is possible to identify whether a wrong discount sticker is attached intentionally or by mistake. A product registration device for detection is described.

JP 2016-177419 A

In general, the size of the discount sticker is smaller than the size of the product, so the ratio of the area showing the discount sticker to the entire image area of the captured image is low. Therefore, the conventional technology as described above has a problem that it is difficult to recognize the discount sticker with high accuracy. Moreover, such a problem is not limited to devices for recognizing the content of discount stickers, but is common to devices that identify or classify objects based on images in which the objects are part of the image. be.

An aspect of the present invention has been made in view of the above problems, and an object thereof is to realize an information processing apparatus and the like capable of improving classification accuracy of an object appearing in an image as compared with conventional ones. With the goal.

In order to solve the above problems, an information processing apparatus according to an aspect of the present invention determines which of a plurality of predetermined classification categories a classification category of a target object corresponds to from a target image obtained by photographing the target object. a machine learning extraction unit for extracting a region in which the target object is shown from the target image based on the result of machine learning to generate a partial image; and based on the result of machine learning, and a classification unit for determining a classification category of the object from the partial image.

Further, in order to solve the above problems, a determination method according to an aspect of the present invention uses one or a plurality of devices to classify an object into a plurality of predetermined classification categories from target images obtained by photographing the object. A determination method for determining which classification category the object belongs to, comprising: an extraction step of extracting a region in which the target object is captured from the target image to generate a partial image based on the results of machine learning; and a classification step of determining a classification category of the object from the partial image based on the learning result.

According to one aspect of the present invention, it is possible to improve the classification accuracy of an object appearing in an image as compared with the conventional art.

1 is a block diagram showing an example of a main configuration of a determination device according to one embodiment of the present invention; FIG. FIG. 10 is a diagram showing an example in which food produced in a food production factory is photographed as an object to be judged by the judgment device; It is a flow chart which shows an example of the judgment processing which the above-mentioned judgment device performs. It is a figure which shows the example of a display of the determination result of the said determination apparatus. 4 is a flow chart showing an example of processing related to machine learning, which is started by the determination device when a predetermined number or more of new object images are stored. FIG. 10 is a diagram showing an example of an image displayed on a display device in processing related to machine learning;

[Outline of Judgment System]
An embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG. First, an overview of a determination device (information processing device) 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing an example of the main configuration of the determination device 1. As shown in FIG.

The determination device 1 determines the classification category of each object from images 201 obtained by photographing a plurality of objects. Although the details will be described below, the determination device 1 first extracts a region in which the object appears in the image 201 based on the results of machine learning, and then automatically classifies the classification category of the object from a partial image of the extracted region. It has a characteristic configuration of making decisions based on learning results. As a result, the classification category of the object can be determined with high accuracy. In addition, although the details will be described below, the determination device 1 also has a characteristic point related to the generation of training data used for the above machine learning.

The determination device 1 includes a control unit 10 that controls all units of the determination device 1 and a storage unit 20 that stores various data used by the determination device 1 . The determination device 1 also includes an input unit 30 for receiving data input to the determination device 1, a communication unit 40 for the determination device 1 to communicate with other devices, and an output unit 50 for the determination device 1 to output data. It has

The control unit 10 includes an image acquisition unit 101, a first extraction unit (extraction unit) 102, a second extraction unit (machine learning extraction unit) 103, an extraction success/failure determination unit 104, a classification unit 105, a classification success/failure determination unit 106, and visual inspection. Determination unit 107, pass/fail determination unit 108, learning unit 109, extraction accuracy determination unit 110, image processing unit 111, training data generation unit (classification unit training data generation unit, extraction unit training data generation unit) 112, and region input reception unit 113 are included. The storage unit 20 also stores an image (target image) 201 , a new target object image (training image) 202 , a classification unit training data set 203 , and an extraction unit training data set 204 .

The image acquisition unit 101 acquires a photographed image of an object for determination by the determination device 1 . In this embodiment, an image obtained by photographing an object is input to the determination device 1 via the input unit 30 and stored in the storage unit 20 as an image 201 . Therefore, the image acquisition unit 101 acquires the image 201 from the storage unit 20 . Of course, the method by which the image acquisition unit 101 acquires the image 201 is not particularly limited. For example, the image acquisition unit 101 may control an imaging device to capture an image of an object and acquire the image 201 from the imaging device.

The first extraction unit 102 extracts a region containing the object from the new object image 202, which is the original image of training data for machine learning of the second extraction unit 103 and the classification unit 105, and generates a partial image. Generate. It should be noted that area extraction (for example, output of coordinate values indicating an area in which an object is captured) and generation of partial images may be executed in separate blocks. Similar to the second extraction unit 103, the first extraction unit 102 has a function of extracting a region in which a target object is shown from an image and generating a partial image. 103 is different. It is desirable that the extraction method of the first extraction unit 102 is capable of extracting from an image from which it is difficult for the second extraction unit 103 to extract regions. For example, the first extraction unit 102 may extract the object without performing machine learning. In this embodiment, an example in which the first extraction unit 102 extracts a target object by feature point matching will be described. Of course, this is just an example, and other methods such as pattern matching can also be applied.

The second extraction unit 103 extracts a region in which the target object is shown from the image 201 and generates a partial image based on the result of machine learning. It should be noted that area extraction (for example, output of coordinate values indicating an area in which an object is captured) and generation of partial images may be executed in separate blocks. By allowing the second extraction unit 103 to perform machine learning using various images in which the appearance of the object (size, orientation, lighting condition, focus, etc.) is different, even if the appearance of the object changes, high accuracy can be obtained. Objects can be extracted.

The machine learning algorithm is not particularly limited, but when high determination accuracy is desired, it is preferable to use a machine learning algorithm such as a neural network, and in particular, it is more preferable to use a deep learning neural network for determination. In this embodiment, an example in which Yolo (You Only Look Once) is applied as the second extraction unit 103 will be described. Yolo has a function of extracting an area in which an object is captured and a function of identifying (classifying) the object.

The extraction success/failure determination unit 104 determines whether or not the extraction by the first extraction unit 102 has succeeded. Also, the extraction success/failure determination unit 104 determines whether or not the extraction by the second extraction unit 103 has succeeded. Specifically, the extraction success/failure determining unit 104 determines success/failure of the extraction based on whether the area of the extracted region is within a predetermined range. In other words, it is assumed here that the area occupied by the object in the image 201 is already known. Note that the method of judging success or failure of extraction is not limited to this example. For example, the extraction success/failure determination unit 104 causes the display device to display an image indicating the extraction result of the first extraction unit 102 (for example, an image obtained by superimposing information indicating an extraction region on an extraction source image), and the viewer of the image ( The success or failure of the extraction may be determined by having a visual inspector) input the success or failure of the extraction. Alternatively, a threshold value may be provided, and the success or failure of extraction may be determined based on whether the accuracy (probability of extraction result) calculated by the second extraction unit 103 exceeds or falls below the threshold value. The same applies to the determination of success or failure of extraction by the second extraction unit 103 . The success or failure of the extraction by the first extraction unit 102 and the success or failure of the extraction by the second extraction unit 103 may be determined in separate blocks.

The classification unit 105 determines the classification category of the object based on the partial images output by the second extraction unit 103 . This determination is made based on the result of machine learning of the correspondence relationship between the image and the classification category of the object appearing in the image. For this machine learning, it is preferable to use, as training data, partial images generated by extracting regions in which objects corresponding to a plurality of classification categories are captured from images in which the objects are captured. This makes it possible to eliminate or reduce the influence of the background of the object on machine learning. It is possible to improve the determination accuracy more than in the case of learning.

Algorithms that can be applied as the classification unit 105 are not particularly limited, but when high determination accuracy is desired, it is preferable to use a machine learning algorithm such as a neural network, particularly a deep learning neural network (for example, ResNet: Residual Network, etc.). It is more preferable that the determination is made by In this embodiment, an example will be described in which the classification unit 105 is a ResNet that uses partial images as input data and outputs scores for each of a plurality of predetermined classification categories. This score is a numerical value that indicates the degree of certainty (probability) that an object captured in a partial image corresponds to each classification category, and the classification category with the highest score is the most likely classification category for the object. Therefore, the classification category with the highest score is the determination result of the classification unit 105 .

The classification success/failure determination unit 106 determines whether the determination result of the classification unit 105 satisfies a predetermined evaluation criterion. Specifically, the classification success/failure determination unit 106 determines whether or not the evaluation criteria are satisfied by comparing the score output by the classification unit 105 with a threshold value. Then, the classification success/failure determination unit 106 determines that the classification by the classification unit 105 fails if none of the scores output by the classification unit 105 for each classification category satisfies the evaluation criteria. If there is, it is determined that the classification is successful. Note that if the score accuracy differs for each classification category, the classification success/failure determination unit 106 preferably uses a threshold value corresponding to each classification category.

When the object shown in the image 201 does not correspond to any of the predetermined classification categories, the visual judgment unit 107 displays the image 201 on the display device and accepts the input of the classification category by the visual inspector. determine the classification category of The visual determination unit 107 similarly determines the classification category of the object when the classification success/failure determination unit 106 determines that the accuracy of the determination result of the classification unit 105 is less than the threshold.

In this embodiment, an example will be described in which the display device is a display device connected to (or equipped with) a visual inspection terminal used by a visual inspector. The visual inspection terminal accepts the input of the visual inspector and notifies the determination device 1 of the input content by wire or wireless communication. Therefore, the visual determination unit 107 determines the classification category of the object by receiving this notification via the communication unit 40 . The visual inspection terminal may be placed in a facility where the object is located (for example, in the factory if the object is a product produced in a factory) or outside the facility. device or a portable device. Moreover, the number of visual inspection terminals and visual inspectors may be plural.

Note that if the visual inspector can visually see the object itself, the visual determination unit 107 receives the input of the classification category of the object by the visual inspector who visually inspects the object, and determines the classification category of the object. You can judge. In this case, it is not necessary to display the image 201 on the display device. Further, the input may be received by the visual inspection terminal and transmitted to the determination device 1 or may be received by the input unit 30 .

A pass/fail determination unit 108 collates the determination result of the classification unit 105 with information indicating an object that should be captured in the image 201 to determine whether the object is captured in the image 201 . The pass/fail determination unit 108 is used, for example, to confirm whether the correct label 6011 is attached to the food 601 when the food 601 is conveyed in the production line 600 as in the example of FIG. 2 described later. .

For example, if the classification unit 105 determines that the label is "XX Farm Wagyu" based on the image 201 captured while transporting the food 601 to which the label 6011 printed with "XX Farm Wagyu" is attached, , the correct label 6011 is affixed. Therefore, in this case, the pass/fail determination unit 108 determines that the application has passed. On the other hand, in this case, when the classification unit 105 determines that the label is “XX farm black pig”, the food 601 is conveyed with the wrong label 6011 attached, so the pass/fail determination unit 108 fails. I judge. This makes it possible to automatically detect that the food 601 with the wrong label 6011 is being conveyed. If such detection is unnecessary, the pass/fail determination unit 108 may be omitted.

The learning unit 109 determines the timing for re-learning the second extraction unit 103 based on the evaluation value (details will be described later) calculated by the extraction accuracy determination unit 110, and re-learns the second extraction unit 103 at the determined timing. I do. For re-learning of the second extraction unit 103, training data (extraction unit training data) stored in the extraction unit training data set 204 is used. Also, the learning unit 109 re-learns the classifying unit 105 using the training data (classifying unit training data) stored in the classifying unit training data set 203 . Note that the determination of the timing of re-learning and the processing of re-learning may be executed in separate blocks. Also, the re-learning of the classifying unit 105 may be performed by a block separate from the learning unit 109 .

The extraction accuracy determination unit 110 calculates an evaluation value of the extraction accuracy of the second extraction unit based on the extraction result of the first extraction unit 102 and the extraction result of the second extraction unit 103 . This evaluation value may indicate the extraction accuracy of the second extraction unit. In this embodiment, the extraction accuracy determination unit 110 determines the degree of overlap between the region extracted from the new object image 202 by the first extraction unit 102 and the region extracted from the new object image 202 by the second extraction unit 103. An example of calculating an evaluation value (IoU: Intersection over Union) will be described. IoU is 100(%) if the region extracted by the first extraction unit 102 and the region extracted by the second extraction unit completely overlap, and 0(%) if they do not overlap at all. Note that if the increase in human labor is acceptable, instead of using the extraction result of the first extraction unit 102, the result of the visual inspector extracting the area where the object is captured and the extraction of the second extraction unit 103 The IoU may be calculated based on the results.

The image processing unit 111 generates one or more images by performing predetermined image processing on the new object image 202 . As a result, the number of training images used for machine learning by the second extraction unit 103 can be increased, and the re-learning effect of the second extraction unit 103 can be enhanced. In particular, it is possible to enhance the adaptability of the second extraction unit 103 to changes in how the object is captured. The predetermined image processing may generate an image that is different from the new object image 202 and that can be used as training data for the second extraction unit 103 . For example, the image processing unit 111 may apply affine transformation to the new object image 202 to generate an image in which the new object image 202 is transformed such as by rotation without changing the content of the new object image 202 . Further, for example, the image processing unit 111 may perform image processing for changing the brightness value, contrast, etc. instead of or in addition to the affine transformation.

The training data generation unit 112 associates the training image (the new object image 202 and the image obtained by processing the new object image 202) with the region extracted by the first extraction unit 102 or the region input reception unit 113, 2 Generate training data for machine learning of the extraction unit 103 (extraction unit training data). Further, the training data generating unit 112 associates the partial image generated by the first extracting unit 102 with the classification category of the object appearing in the partial image, and trains the classification unit 105 for machine learning training data (classification unit training data). Note that the generation of the extraction unit training data and the generation of the classification unit training data may be executed in separate blocks.

When it is difficult for both the first extraction unit 102 and the second extraction unit 103 to extract the area in which the object is captured, the area input reception unit 113 causes the display device of the visual inspection terminal described above to display an image. to accept the input of the region by the viewer (visual inspector) of the image. Then, the area input reception unit 113 extracts the input area and generates a partial image. Region extraction and partial image generation may be performed in separate blocks.

The image 201 is a photographed image of the determination target, as described above. The new target object image 202 is a target object that does not correspond to any of the classification categories that the classification unit 105 has already learned, that is, a new (unlearned) target object. It is an image with The new object image 202 is used as training data for the second extraction unit 103 and the classification unit 105 .

The classifier training data set 203 is a training data set used for machine learning of the classifier 105 . Also, the extraction unit training data set 204 is a training data set used for machine learning of the second extraction unit 103 .

[Example of use of determination device]
The use of the determination device 1 is not particularly limited, but it can be used for quality control in food production factories, for example. This will be described with reference to FIG. FIG. 2 is a diagram showing an example in which food produced in a food production factory is photographed as an object to be judged by the judgment device 1. As shown in FIG.

FIG. 2(a) shows part of a food production line 600. As shown in FIG. The production line 600 includes a transport section 602 that transports food 601 in the direction of the white arrow in the figure. Further, the transport unit 602 is provided with an imaging table 604 that holds the imaging device 603 above the transport unit 602 . As a result, the food 601 being conveyed by the conveying section 602 can be photographed from directly above.

FIG. 2(b) shows an example of an image 201 taken by the imaging device 603 of FIG. 2(a). This image 201 shows food 601, which is packed meat. The image 201 also shows a label 6011 and a price tag sticker 6012 attached to the pack of food 601 . The label 6011 is affixed as an indicator of the food 601 and may indicate at least one of the manufacturer, producer, and brand of the food 601, for example. The price tag sticker 6012 indicates the price of the food 601 and the like, and a bar code or the like is also printed on the price tag sticker 6012 .

By determining the type of the label 6011 attached to the food 601 on the production line 600 by the determination device 1, it is possible to confirm whether the label 6011 attached to the food 601 is correct. As in the example of FIG. 2B, although the area of the label 6011 occupying the image 201 is small, the determination apparatus 1 extracts the area of the label 6011 and determines the type of the label 6011. Highly accurate judgment becomes possible. As a result, it is possible to prevent problems such as misidentification of the quality of the food 601 due to incorrect attachment of the label 6011 .

Since the price tag sticker 6012 is similar in appearance to the label 6011, in order to accurately extract the region where the label 6011 is shown, the determination device 1 should have a function of identifying the label 6011 and the price tag sticker 6012. It is preferable to

For example, in this embodiment, Yolo is used as the second extraction unit 103. Since Yolo has a function to identify (classify) objects, this function can be used to classify objects at the time of extraction. You can do it. In this case, the determination device 1 generates a partial image by extracting an area whose classification result is a label. As a result, it is possible to prevent the area where the price tag sticker 6012 is shown from being mistakenly extracted, and to improve the accuracy of extracting the label area. In this case, machine learning is performed in advance for the Yolo to be used so that the label 6011 and the price tag sticker 6012 can be classified.

Also, for example, the classification unit 105 (ResNet) may be machine-learned so that it can classify the label 6011 and the price tag sticker 6012 . In this case, even if the region of the price tag sticker 6012 is extracted by the second extraction unit 103 or the first extraction unit 102, the classification unit 105 classifies the object appearing in that region as the price tag sticker 6012. , label 6011 is unaffected.

Alternatively, the price tag sticker 6012 and the label 6011 can be identified, for example, based on the position of the price tag sticker 6012 or label 6011 (the position within the area where the food 601 is shown). For example, if it is decided that the price tag sticker 6012 should be attached to the lower right corner of the food 601 when viewed from the front, the first extraction unit 102 and the second extraction unit 103 extract the area other than the lower right corner of the food 601. may be extracted as a label region. Further, for example, if it is decided to affix the label 6011 to the diagonal position of the price tag sticker (the upper left corner of the food 601), the first extraction unit 102 and the second extraction unit 103 select the area of the upper right corner of the food 601. You may extract the label region from

[Flow of judgment processing]
The flow of determination processing executed by the determination device 1 will be described with reference to FIG. FIG. 3 is a flow chart showing an example of determination processing (determination method) executed by the determination device 1 . In the following, an example will be described in which the determination device 1 determines the classification category of the label attached to the surface of the packed food.

In S<b>1 , the image acquisition unit 101 acquires the image 201 stored in the storage unit 20 . This image 201 may be, for example, an image of food 601 captured by an imaging device 603 (see FIG. 2). Also, the pass/fail determination unit 108 acquires a label ID indicating the type of the label. In addition, in the production line 600 as shown in FIG. 2, since food to which a predetermined label is attached is conveyed, the label ID of the label attached to the conveyed food can be acquired. However, since it is possible that another type of label is attached by mistake, the determination device 1 performs determination to detect the food 601 to which the wrong label is attached.

In S2, the image acquisition unit 101 determines whether the image 201 acquired in S1 is an image of a new label. A new label is a label that has not undergone machine learning using the label image. Whether or not the image is a new label image can be determined, for example, by comparing the label ID acquired in S1 with the label ID of each label used in machine learning. If it is determined in S2 that the label is a new label (YES in S2), the process proceeds to S3. On the other hand, if it is determined in S2 that the label is not a new label (NO in S2), the process proceeds to S5.

In S3, the image acquisition unit 101 stores the image 201 acquired in S1 in the storage unit 20 as an image of the new label. The new object image 202 in FIG. 1 corresponds to this new label image. As described above, new object images 202 are used to generate training data for machine learning.

In S4, the visual determination unit 107 performs visual determination. Specifically, the visual determination unit 107 displays the image 201 (or the new object image 202 stored in S3) on the display device of the visual inspection terminal described above, and accepts the input of the classification category by the visual inspector. determines the classification category of the object. Since the determination device 1 is provided with the visual determination unit 107, accurate determination can be performed even when the classification category of the new label or the determination result of the second extraction unit 103 and the classification unit 105 cannot be obtained with sufficient accuracy. It becomes possible to do

In addition, the visual judgment unit 107 makes a visual inspector judge whether the label is acceptable (judgment as to whether or not the correct label is attached), and receives the input of the judgment result. After that, the visual determination unit 107 may cause the output unit 50 to output the determination result. The determination process ends when the process of S4 ends.

Further, the visual determination unit 107 may cause the visual inspector to input the area of the object. Then, the training data generation unit 112 may associate the input region with the image to be determined by the visual determination unit 107 and add them to the extraction unit training data set 204 . Furthermore, the training data generation unit 112 may associate the input partial image of the region with the input classification category and add them to the classification unit training data set 203 .

In S5 (extraction step), the second extraction unit 103 extracts the label area, which is the area in which the label appears, from the image 201 acquired in S1. Then, the second extraction unit 103 generates a partial image of the image 201 based on the extraction result. This partial image is an image that includes the label area portion of the image area of the image 201 . Note that the partial image preferably includes the entire label area portion and does not include other portions. However, as long as it does not greatly affect the determination accuracy of the classification unit 105, it is permissible for the partial image to lack a part of the label region or include a portion other than the label. Moreover, the computational processing in this extraction may increase the processing load. Therefore, the second extraction unit 103 may cause an external server to perform this arithmetic processing and obtain the extraction result from the server.

In S6, the extraction success/failure determination unit 104 determines whether or not the extraction in S5 was successful. This determination may be made, for example, based on the relationship between the area of the region extracted in S5 and the known label area. Specifically, the extraction success/failure determining unit 104 determines that the extraction is successful if the area of the region extracted in S5 is within a predetermined range of the known label area, and determines that the extraction is unsuccessful if it is outside the predetermined range. may If the extraction is determined to be successful in S6 (YES in S6), the process proceeds to S7. On the other hand, if it is determined in S6 that the extraction has failed (NO in S6), the process proceeds to S4.

In S7 (classification step), the classification unit 105 determines the classification category of the label. Specifically, the classification unit 105 uses the partial image generated in S5 as input data, and for each of a plurality of preset classification categories (types of labels), calculates a score indicating the probability of matching the classification category. Output. Since the partial image is used as the input data, a highly accurate classification result can be obtained compared to the configuration in which the image 201 acquired in S1 is used as the input data. It should be noted that the arithmetic processing in this determination may increase the processing load. For this reason, the classification unit 105 may cause an external server to perform this calculation process, and acquire the score, which is the calculation result, from the server.

In S8, the classification success/failure determination unit 106 determines whether or not the classification in S7 was successful. Specifically, the classification success/failure determination unit 106 determines whether or not the determination result of S7 includes a score equal to or greater than a threshold. If the scores above the threshold are included, the classification is successful, and if not included, the classification is unsuccessful. In S8, if the classification success/failure determination unit 106 determines that the classification is successful (the scores equal to or greater than the threshold are included) (YES in S8), it notifies the classification category with the highest score to the success/failure determination unit 108, and the process proceeds to S9. proceed to On the other hand, if it is determined in S8 that the classification has failed (the score above the threshold is not included) (NO in S8), the process proceeds to S4.

In S9, the pass/fail judgment unit 108 makes a pass/fail judgment of the label based on the classification result of S7. Specifically, if the classification category notified from the classification success/failure determination unit 106 (the classification category with the highest score) matches the type indicated by the label ID obtained in S1, the pass/fail determination unit 108 determines that the classification is passed. Then, the judgment processing ends. On the other hand, if they do not match, the pass/fail determination unit 108 determines that the pass/fail is rejected, and the process proceeds to S4. As a result, for images 201 that have been determined to be unacceptable, if the determination by the pass/fail determination unit 108 (classification by the classification unit 105) is erroneous, the correct classification category is determined by visual inspection. Therefore, the correct classification category determined by visual inspection can be associated with the image 201 and used as training data for the classification unit 105 . In addition, when the determination by the pass/fail determination unit 108 is erroneous, it can be detected by visual inspection. Note that the pass/fail determination unit 108 may cause the output unit 50 to output the determination result.

Note that the mode of outputting the determination result is arbitrary. For example, if the output unit 50 is a display device, an image indicating pass or fail may be displayed. Moreover, you may display the score of a classification category. In addition, output may be performed in a mode other than display. For example, in the case of rejection, by outputting a sound such as a buzzer, it may be notified that there is food 601 with an incorrect label attached. Further, for example, if the production line 600 is provided with a mechanism for branching the conveying route of the food 601 being conveyed, the mechanism is controlled to exclude the food 601 determined to be rejected from the production line 600. may

[Display example of judgment result]
As described above, the pass/fail determination unit 108 may cause the output unit 50 to output the determination result. Here, an example of displaying and outputting the determination result of the classification unit 105 will be described with reference to FIG. FIG. 4 is a diagram showing a display example of determination results.

The determination results in the display example of FIG. 4 include the top three classification categories (first to third candidates) with the highest scores and their score values. Such a display allows the user to recognize the certainty of the determination result. In this example, the score of the label "XX Farm Wagyu" of the first candidate is sufficiently large compared to the second and third candidates, thereby making the user aware that this judgment result is reliable. be able to. The display example of FIG. 4 also includes images of the labels of the first to third candidates. This makes it possible for the user to easily recognize the appearance of the label as a result of determination.

[Flow of processing related to machine learning using new object images]
In the determination device 1, when a predetermined number (for example, 10) or more new object images 202 are stored, processing related to machine learning is performed. This will be described with reference to FIG. FIG. 5 is a flowchart showing an example of processing related to machine learning that is started when a predetermined number or more of new object images 202 are stored. It should be noted that the predetermined number of new object images 202 are all images containing labels belonging to the same classification category. Further, the fact that a predetermined number of new object images 202 have been stored may be configured to be detected by the image acquiring unit 101, for example.

In S21, the first extraction unit 102 extracts a label region from the new object image 202, which is the image of the new label. Then, the first extraction unit 102 generates a partial image of the new object image 202 based on the extraction result. These processes are performed for each new object image 202 .

In addition, in order to improve the extraction accuracy of S21, preprocessing may be performed on the new object image 202 before the processing of S21. For example, if the product to which the new label is affixed is meat packaged with a transparent film, it is presumed that the red (meat color) area is not the label area. Therefore, by detecting a red region in the new target object image 202 and excluding that region from the extraction target as preprocessing of S21, the extraction range can be appropriately narrowed down. Accuracy can be improved.

In S22, the extraction success/failure determination unit 104 determines whether the extraction in S21 succeeded or failed. For this determination, for example, the new object image 202 superimposed with the information indicating the extraction result of S21 is displayed on the display device of the above-described visual inspection terminal, the visual inspector judges whether the extraction is successful or not, and the judgment result is input. may be done by accepting Of course, the extraction success/failure determination unit 104 may determine the success/failure of extraction based on the area of the extracted region, and in this case, determination by a visual inspector is unnecessary. However, if the extraction is determined to be successful in S22, the partial image based on the extraction result is used as training data for the classification unit 105 as it is. The same applies to the determination in S25, which will be described later.

In S23, the training data generation unit 112 associates the partial images of the new target object image 202 determined to be successfully extracted in S22 with the label classification categories, and creates a training data set for the classification unit 105 (“classification unit training data set 203''). As described above, the determination device 1 automatically extracts the label region and generates the training data, so that it is possible to significantly reduce the labor involved in generating the training data for the classifier.

In S24, the second extraction unit 103 extracts a label region from the new target object image 202 determined as extraction failure in S22, and generates a partial image. If there are a plurality of new object images 202 determined to be unsuccessful in extraction in S22, the second extraction unit 103 extracts label regions from each of them to generate partial images. Note that the outer shape of the label 6011 is often the same even if the classification category changes, so there is a possibility that the second extraction unit 103 can accurately extract the label area even from the new object image 202. . In this way, when the first extraction unit 102 fails to extract, the second extraction unit 103 extracts, so that the label area can be automatically extracted from most of the new object image 202. Become.

In S25, similarly to S22, the extraction success/failure determination unit 104 determines whether the extraction in S24 succeeded or failed. The processing of S27 is performed for the new target object image 202 determined to be successfully extracted in S25 (YES in S25). On the other hand, the processing of S26 is performed for the new target object image 202 determined to be unsuccessful in extraction in S25 (NO in S25).

Here, in S25, the extraction success/failure determination unit 104 displays, for example, an image such as (a) in FIG. good. FIG. 6 is a diagram showing an example of an image displayed by the extraction success/failure determination unit 104. As shown in FIG. The image of (a) of FIG. 2 includes a new object image 202 in which the label 6011 is shown. In this image, a frame line 301 indicating the label area extracted by the second extraction unit 103 is displayed superimposed on the new object image 202 .

A visual inspector checks such a display whether the label 6011 is within the frame line 301, and if it is within the frame line 301, whether the frame line 301 includes many areas other than the label 6011. do. If any of the confirmation results are positive, it is determined that the extraction is successful, and the fact that the extraction is successful is input to the input unit 30. On the other hand, if any of the confirmation results are negative, it is determined that the extraction has failed. to input to the input unit 30 that the extraction has failed. It should be noted that the input unit 30 may be configured to input only when the extraction fails. In this case, the extraction success/failure determination unit 104 determines that the extraction is successful if the visual inspector's input is not confirmed within a predetermined time.

In S26, the area input reception unit 113 extracts the area input by the visual inspector as a label area. For example, when an image as shown in FIG. 6(a) is displayed, the visual inspection terminal receives an input operation on the frame line 301, specifies the label area, and notifies the determination device 1 of the label area. . Then, the area input reception unit 113 extracts the notified label area.

In this case, the position and range of the frame line 301 can be arbitrarily changed by an input operation on the visual inspection terminal. As a result, the visual inspector changes the position and range of the frame line 301 by performing an input operation on the visual inspection terminal so that the label 6011 matches the area where the label 6011 is shown, as shown in FIG. 6B. can do. In this state, when the visual inspector performs an operation to confirm the label area, the visual inspection terminal can specify the image area indicated by the frame line 301 at the time of the operation as the label area.

In S27, the training data generation unit 112 converts the partial images of the new object image 202 determined to be extracted successfully in S25 and the partial images of the new object image 202 from which the label region was extracted in S26 into a classification unit training data set. 203. As in S23, the training data generation unit 112 associates and adds the partial image and the classification category of the label.

In S<b>28 , the image processing unit 111 processes the new object image 202 . The new object images 202 to be processed are the new object images 202 determined to be extracted successfully in S22, the new object images 202 determined to be extracted successfully in S25, and the new object images from which the label regions were extracted in S26. 202.

In S29, the training data generation unit 112 adds the area extracted in S21, S24, or S26 to the new object image 202 after processing in S28 (which may include the new object image 202 before processing). Generate extractor training data in correspondence. Then, the generated extractor training data is added to the extractor training data set 204 . As described above, the determination device 1 automatically extracts the label region and generates the training data, so that the effort required to generate the extraction unit training data can be significantly reduced.

In S30, an extraction test of the second extraction unit 103 is performed. In this extraction test, the second extraction unit 103 extracts a label region from the new object image 202 after processing in S28. If there are a plurality of new object images 202 after processing in S28, the second extraction unit 103 extracts a label region from each of them.

In S31, the extraction accuracy determination unit 110 calculates an evaluation value indicating the accuracy of the extraction in S30, and the learning unit 109 determines the timing for re-learning the second extraction unit 103 based on this evaluation value. Specifically, the extraction accuracy determination unit 110 calculates an evaluation value (IoU) indicating the degree of overlap between the label area extracted in S21 and the label area extracted by the second extraction unit 103 in S30. Since the new target object image 202 is an image that is not used for machine learning by the second extraction unit 103, basically the first extraction unit 102 has higher extraction accuracy from the new target object image 202. . Therefore, it can be said that the higher the IoU, the higher the extraction accuracy of the second extraction unit 103 . Also, in S31, the extraction accuracy determination unit 110 may calculate an evaluation value indicating the degree of overlap between the label area extracted in S24 or S26 and the label area extracted by the second extraction unit 103 in S30.

Then, the learning unit 109 determines whether or not the calculated evaluation value is equal to or greater than a predetermined threshold. The threshold may be appropriately determined according to the required determination accuracy, and may be set to 80(%), for example. Note that when label regions are extracted from a plurality of new target object images 202 in S30, in S31 the extraction accuracy determination unit 110 calculates an index for each extraction result, and the learning unit 109 determines if the average value is equal to or greater than the threshold. You can determine if there is If it is determined in S31 that it is equal to or greater than the threshold (YES in S31), the processing of FIG. 5 ends. On the other hand, if it is determined to be less than the threshold in S31 (NO in S31), the process proceeds to S32.

In S32, the learning unit 109 re-learns the second extraction unit 103 using the extraction unit training data set 204, thereby ending the processing in FIG. Since the extraction unit training data set 204 includes an image obtained by processing the new object image 202, this re-learning allows the second extraction unit 103 to accurately extract the label region from the image of the new label. becomes possible. In this way, since the determination device 1 automatically determines the timing of re-learning of the second extraction unit 103, the user of the determination device 1 does not need to determine the timing of re-learning, and re-learning can be performed at an appropriate timing. It is possible to maintain the judgment accuracy.

Note that if an increase in the workload of the visual inspector is allowed compared to the example of FIG. , the visual inspector may enter the region of the object. Then, the training data generation unit 112 may associate the input region with the image to be determined by the visual determination unit 107 and add them to the extraction unit training data set 204 . Furthermore, the training data generation unit 112 may associate the input partial image of the region with the input classification category and add them to the classification unit training data set 203 .

[Other application examples]
The determination device 1 can be used to determine the classification category of any object, not limited to the label attached to the food. In particular, since the determination device 1 includes the second extraction unit 103, it is suitable for determining the classification category of an object that occupies a narrow area in the image. For example, it can be suitably used for determining classification categories of tags attached to clothing.

The determination device 1 also generates extraction unit training data based on the extraction result of the first extraction unit 102 and determines the extraction accuracy of the second extraction unit 103 . From this point of view, it can be said that it is preferable that the first extraction unit 102 can easily extract the image of the target object or the determination target. For example, when the first extraction unit 102 performs extraction by feature point matching, the target object preferably has a simple outer shape such as a rectangle or a circle. It is preferable that the

As a specific example, the determination device 1 can also be used to extract a region in which a sign is captured from an image of a road surface and determine the classification category of the sign. Further, for example, the determination device 1 can be used for the purpose of extracting a region in which a receipt is shown or a specific item in the receipt from an image of the receipt and determining the contents thereof. In this case, the classification unit 105 preferably uses a machine learning algorithm with high character recognition performance, such as TextBoxes. The same applies to pieces of paper such as business cards. In this way, the determination device 1 can also classify an object on which characters are drawn as a determination target, taking into consideration the contents of the characters.

[Example of system implementation]
The function of the determination device 1 can also be realized by a system composed of a plurality of mutually communicable devices. In this case, each of the processes described above is performed by one of the plurality of devices that constitute the system. For example, an information processing apparatus including the first extraction unit 102 and the training data generation unit 112 may be configured to generate classification unit training data, and the determination device 1 may be configured to perform machine learning using the classification unit training data. good. Further, for example, an information processing device including the extraction accuracy determination unit 110 and the learning unit 109 may be configured to determine the timing for re-learning the second extraction unit 103, and the determination device 1 performs re-learning at the determined timing. It is good also as a structure to carry out. The re-learning may be performed under the control of the information processing device, or may be performed by the user operating the determination device 1 .

[Example of realization by software]
The control block of the determination device 1 (especially each unit included in the control unit 10) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software. .

In the latter case, the determination device 1 includes a computer that executes commands of a program (object determination program) that is software for realizing each function. This computer includes, for example, one or more processors, and a computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. In addition, a RAM (Random Access Memory) for developing the above program may be further provided. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

[Additional notes]
An information processing apparatus according to an aspect of the present invention is an information processing apparatus that determines to which of a plurality of predetermined classification categories a classification category of an object corresponds from a target image obtained by photographing the object. a machine learning extraction unit for extracting a region in which the object is captured from the target image based on the result of machine learning and generating a partial image; and extracting the object from the partial image based on the result of machine learning. and a classification unit that determines the classification category.

The classification unit performs machine learning using partial images generated by extracting regions in which objects corresponding to each of the plurality of classification categories are shown as training data from images in which the objects are shown. The classification category of the object may be determined based on.

When the object in the target image does not correspond to any of the plurality of classification categories, or when the accuracy of the determination result of the classification unit is less than a threshold, the information processing device determines whether the target image or The configuration may include a visual determination unit that determines the classification category of the object by receiving an input of the classification category by the viewer of the object.

The information processing device includes an extraction unit that extracts a region in which the object is shown from a training image that is a source of classification unit training data for machine learning of the classification unit to generate a partial image, and the extraction unit. and a classification section training data generation section that generates the classification section training data by associating the partial image generated by with the classification category of the object appearing in the partial image.

In the information processing device, when the extraction unit fails to extract the region from the training image, the machine learning extraction unit extracts the region from the training image to generate the classification unit training data. may be configured to generate a partial image to be used for.

The information processing device includes an extraction accuracy determination unit that calculates an evaluation value of the extraction accuracy of the machine learning extraction unit based on the extraction result of the extraction unit and the extraction result of the machine learning extraction unit; and a learning unit that determines the timing of re-learning of the machine learning extraction unit.

The information processing device includes an extraction unit training data generation unit that generates extraction unit training data for machine learning of the machine learning extraction unit by associating the training image with the region extracted by the extraction unit. It may be a configuration that

In the information processing device, the extraction unit training data generation unit generates each of one or more images generated by performing predetermined image processing on the training image, and the region extracted by the extraction unit. may be associated with each other to generate the extraction unit training data.

A determination method according to an aspect of the present invention determines to which of a plurality of predetermined classification categories a classification category of an object corresponds from a target image obtained by photographing the object using one or more devices. A determination method, comprising: an extraction step of extracting a region in which the object is captured from the target image based on a result of machine learning to generate a partial image; and a classification step of determining the classification category of the object.

An object determination program for causing a computer to function as the information processing apparatus, the object determination program for causing the computer to function as the machine learning extraction section and the classification section is also included in the scope of the present invention.

An information processing apparatus according to another aspect of the present invention is an information processing apparatus that generates classification unit training data used for machine learning of a classification unit that determines, from an image, a classification category of an object appearing in the image, An extraction unit that extracts a region in which the object appears from a training image that is the basis of the classification unit training data and generates a partial image, a partial image generated by the extraction unit, and the object that appears in the partial image and a classifier training data generation unit that generates the classifier training data by associating the classification categories with the classification categories.

An information processing apparatus according to still another aspect of the present invention is an information processing apparatus that determines re-learning timing of a machine-learning extraction unit that extracts a region in which a target object is shown from an image based on a result of machine learning. Based on the extraction result of the extraction unit that extracts the area in which the object is shown from the image and the extraction result that the machine learning extraction unit extracts the area in which the object is shown from the image , an extraction accuracy determination unit that calculates an evaluation value of the extraction accuracy of the machine learning extraction unit; and a learning unit that determines timing for re-learning the machine learning extraction unit based on the evaluation value.

1 Determination device (information processing device)
102 first extraction unit (extraction unit)
103 second extraction unit (machine learning extraction unit)
105 classification unit 107 visual determination unit 109 learning unit 110 extraction accuracy determination unit 112 training data generation unit (classification unit training data generation unit, extraction unit training data generation unit)
201 images (target images)

Claims (7)

An information processing device for determining to which of a plurality of predetermined classification categories a classification category of an object corresponds from a target image obtained by photographing the object,
A machine that generates a partial image by extracting an area in which the object appears from the target image based on the results of machine learning of the relationship between the image and the coordinate values indicating the area in the image in which the object appears. a learning extractor;
a classification unit that determines the classification category of the object from the partial image based on the results of machine learning of the correspondence relationship between the image and the classification category of the object that appears in the image;
an extracting unit that extracts a region in which the object is shown from a training image that is a source of training data for the classifying unit for machine learning of the classifying unit and generates a partial image;
a classification unit training data generation unit that generates the classification unit training data by associating the partial image generated by the extraction unit with the classification category of the object appearing in the partial image;
When the extraction unit fails to extract the region from the training image, the machine learning extraction unit extracts the region from the training image to generate a partial image used to generate the classification unit training data. death,
The information processing apparatus, wherein the extracting unit extracts the area in which the object is captured without using machine learning . The classification unit performs machine learning using partial images generated by extracting regions in which objects corresponding to each of the plurality of classification categories are captured from images in which the objects are captured as training data. 2. The information processing apparatus according to claim 1, wherein the classification category of the object is determined based on. When the object in the target image does not fall under any of the plurality of classification categories, or when the accuracy of the determination result of the classification unit is less than a threshold, a viewer of the target image or the object 3. The information processing apparatus according to claim 1, further comprising a visual judgment unit for judging the classification category of the object by receiving an input of the classification category. an extraction accuracy determination unit that calculates an evaluation value of the extraction accuracy of the machine learning extraction unit based on the extraction result of the extraction unit and the extraction result of the machine learning extraction unit;
4. The information processing apparatus according to any one of claims 1 to 3, further comprising a learning unit that determines timing for re-learning by the machine learning extraction unit based on the evaluation value. An extraction unit that associates the training image with the region extracted by the extraction unit or the region extracted by the machine learning extraction unit to generate extraction unit training data for machine learning of the machine learning extraction unit. 5. The information processing apparatus according to any one of claims 1 to 4 , further comprising a training data generator. The extraction unit training data generation unit extracts each of one or more images generated by performing predetermined image processing on the training image and the region extracted by the extraction unit or extracted by the machine learning extraction unit. 6. The information processing apparatus according to claim 5 , wherein the extraction unit training data is generated by associating the region with the extracted region. 2. An object determination program for causing a computer to function as the information processing apparatus according to claim 1, wherein the computer functions as the machine learning extraction unit , the classification unit , the extraction unit, and the classification unit training data generation unit . Object determination program for

Images