JP2022070747A - Information processing apparatus and information processing method - Google Patents

Abstract

To provide a technique for enabling processing by a learning model according to a situation even if processing is difficult based on only information collected in the past, or even if there is no information collected in the past.SOLUTION: One or more learning models are selected, as candidate learning models, from a plurality of learning models learned under learning environments different from each other based on information concerning image capturing of an object. One or more candidate learning models are selected from the candidate learning models based on a result of object detection processing by the selected candidate learning models. The object detection processing is performed using a candidate learning model of at least one of the selected candidate learning models.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for prediction based on captured images.

In recent years, in agriculture, efforts have been actively made to solve problems by using IT in order to help solve various problems such as yield prediction, optimum harvest time prediction, pesticide spraying amount control, and field restoration planning. There is.

For example, in Patent Document 1, the growth situation and the harvest prediction can be grasped at an early stage by appropriately referring to the sensor information acquired from the place where the crop is grown and the database storing the information, and the abnormal state of the growth can be detected at an early stage. And how to deal with it is disclosed.

Further, in Patent Document 2, field management is performed to suppress variations in the quality and yield of agricultural products by making arbitrary inferences by referring to the registered information based on the information acquired from various sensors related to agricultural products. The method is disclosed.

Japanese Unexamined Patent Publication No. 2005-137209 Japanese Unexamined Patent Publication No. 2016-49102

However, in the method that has been proposed conventionally, a sufficient number of cases acquired in the past regarding the field where the prediction etc. are carried out are retained, and adjustment work has been completed so that the predicted items can be estimated accurately based on the information on the cases. Is the premise.

On the other hand, the quality of agricultural products is generally greatly affected by environmental changes such as weather and climate, and varies greatly depending on the spraying conditions of fertilizers, pesticides, etc. by workers. If the conditions due to all external factors are unchanged every year, it is not even necessary to predict the yield and harvest time, but unlike industry, agriculture has many external factors that workers cannot control themselves. , Prediction is very difficult. In addition, when predicting the yield, etc. when inexperienced weather continues, it is difficult to make a correct prediction with the estimation system adjusted from the above-mentioned cases acquired in the past.

The most difficult case is when the above prediction system is newly introduced in the field. For example, consider the case of predicting the yield in a specific field and detecting a non-producing area for the purpose of repairing a poorly growing area (dead branch / lesion). For these tasks, images and parameters related to crops collected in the past in the above fields are usually stored in a database. Then, when actually making predictions for the field, adjustments are made by mutually referring to the observed images taken in the current field and other data related to growth information acquired from the sensor, and making adjustments. Predict well. However, as described above, when these prediction systems and non-production area detectors are introduced into different new fields, they cannot be applied immediately because the conditions (of the fields) often do not match. In such cases, it was necessary to collect and adjust a sufficient number of data in the new field.

In addition, when the above prediction system and the non-production area detector are manually adjusted, the parameters related to the growth of the crop become high-dimensional, so that it takes a lot of time and effort. In addition, even when deep learning or a machine learning method similar to them is used, manual labeling (annotation) work is usually required to achieve good performance for new inputs. Therefore, the work cost is large.

Originally, it is preferable to perform good prediction / estimation with simple settings that reduce the load on the user, even when a new prediction system is introduced, or even in the case of a natural disaster or weather that has never occurred in the past.

In the present invention, there is a technique for enabling processing by a learning model according to a situation even when it is difficult to process only from the information collected in the past or when there is no information collected in the past. offer.

The uniformity of the present invention includes a first selection means for selecting one or more learning models as candidate learning models from a plurality of learning models learned in different learning environments based on information related to image shooting of an object, and the first selection means. A second selection means that selects one or more candidate learning models from the candidate learning model based on the result of object detection processing by the candidate learning model selected by the selection means, and a candidate learning model selected by the second selection means. It is characterized by comprising a detection means for performing an object detection process on a captured image of the object by using at least one of the candidate learning models.

According to the configuration of the present invention, even when it is difficult to process only from the information collected in the past or when there is no information collected in the past, it is possible to process by the learning model according to the situation. be able to.

The figure which shows the configuration example of a system. A flow chart of the processing performed by the system. The flowchart which shows the detail of the process in step S23. The flowchart which shows the detail of the process in step S233. The figure which shows an example of the image | photographing method of a field by a camera 10. The figure which shows the difficult case. The figure which shows the result of performing the annotation work on the photographed image. The figure which shows the display example of GUI. The figure which shows the display example of GUI. A flow chart of the processing performed by the system. The flowchart which shows the detail of the process in step S83. The flowchart which shows the detail of the process in step S833. The figure which shows the detection example of the detection area. The figure which shows the display example of GUI. (A) is a diagram showing a configuration example of a query parameter, (B) is a diagram showing a configuration example of a parameter set of a learning model, and (C) is a diagram showing a configuration example of a query parameter.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

[First Embodiment]
In the present embodiment, a system for performing analysis processing of the field such as prediction of the yield of agricultural products in the field and detection of repaired parts from the photographed image of the field taken by the camera will be described.

First, a configuration example of the system according to the present embodiment will be described with reference to FIG. As shown in FIG. 1, the system according to the present embodiment includes a camera 10, a cloud server 12, and an information processing device 13.

First, the camera 10 will be described. The camera 10 captures a moving image of the field and outputs an image of each frame in the moving image as a “photographed image of the field”. Alternatively, the camera 10 regularly or irregularly captures a still image of the field, and outputs the captured still image as a “photographed image of the field”. In order to accurately make the predictions described below from the captured images, it is desirable that the images captured in the same field are captured in the same environment and conditions as much as possible. The captured image output from the camera 10 is transmitted to the cloud server 12 and the information processing device 13 via a communication network 11 such as a LAN or the Internet.

The method of photographing the field with the camera 10 is not limited to a specific photographing method. An example of a method of photographing a field with a camera 10 will be described with reference to FIG. 3 (A). In FIG. 3A, the camera 33 and the camera 34 are used as the camera 10. In a general field, crop trees systematically planted by farmers are lined up, for example, as shown in FIG. 3A, such as a row of crop trees 30 and a row of crop trees 31. In addition, many rows of crop trees are planted side by side. The agricultural work tractor 32 is provided with a camera 34 for photographing the row 31 of the crop trees on the left side in the traveling direction indicated by the arrow, and a camera 33 for photographing the row 30 of the crop trees on the right side. .. Therefore, when the agricultural work tractor 32 moves between the row 30 and the row 31 in the traveling direction indicated by the arrow, the camera 34 captures a plurality of captured images of the crop trees in the row 31, and the camera 33 captures a plurality of captured images of the agricultural product tree in the row 31. I will take multiple images of the trees of the agricultural products in.

In many fields where the agricultural tractor 32 is designed to work and crop trees are planted at regular intervals, the camera 33 installed on the agricultural tractor 32 as shown in FIG. 3 (A). By taking a picture of a crop tree at, 34, it is relatively easy to take a picture of more crop trees at a certain height and at a certain distance from the crop tree. Therefore, it is possible to take an image of all the target fields under almost the same conditions, and it is easy to take an image under desirable conditions.

If it is possible to photograph the field under substantially the same conditions, another imaging method may be adopted. An example of a method of photographing a field with a camera 10 will be described with reference to FIG. 3 (B). In FIG. 3B, the camera 38 and the camera 39 are used as the camera 10. As shown in FIG. 3B, in a field or the like where the distance between the row of crop trees 35 and the row of crop trees 36 is narrow and it is not possible to travel by a tractor, the camera 38 and the camera attached to the drone 37 are used. It may be taken by 39. The drone 37 is provided with a camera 39 for photographing the row 36 of the crop trees on the left side in the traveling direction indicated by the arrow, and a camera 38 for photographing the row 35 of the crop trees on the right side. Therefore, when the drone 37 moves between the row 35 and the row 36 in the traveling direction indicated by the arrow, the camera 39 captures a plurality of captured images of the crop trees in the row 36, and the camera 38 captures a plurality of captured images of the crop trees in the row 35. You will be shooting multiple images of the tree.

Further, a camera installed on the self-propelled robot may be used to take a photographed image of a crop tree. Further, the number of cameras used for shooting is set to 2 in FIGS. 3A and 3B, but the number is not limited to a specific number.

Regardless of the shooting method used to capture the captured image of the tree of the agricultural product, the camera 10 includes the captured information (shooting position (for example, the shooting position measured by GPS)) at the time of shooting the captured image. Exif information in which the shooting date and time, information related to the camera 10 and the like are recorded) is attached and output.

Next, the cloud server 12 will be described. The captured image and Exif information transmitted from the camera 10 are registered in the cloud server 12. In addition, a plurality of learning models (detectors / settings) for detecting an image area related to agricultural products from captured images are registered in the cloud server 12, and each learning model is a model learned in different learning environments. be. Then, the cloud server 12 selects a learning model candidate to be used when detecting an image area related to an agricultural product from a photographed image from a plurality of learning models held by self-confidence, and presents the candidate to the information processing apparatus 13.

The CPU 191 executes various processes using computer programs and data stored in the RAM 192 and the ROM 193. As a result, the CPU 191 controls the operation of the entire cloud server 12, and also executes or controls various processes described as those performed by the cloud server 12.

The RAM 192 has an area for storing computer programs and data loaded from the ROM 193 and the external storage device 196, and an area for storing data received from the outside via the I / F 197. Further, the RAM 192 has a work area used by the CPU 191 to execute various processes. As described above, the RAM 192 can appropriately provide various areas.

The ROM 193 stores setting data of the cloud server 12, computer programs and data related to the startup of the cloud server 12, computer programs and data related to the basic operation of the cloud server 12, and the like.

The operation unit 194 is a user interface such as a keyboard, a mouse, and a touch panel, and various instructions can be input to the CPU 191 by the user operating the operation unit 194.

The display unit 195 has a screen such as a liquid crystal screen or a touch panel screen, and can display the processing result by the CPU 191 with an image, characters, or the like. The display unit 195 may be a projection device such as a projector that projects an image or characters.

The external storage device 196 is a large-capacity information storage device such as a hard disk drive device. The external storage device 196 stores computer programs and data for causing the CPU 191 to execute or control the OS (operating system) and various processes described as those performed by the cloud server 12. The data stored in the external storage device 196 also includes the data related to the above learning model. The computer programs and data stored in the external storage device 196 are appropriately loaded into the RAM 192 according to the control by the CPU 191 and become the processing target by the CPU 191.

The I / F 197 is a communication interface for performing data communication with the outside, and the cloud server 12 transmits / receives data to / from the outside via the I / F 197. The CPU 191 and the RAM 192, the ROM 193, the operation unit 194, the display unit 195, the external storage device 196, and the I / F 197 are all connected to the system bus 198. The configuration of the cloud server 12 is not limited to the configuration shown in FIG.

The captured image and Exif information output from the camera 10 are temporarily stored in the memory of another device, and the captured image and Exif information are transferred from the memory to the cloud server 12 via the communication network 11. May be.

Next, the information processing apparatus 13 will be described. The information processing device 13 is a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device. The information processing apparatus 13 presents a learning model candidate presented by the cloud server 12 to the user, accepts the selection of the learning model from the user, and notifies the cloud server 12 of the learning model selected by the user. The cloud server 12 detects an image area related to agricultural products (object detection processing) from an image taken by the camera 10 by using a learning model (a learning model selected by a user from candidates) notified from the information processing device 13. Perform the above analysis process.

The CPU 131 performs various processes using computer programs and data stored in the RAM 132 and the ROM 133. As a result, the CPU 131 controls the operation of the entire information processing apparatus 13, and also executes or controls various processes described as those performed by the information processing apparatus 13.

The RAM 132 has an area for storing computer programs and data loaded from the ROM 133, and an area for storing data received from the camera 10 and the cloud server 12 via the input I / F 135. Further, the RAM 132 has a work area used by the CPU 131 to execute various processes. As described above, the RAM 132 can appropriately provide various areas.

The ROM 133 stores setting data of the information processing device 13, computer programs and data related to the activation of the information processing device 13, computer programs and data related to the basic operation of the information processing device 13, and the like.

The output I / F 134 is an interface used by the information processing apparatus 13 to output / transmit various types of information to the outside.

The input I / F 135 is an interface used by the information processing apparatus 13 to input / receive various types of information from the outside.

The display device 14 has a liquid crystal screen and a touch panel screen, and can display the processing result by the CPU 131 with images, characters, and the like. The display device 14 may be a projection device such as a projector that projects an image or characters.

The user interface 15 includes a keyboard and a mouse, and various instructions can be input to the CPU 131 by being operated by the user. The configuration of the information processing device 13 is not limited to the configuration shown in FIG. 1, and for example, it has a large-capacity information storage device such as a hard disk drive device, and a computer program or data such as a GUI described later is stored in the hard disk drive device. You may save it. Further, the user interface 15 may include a touch sensor such as a touch panel.

Next, the flow of the task of predicting the yield of the crops harvested in the field at an earlier stage than the harvest time from the photographed image of the field taken by the camera 10 will be described. When predicting the yield by simply counting the fruits to be harvested at the time of harvest, the purpose is achieved by simply detecting the target fruit from the photographed image by a method called specific object detection. To. This is a method of detecting this characteristic appearance by a learned discriminator because the fruit itself has a very characteristic appearance.

In the present embodiment, when the crop is a fruit, the yield of the fruit is predicted at a stage earlier than the harvest time, not only by counting the fruit after the fruit has matured. For example, the yield can be predicted by detecting the inflorescences that will become fruits later, and the yield can be predicted by detecting the dead branches and lesion areas where the fruit is unlikely to grow, or from the state of the overgrown leaves. Predict the yield. In order to make such a prediction, it is necessary to have a prediction method that corresponds to the difference in the growth condition of crops depending on the shooting time and climate. In other words, it is necessary to select a prediction method with good prediction performance according to the situation of the crop. In this case, it is expected that the above prediction will be made appropriately by a learning model that matches the field to be predicted.

Here, various objects shown in the photographed image are classified into classes such as a trunk class, a branch class, a dead branch class, and a prop class of a crop, and the yield is predicted by the class. Since the appearance of objects belonging to classes such as the trunk class and branch class of trees changes depending on the shooting time, it is difficult to make a universal prediction. A difficult case like this is shown in FIG.

4 (A) and 4 (B) show an example of a photographed image taken by the above camera 10. Although the cropped trees are shown at almost equal intervals in these captured images, the task of detecting the fruits from the captured images cannot be executed because the fruits to be harvested have not yet been obtained. The tree in the photographed image of FIG. 4 (A) is a crop tree photographed at a relatively early stage of the season, and the tree in the photographed image of FIG. 4 (B) is photographed at a stage where leaves are overgrown to some extent. It is a tree that has been made. In the photographed image of FIG. 4A, since all the branches of the tree have leaves to the same extent, it can be determined that there is no region with poor growth, and it can be determined that all the regions can be harvested. On the other hand, in the photographed image of FIG. 4B, the way the leaves of the branches grow in the vicinity of the central region 41 in the photographed image is clearly different from the others, and it is easy to judge that the growth is poor. However, the state of the central region 41 (the region with few leaves) can be found as a similar pattern even in the vicinity of the region 40 in the captured image of FIG. 4 (A). These two cases show that anomalous areas of agricultural trees cannot be determined by local patterns. That is, it is not possible to make a judgment by inputting only a local pattern as in the above-mentioned specific object detection, and it is necessary to reflect the context obtained from the entire image.

That is, sufficient performance cannot be exhibited unless the above-mentioned specific object detection is performed using a learning model learned from images taken under the same conditions of crops with similar growth conditions in the past.

Under conditions that differ from previously taken due to some external factor, such as when an image taken in a new field that has not been taken in the past is input, or due to continued sunshine, extremely heavy rainfall, etc. In order to deal with all cases, such as when an image is input but also when an image taken at a convenient time by the user is input, learning learned under conditions close to the conditions of the input image each time. You need to get a model.

Here, what kind of annotation work is required when performing annotation work and learning by deep learning every time a field is photographed will be described. For example, the results of annotation work on the captured images of FIGS. 4 (A) and 4 (B) are shown in FIGS. 5 (A) and 5 (B), respectively.

The rectangular areas 500 to 504 in the captured image of FIG. 5A are image areas designated by the annotation work. The rectangular area 500 is an image area designated as a normal branch area, and the rectangular areas 501 to 504 are image areas designated as a tree trunk area. Since the rectangular region 500 is an image region representing a normal state with respect to the growth of the tree, the image region is a region largely related to the prediction of yield. Hereinafter, a region such as the rectangular region 500, which represents a normal state with respect to the growth of a tree, and a region where fruits and the like can be harvested are referred to as a production region.

Rectangular areas 505 to 507 and 511 to 514 in the captured image of FIG. 5B are image areas designated by the annotation work. The rectangular areas 505 and 507 are image areas designated as normal branch areas, and the rectangular areas 506 are image areas designated as abnormal dead branch areas. A region representing an abnormal state, a region where fruits and the like cannot be harvested, such as a rectangular region 506, is referred to as a non-production region. The rectangular areas 511 to 514 are image areas designated as the area of the trunk of the tree. Since the image regions determined to be the regions (production regions) where fruits and the like can be harvested are the rectangular regions 505 and 507, the rectangular regions 505 and 507 are regions that are greatly related to the prediction of yield.

It is very costly to carry out such annotation work on a large number (for example, hundreds to thousands) of captured images each time the field is photographed. Therefore, in the present embodiment, good prediction results are obtained without performing such more troublesome annotation work. In this embodiment, a learning model is acquired by deep learning. However, the acquisition method of the learning model is not limited to a specific acquisition method. Also, various object detectors may be applied instead of the learning model.

Next, about the processing performed by the system according to the present embodiment in order to perform analysis processing such as prediction of yield in the field and calculation of non-production rate for the entire field based on the photographed image of the field taken by the camera 10. , 2A will be described according to the flowchart of FIG. 2A.

In step S20, the camera 10 generates a photographed image of the field by photographing the field while a moving object such as a farm work tractor 32 or a drone 37 is moving.

In step S21, the camera 10 attaches the above Exif information (shooting information) to the captured image generated in step S20, and processes the captured image to which the Exif information is attached to the cloud server 12 and information processing via the communication network 11. It transmits to the device 13.

In step S22, the CPU 131 of the information processing apparatus 13 acquires information (crop variety and age, crop growing method, pruning method, etc.) about the field and the crop photographed by the camera 10 as the photographed field parameter. For example, the CPU 131 displays the GUI (graphical user interface) shown in FIG. 6A on the display device 14 and accepts input of a shooting field parameter from the user.

In the GUI of FIG. 6A, a map of the entire field is displayed in the area 600. The map of the field displayed in the area 600 is divided into a plurality of sections, and an identifier (ID) unique to the section is displayed in each section. The user operates the user interface 15 to specify a portion in the area 600 corresponding to the division taken by the camera 10 (that is, the division to be subjected to the above analysis processing), or the identifier of the division is used as the area. Enter in 601. When the user operates the user interface 15 to specify a portion in the area 600 corresponding to the division taken by the camera 10, the identifier of the division is displayed in the area 601.

The user can operate the user interface 15 to input a crop name (agricultural product name) in the area 602. Further, the user can operate the user interface 15 to input the crop varieties in the area 603. Further, the user can operate the user interface 15 to input Trellis in the area 604. Trellis is, for example, a method of designing a vine for growing vines in a vineyard when the crop is vines. Further, the user can operate the user interface 15 to input the planted year in the area 605. The Planted Year indicates, for example, when the crop is vine, the time when the vine is planted. It is not essential to enter the photographed field parameters for all of these items.

Then, when the user operates the user interface 15 to instruct the registration button 606, the CPU 131 of the information processing apparatus 13 transmits the photographing field parameters of the above items input in the GUI of FIG. 6A to the cloud server 12. And send. The CPU 191 of the cloud server 12 stores (registers) the photographed field parameters transmitted from the information processing apparatus 13 in the external storage device 196.

Further, when the user operates the user interface 15 to instruct the correction button 607, the CPU 131 of the information processing apparatus 13 enables the correction of the photographing field parameters already input in the GUI of FIG. 6A.

The GUI of FIG. 6A is a GUI for inputting a photographed field parameter on the premise of managing a grape field in particular, but even if the purpose is the same, the photographed field parameter to be input by the user is used. Is not limited to that shown in FIG. 6 (A). Further, even when the crop is not grape, the photographed field parameter to be input by the user is not limited to that shown in FIG. 6 (A). For example, when the crop name to be input to the area 602 is changed, the title of the areas 603 to 605 and the photographed field parameter to be input may be changed.

Basically, the photographed field parameters input by the GUI in FIG. 6 (A) can be used as they are once they are determined. Therefore, for example, when the field is photographed every year and the yield is predicted, it has already been registered. It can be used by recalling the photographed field parameters. If the photographed field parameters have already been registered for the desired category, from the next time onward, as shown in FIG. 6 (B), the location corresponding to the desired category is indicated in the area 600 to correspond to the category. The imaging field parameters are displayed in areas 609-613.

Here, it is desirable to input all the correct photographed field parameters for the learning model selection in the subsequent stage, but even if there are photographed field parameters that could not be input because they are unknown to the user, they remain unknown and follow. Can be processed.

In step S23, a process for selecting a learning model candidate to be used for detecting an object such as an agricultural product from the captured image is performed. The details of the process in step S23 will be described with reference to the flowchart of FIG. 2B.

In step S230, the CPU 191 of the cloud server 12 has Exif information attached to each photographed image acquired from the camera 10 and a photographed field parameter registered in the external storage device 196 (photographing of a category corresponding to the photographed image). Field parameters) and generate query parameters from.

A configuration example of the query parameter is shown in FIG. 11 (A). The query parameter of FIG. 11A is a query parameter generated when the photographed field parameter of FIG. 6B is input.

In the "query name", "F5" entered in the area 609 is set. The "Shiraz" input to the area 611 is set in the "product type". In "Trellis", "Scott-Henry" input to the area 612 is set. In the "tree age", the number of years elapsed from "2001" input to the area 613 to the shooting date and time (year) included in the Exif information is set as the tree age "19". In the "shooting date", the shooting date and time (month and day) "Oct 20" included in the Exif information is set. The "shooting time zone" is from the latest shooting date / time (time) to the latest shooting date / time (time) among the shooting date / time (time) in the Exif information attached to each shot image received from the camera 10. The time zone "12: 00-14: 00" between them is set. In "latitude, longitude", the shooting position "35 ° 28'S, 149 ° 12'E" included in the Exif information is set.

The method of generating the query parameter is not limited to the above method, and for example, the data already used by the farmer of the crop in the field management may be read, and a set of parameters matching the above items may be used as the query parameter.

In some cases, information on some items may be unknown. For example, if the information about the Planted Year and the variety is not known, all the items as illustrated in FIG. 11 (A) cannot be filled. As shown in FIG. 11C, the query parameters in this case are partially blank.

Next, in step S231, the CPU 191 of the cloud server 12 has M (1 ≦ M <E) candidate learning models among the E learning models (E is an integer of 2 or more) stored in the external storage device 196. Select the learning model (candidate learning model) of. In the selection, a learning model learned based on an environment similar to the environment indicated by the query parameter is selected as a candidate learning model. The external storage device 196 stores, for each of the E learning models, a parameter set indicating the environment in which the learning model was trained. FIG. 11B shows a configuration example of the parameter set of each learning model in the external storage device 196.

"Model name" is the name of the learning model, "variety" is the variety of the crop learned by the learning model, and "Trellis" is to grow grapes in the "grape field" learned by the learning model. How to design a vine. " The "tree age" is the age of the crop learned by the learning model, and the "shooting date" is the shooting date and time of the captured image of the crop used by the learning model for learning. The "shooting time zone" is the period from the oldest shooting date and time to the latest shooting date and time of the captured images of the crops used for learning by the learning model, and the "latitude, longitude" is the period from the learning model. The shooting position of the captured image of the agricultural product used for learning is "35 ° 28'S, 149 ° 12'E".

Some training models are trained by mixing data sets collected in blocks of multiple fields. Therefore, for example, there may be a learning model whose model names are "M004" and "M005", in which a parameter set is set so as to include a plurality of settings (variety, tree age, etc.).

Therefore, the CPU 191 of the cloud server 12 obtains the similarity between the query parameters and the parameter set for each learning model shown in FIG. 11B, and uses the top M learning models as candidate learning models in descending order of the similarity. select.

When the parameter set of each training model of model name = M001, M002, ... Is expressed as M ₁ , M ₂ , ..., The CPU 191 of the cloud server 12 has a similarity D between the query parameter Q and the parameter set M _x . (Q, M _x ) is obtained by calculating the following equation (1).

Here, q _k represents the kth element from the beginning in the query parameter Q. In the case of FIG. 11A, the query parameter Q includes six elements of "cultivar", "Trellis", "tree age", "shooting date", "shooting time zone", and "latitude, longitude". Therefore, k = 1 to 6.

m _{x and k} represent the kth element from the beginning in the parameter set M _x . In the case of FIG. 11B, the parameter set includes six elements of "cultivar", "Trellis", "age", "shooting date", "shooting time zone", and "latitude, longitude". Therefore, k = 1 to 6.

f _k ( _ak , b _k ) is a function for finding the distance between the elements a _k and b _k , and is preset. Although f _k ( _ak , b _k ) may be carefully set by experiment in advance, the distance definition by the above equation (1) basically has a larger value as the learning model has different properties. Therefore, you can simply set as follows.

That is, basically, the element is divided into two types, one is a classification element (variety, Trellis) and the other is a continuous value element (tree age, shooting date ...). Therefore, the function that defines the distance between the classification elements is defined as the following equation (2), and the function that defines the distance between the continuous value elements is defined as the following equation (3).

Functions for all elements (k) are implemented in advance on a rule basis. In addition, α _k is determined according to the degree of influence of each element on the final inter-model distance. For example, since the difference depending on the "variety" (k = 1) does not appear so much in the difference in the image, α ₁ is as close to 0 as possible, and the difference in “Trellis” (k = 2) has a great influence, so α ₂ is Make adjustments in advance, such as setting a large value.

Further, in the case of a learning model in which a plurality of settings are registered for "variety" and "tree age", such as a learning model in which the model names in FIG. 11B are "M004" and "M005", for example, "variety". In the case of, the distance is obtained for each setting registered in the "variety", and the average distance is set as the distance corresponding to the "variety". Similarly, in the case of "tree age", the distance is obtained for each setting registered in "tree age", and the average distance is defined as the distance corresponding to "tree age".

If the CPU 191 of the cloud server 12 selects M learning models as candidate learning models based on the above similarity, the selection method is not limited to the specific selection method. For example, the CPU 191 of the cloud server 12 may select M learning models having a degree of similarity equal to or higher than a threshold value.

However, if any of the elements in the query parameter is empty, the processing in step S231 is not performed, and as a result, all the training models are used as candidate learning models and the subsequent processing is performed.

The effects of selecting a candidate learning model are wide-ranging. First, by excluding learning models that are unlikely to be prior knowledge in this step, it is possible to significantly reduce the processing time related to ranking creation by scoring the learning model that follows. In addition, even when scoring a learning model based on rules, if a learning model that does not need to be compared is included as a candidate, the selection accuracy of the learning model may be reduced, but the possibility is minimized. Can be fastened.

Next, in step S232, the CPU 191 of the cloud server 12 selects P (P is an integer of 2 or more) captured images from the captured images received from the camera 10 as model selection target images. The method of selecting P shot images from the shot images received from the camera 10 is not limited to a specific selection method. For example, the CPU 191 may randomly select P shot images from the shot images received from the camera 10, or may select them according to some criteria.

Next, in step S233, a process for selecting one of the M candidate learning models as the selective learning model is performed using the P captured images selected in step S232. The details of the process in step S233 will be described according to the flowchart of FIG. 2C.

In step S2330, the CPU 191 of the cloud server 12 describes, for each of the M candidate learning models, "for each of the P captured images, an object that is a process of detecting an object from the captured image using the candidate learning model. "Detection processing" is performed.

As a result, for each of the P captured images, the "result of the object detection process for the captured image" of each of the M candidate learning models can be obtained. In the present embodiment, the "result of the object detection process for the captured image" is the position information of the image region (rectangular region, detection region) of the object detected from the captured image.

In step S2331, the CPU 191 obtains a score for each of the "results of object detection processing for each of the P captured images" of the M candidate learning models. Then, the CPU 191 ranks (ranks) M candidate learning models based on the score, and selects N (N ≦ M) candidate learning models from the M candidate learning models.

At this time, since there is no annotation information in the captured image, accurate detection accuracy evaluation cannot be performed. However, for objects that are systematically designed and maintained, such as farms, it is possible to predict and evaluate the accuracy of object detection processing using the following rules. The score for the result of the object detection process by the candidate learning model is obtained, for example, as follows.

In a general field, as shown in FIGS. 3 (A) and 3 (B), crops are planted at equal intervals. Therefore, when detecting an object as in the annotation (rectangular area) illustrated in FIGS. 5A and 5B, it is normal that the rectangular area is continuously detected from the left end to the right end of the image. It is the state detected in.

For example, when it is detected as a region where fruits and the like can be harvested from the left end to the right end of the photographed image as shown in FIG. 5A, the production region should be detected as in the rectangular region 500. Further, even when there is a rectangular region 506 which is a non-produced region in the captured image as shown in FIG. 5B, it should be detected as rectangular regions 505, 506, 507 from the left end to the right end of the captured image. If the object detection process for the captured image is performed using a learning model that does not meet the conditions of the captured image, there is a possibility that an undetected rectangular region may occur among the above rectangular regions. The more the learning model corresponds to the condition far from the condition of the captured image, the higher the possibility of such a situation. Therefore, as the simplest scoring method for evaluating the candidate learning model, for example, the following method can be considered.

The detection area of a plurality of objects is detected from the captured image of interest by the focus candidate learning model. Therefore, the detection region is searched in the vertical direction of the focus shooting image, the number of pixels Cp in the region without the detection region is counted, and the ratio of the number of pixels Cp to the number of pixels in the width of the focus shooting image is the focus shooting image. Penalty score. In this way, a penalty score is obtained for each of the P captured images subjected to the object detection process by the focus candidate learning model, and the total value of the obtained penalty scores is used as the score of the focus candidate learning model. By performing such processing for each of the M candidate learning models, the score of each candidate learning model is determined. Then, the M candidate learning models are ranked in ascending order of score, and the top N candidate learning models are selected in ascending order of score. At the time of the selection, the condition that "the score is less than the threshold value" may be added.

Further, as the score of the candidate learning model, a score inferred from the detection region of the trunk portion of the tree normally planted at equal intervals may be obtained. As shown in FIG. 5A, the tree trunk should be detected at approximately equal intervals with the rectangular areas 501, 502, 503, and 504. Therefore, “the number of detected tree trunk areas” with respect to the width of the captured image. The expected number is fixed. Since there is a high possibility that a detection error has occurred in captured images that are less / more than the expected number, this number of detections may be reflected in the score.

Then, the CPU 191 includes "results of object detection processing for the P captured images" of each of the P captured images and N candidate learning models selected from the M candidate learning models, and the N candidate learning models. Information (model name, etc.) is transmitted to the information processing apparatus 13. As described above, in the present embodiment, the "result of the object detection process for the captured image" is the position information of the image area (rectangular area, detection area) of the object detected from the captured image, and such position information. Is transmitted to the information processing apparatus 13 as data in a file format such as, for example, json format or txt format.

Next, the user is made to select one of the selected N candidate learning models. At the end of the process of step S2331, N candidate learning models still remain, and the output used as the basis for comparing the performance is the result of the object detection process for the above P captured images. It is necessary to compare the results of the object detection processing for the P shot images. In such a state, it is difficult to appropriately select one candidate learning model as a selective learning model (narrow down to one).

Therefore, in step S2332, the CPU 131 of the information processing apparatus 13 performs scoring (display image scoring) for presenting information that can be easily compared by the user's subjectivity with respect to the captured images of P images. In the display image scoring, for each of the P captured images, the larger the score is determined as the arrangement pattern of the detection region is significantly different among the N candidate learning models. Such a score can be obtained, for example, by calculating the following equation (4).

Here, Score (z) is a score for the captured image Iz. The T _Iz (Ma, Mb) is the result of the object detection process (arrangement pattern of the detection area) performed by the candidate learning model Ma on the captured image Iz, and the object performed by the candidate learning model Mb on the captured image Iz. This is a function for obtaining a score based on the difference between the result of the detection process (arrangement pattern of the detection area) and the difference. Various functions can be applied to such functions, and are not limited to specific functions. For example, for each detection region Ra detected by the candidate learning model Ma from the captured image Iz, the position of the detection region Rb'closest to the detection region Ra among the detection regions Rb detected by the candidate learning model Mb from the captured image Iz (for example,). A function that finds the difference between the position of the upper left corner and the position of the lower right corner) and the position of the detection area Ra (for example, the position of the upper left corner and the position of the lower right corner) and returns the sum of the obtained differences is _TIz (Ma, Mb). ) May be used.

Since the results of object detection processing by the top N candidate learning models are often similar, there are many cases where there is almost no difference even when comparing randomly selected images, and the basis for selecting a learning model. do not become. Therefore, it is easy to judge whether the learning model is good or bad by looking only at the high-ranked captured images scored by the above equation (4).

In step S2333, the CPU 131 of the information processing apparatus 13 has the top F images (specified number from the top) in descending order of the score among the P shot images received from the cloud server 12 for each of the above N candidate learning models. The captured image of the above and the result of the object detection process for the captured image received from the cloud server 12 are displayed on the display device 14 (display control). At that time, the captured images of F images are displayed side by side from the left in descending order of score.

FIG. 7A shows a display example of the GUI displaying the captured image and the result of the object detection process for each candidate learning model. FIG. 7A shows the case of N = 3 and F = 4.

In the top row, the model name "M002" of the candidate learning model with the highest score is displayed together with the radio button 70, and on the right side, the top four captured images are arranged in order from the left in descending order of score. It is displayed. In the captured image, a frame showing the detection area of the object detected by the candidate learning model of the model name “M002” is displayed superimposed on the captured image.

In the middle row, the model name "M011" of the candidate learning model with the second highest score is displayed together with the radio button 70, and on the right side, the top four captured images in descending order of score are displayed in order from the left. They are displayed side by side. In the captured image, a frame showing the detection area of the object detected by the candidate learning model of the model name “M011” is displayed superimposed on the captured image.

In the lower row, the model name "M009" of the candidate learning model with the third highest score is displayed together with the radio button 70, and on the right side, the top four captured images in descending order of score are displayed in order from the left. They are displayed side by side. In the captured image, a frame showing the detection area of the object detected by the candidate learning model of the model name “M009” is displayed superimposed on the captured image.

In this GUI, the captured images arranged in the same row are displayed so as to be the same captured image so that the results of the object detection processing by each candidate learning model can be easily compared at a glance.

Then, the user visually confirms the difference in the result of the object detection processing by the N candidate learning models for the F images, and selects one of the N candidate learning models by using the user interface 15.

In step S2334, the CPU 131 of the information processing apparatus 13 accepts a candidate learning model selection operation (user operation, user input) by the user. In step S2335, the CPU 131 of the information processing apparatus 13 determines whether or not the selection operation (user input) of the candidate learning model by the user has been performed.

In the case of FIG. 7A, when the user selects the candidate learning model having the model name “M002”, the radio button 70 in the top row is selected by using the user interface 15. Further, when the user selects the candidate learning model having the model name "M011", the user selects the radio button 70 in the middle row using the user interface 15. Further, when the user selects the candidate learning model having the model name "M009", the user selects the radio button 70 in the lower row by using the user interface 15. In FIG. 7A, since the radio button 70 corresponding to the model name “M002” is selected, the frame 74 indicating that the candidate learning model having the model name “M002” is selected is displayed.

Then, when the user operates the user interface 15 to instruct the enter button 71, the CPU 131 determines that "the user has performed the candidate learning model selection operation (user input)" and corresponds to the selected radio button 70. Select the candidate learning model as the selective learning model.

As a result of this determination, when the user has performed the candidate learning model selection operation (user input), the process proceeds to step S2336, and the user has not performed the candidate learning model selection operation (user input). The process proceeds to step S2334.

In step S2336, the CPU 131 of the information processing apparatus 13 finally confirms whether only one learning model is selected. Then, when only one learning model is finally selected, the process proceeds to step S24, and when finally only one learning model is not selected, the process proceeds to step S24. Proceed to S2332.

Here, if the user cannot narrow down to one by just looking at the display of FIG. 7A, select a plurality of radio buttons 70 to select a plurality of candidate learning models. Is also good. For example, the user operates the user interface 15 to select the radio button 70 corresponding to the model name “M002” and the radio button 70 corresponding to the model name “M011” in FIG. 7A, and press the enter button 71. If specified, the number "2" of the selected radio buttons 70 is set to N, and the process proceeds to step S2332 via step S2336. In this case, in step S2332 and subsequent steps, the same processing is performed for N = 2 and F = 4. In this way, the process is repeated until the number of learning models finally selected becomes "1".

Further, the user may select the learning model by using the GUI of FIG. 7 (B) instead of the GUI of FIG. 7 (A). The GUI of FIG. 7A is a GUI that allows the user to directly select which learning model is better. On the other hand, in the GUI of FIG. 7B, a check box 72 is provided for each captured image, and the user can use the captured images in the captured image row for each captured image row arranged in a vertical row. The check box 72 of the captured image determined that the result of the object detection process is preferable is specified by operating the user interface 15 and turned on (a check mark is added). Then, when the user operates the user interface 15 to instruct the enter button 75, the CPU 131 of the information processing apparatus 13 has a check box 72 among the candidate learning models whose model names are "M002", "M011", and "M009". The candidate learning model with the largest number of captured images turned on is selected as the selection learning model. In the example of FIG. 7B, three check boxes 72 of the four captured images of the candidate learning model having the model name “M002” are selected, and the candidate learning model having the model name “M011” is selected. The check box 72 of one of the four captured images is turned on, and the check box 72 of any of the four captured images of the candidate learning model of the candidate learning model whose model name is "M009" is turned on. not. In this case, the candidate learning model whose model name is "M002" is selected as the selective learning model. Such a method of selecting a selection learning model by GUI is effective, for example, when the value of F increases and it is difficult for the user to determine which candidate learning model is the best.

If there is a candidate learning model with the same number or a small difference in "the number of captured images with the check box 72 selected", "in the state where only one learning model is finally selected" in step S2336. It is determined that there is no such thing, and the process proceeds to step S2332. Then, in step S2332 and subsequent steps, processing is performed on a candidate learning model having the same number or a small difference in "the number of captured images in which the check box 72 is selected". Even in such a case, the process is repeated until the number of learning models finally selected becomes "1".

Further, since the captured image displayed on the left side is a captured image in which the difference in the result of the object detection processing between the candidate learning models is large, the captured image displayed on the left side is assigned a larger weight value. Is also good. In this case, the total weight value of the captured images for which the check box 72 is selected may be obtained for each candidate learning model, and the candidate learning model having the largest total obtained may be selected as the selective learning model.

Then, the CPU 131 of the information processing apparatus 13 notifies the cloud server 12 of information indicating the selective learning model (for example, the model name of the selective learning model) regardless of the method of selecting the selective learning model.

In step S24, the CPU 191 of the cloud server 12 uses a captured image (a captured image transmitted by the camera 10 to the cloud server 12 and the information processing device 13) using a selective learning model specified by the information notified from the information processing device 13. ) Is subject to object detection processing.

In step S25, the CPU 191 of the cloud server 12 performs analysis processing such as prediction of the yield of the target field and calculation of the non-production rate for the entire field from the detection area obtained as a result of the object detection processing in step S24. This calculation is performed by taking into account both the production area rectangle detected from all the captured images and the non-production area determined to be the dead branch area, the lesion area, and the like.

Although the learning model according to this embodiment is a model learned by deep learning, various object detection techniques such as rule-based detectors defined by various parameters, fuzzy inference, and genetic algorithms are learned as learning models. You may use it as.

[Second Embodiment]
Hereinafter, the differences from the first embodiment will be described, and the same as the first embodiment will be described unless otherwise specified below. In this embodiment, a system for performing an visual inspection on a production line of a factory will be described as an example. The system according to this embodiment detects an abnormal region of an industrial product to be inspected.

Conventionally, in visual inspection on a factory production line, the shooting conditions of the inspection device (device that photographs and inspects the appearance of the product) are carefully adjusted for each production line, and the inspection device is used every time each production line starts up. It was common to adjust the setting of. However, in recent years, it has been desired for manufacturing sites to immediately respond to the diversification of customer needs and changes in the market. And even if it is a small lot, we will start up the line in a short period of time to manufacture the quantity that meets the demand, and when sufficient supply is completed, we will immediately dismantle the line and prepare for the next production line. There is a growing need for.

At that time, if the visual inspection settings are set each time based on the experience and intuition of the specialists at the manufacturing site as in the past, it is not possible to respond to a quick start-up. Expert experience if similar product inspections have been performed in the past, if the setting parameters related to them can be retained and the past setting parameters can be recalled when performing similar inspections. Anyone can set the inspection device without relying on.

Similar to the first embodiment, the above object is similarly achieved by allocating the already held learning model to the inspection target image of the new product. Therefore, the above information processing apparatus 13 can be applied to the second embodiment as well.

The setting process (setting process for visual inspection) of the inspection device by the system according to the present embodiment will be described with reference to the flowchart of FIG. 8A. It is assumed that the setting process for visual inspection is performed at the start of the inspection step on the production line.

A plurality of learning models (models / settings for visual inspection) for performing visual inspection on captured images are registered in the external storage device 196 of the cloud server 12, and each learning model is learned in different learning environments. It is a model.

The camera 10 is a camera for photographing a product (product to be inspected) to be visually inspected. Similar to the first embodiment, the camera 10 may be a camera that shoots images periodically or irregularly, or may be a camera that shoots moving images. In order to accurately detect the abnormal area in the product to be inspected from the captured image, when the product to be inspected including the abnormal area enters the inspection process, the image is taken under the condition that the abnormal area is emphasized as much as possible. Is desirable. The camera 10 may be a multi-camera as long as the product to be inspected is photographed under a plurality of conditions.

In step S80, the camera 10 captures the inspection target product to generate a captured image of the inspection target product. In step S81, the camera 10 transmits the captured image generated in step S20 to the cloud server 12 and the information processing device 13 via the communication network 11.

In step S82, the CPU 131 of the information processing apparatus 13 has information on the product to be inspected taken by the camera 10 (part name and material of the product to be inspected, date of manufacture, imaging system parameters at the time of shooting, lot number and temperature, etc.). Humidity, etc.) is acquired as the product parameter to be inspected. For example, the CPU 131 displays the GUI on the display device 14 and accepts the input of the inspection target product parameter from the user. Then, when the user operates the user interface 15 and inputs a registration instruction, the CPU 131 of the information processing apparatus 13 transmits the inspection target product parameters of each of the above items input in the GUI to the cloud server 12. The CPU 191 of the cloud server 12 stores (registers) the inspection target product parameters transmitted from the information processing device 13 in the external storage device 196.

In step S83, a process for selecting a learning model to be used for detecting the above-mentioned inspection target product from the captured image is performed. The details of the process in step S83 will be described with reference to the flowchart of FIG. 8B.

In step S831, the CPU 191 of the cloud server 12 selects M learning models (candidate learning models) that are candidates from the E learning models stored in the external storage device 196. The CPU 191 generates a query parameter from the inspection target product parameter registered in the external storage device 196 in the same manner as in the first embodiment, and learns about an environment similar to the environment indicated by the query parameter (similar to the past). Select the learning model) used in the test.

If the query parameter includes "base" as the "part name", it will be easier to select the learning model used in the past base inspection, and if "glass epoxy" is included as the "material". The learning model used to inspect the glass epoxy substrate is easier to choose.

In step S831, as in the first embodiment, M candidate learning models are selected using the parameter set of the learning model and the query parameters, but in that case, the above equation is used as in the first embodiment. (1) is used.

Next, in step S832, the CPU 191 of the cloud server 12 selects P shot images from the shot images received from the camera 10 as model selection target images. For example, products flowing into the main inspection process of the main production line are randomly selected, and P shot images are acquired from the shot images taken by the camera 10 with the same settings as in the actual operation. Normally, the number of abnormal products generated on the production line is small, so if the number of products to be photographed in the process is small, the processing in the subsequent steps does not work well. Therefore, as a guide, it is desirable to shoot several hundred or more products.

Next, in step S833, a process for selecting one of the M candidate learning models as the selective learning model is performed using the P captured images selected in step S832. The details of the process in step S833 will be described according to the flowchart of FIG. 8C.

In step S833, the CPU 191 of the cloud server 12 describes, for each of the M candidate learning models, "for each of the P captured images, an object that is a process of detecting an object from the captured image using the candidate learning model. "Detection processing" is performed. Also in this embodiment, the result of the object detection process for the captured image is the position information of the image region (rectangular region, detection region) of the object detected from the captured image.

In step S8331, the CPU 191 obtains a score for each of the "results of object detection processing for each of the P captured images" of the M candidate learning models. Then, the CPU 191 ranks M candidate learning models (ranking creation) based on the score, and selects N candidate learning models from the M candidate learning models. The score for the result of the object detection process by the candidate learning model is obtained, for example, as follows.

For example, in a task of detecting an abnormality on a printed circuit board, an object detection process is performed on various specific local patterns on a fixed printed pattern. Here, it is assumed that the detection regions 901 to 906 as shown in FIG. 9A can be obtained from the captured image of the normal product in the specific learning model. Since the frequency of abnormalities in products produced on the production line is extremely low, a good learning model for performing the above tasks is a learning model that can output stable results against possible variations in captured images. .. For example, the appearance of the image obtained by photographing the product may be slightly changed due to the change in the environment on the area sensor side, so that the detection area 906 of the detection areas 901 to 906 may not be detected as shown in FIG. 9B. In such cases, penalties should be given to the evaluation score of the learning model whose detection area changes for inputs with only slight differences.

Therefore, for example, the CPU 191 of the cloud server 12 determines, for each of the M candidate learning models, a larger score as the arrangement pattern of the detection region by the candidate learning model differs greatly among the P captured images. Such a score can be obtained, for example, by calculating the above equation (4). Then, the M candidate learning models are ranked in ascending order of score, and the top N candidate learning models are selected in ascending order of score. At the time of the selection, the condition that "the score is less than the threshold value" may be added.

In step S8332, the CPU 131 of the information processing apparatus 13 performs scoring (display image scoring) for presenting information that can be easily compared by the user's subjectivity for the P images taken in the first embodiment (step S2332). ).

In step S8333, the CPU 131 of the information processing apparatus 13 takes the top F captured images in descending order of the scores among the P captured images received from the cloud server 12 for each of the N candidate learning models selected in step S8331. And the result of the object detection process for the captured image received from the cloud server 12 are displayed on the display device 14. At that time, the captured images of F images are displayed side by side from the left in descending order of score.

FIG. 10A shows a display example of the GUI displaying the captured image and the result of the object detection process for each candidate learning model. FIG. 10A shows the case of N = 3 and F = 4.

In the top row, the model name "M005" of the candidate learning model with the highest score is displayed together with the radio button 100, and on the right side, the top four captured images are arranged in order from the left in descending order of score. It is displayed. In the captured image, a frame showing a detection area detected by the candidate learning model of the model name “M005” is displayed superimposed on the captured image.

In the middle row, the model name "M023" of the candidate learning model with the second highest score is displayed together with the radio button 100, and on the right side, the top four captured images in descending order of score are displayed in order from the left. They are displayed side by side. In the captured image, a frame showing a detection area detected by the candidate learning model of the model name “M023” from the captured image is superimposed and displayed.

In the lower row, the model name "M014" of the candidate learning model with the third highest score is displayed together with the radio button 100, and on the right side, the top four captured images in descending order of score are displayed in order from the left. They are displayed side by side. In the captured image, a frame showing a detection area detected by the candidate learning model of the model name “M014” is displayed superimposed on the captured image.

In this GUI, the captured images arranged in the same row are displayed so as to be the same captured image so that the results of the object detection processing by each candidate learning model can be easily compared at a glance.

In this case, the difference in the arrangement pattern of the detection region is as shown in FIG. 10A because the appearance of the product is almost fixed and most of them are normal products. The F shot images are displayed side by side in descending order of score, but the one with the highest score tends to be the case where the difference in shooting conditions at the time of individual shooting is large or the individual contains an abnormal region. Therefore, compared to the conventional method in which the user manually annotates the photographed image of the product to the abnormal area and manually searches for defective products from many products before setting the inspection device. Even if the work is not performed at all, the photographed image of the product that may include an abnormal region can be preferentially presented to the user just by looking at this GUI, which saves labor. The user may select a learning model that can correctly detect the abnormal region while comparing the results of the object detection process with the GUI of FIG. 10 (A).

Then, the user visually confirms the difference in the result of the object detection processing by the N candidate learning models for the F images, and selects one of the N candidate learning models by using the user interface 15.

In step S8334, the CPU 131 of the information processing apparatus 13 accepts a candidate learning model selection operation (user input) by the user. In step S8335, the CPU 131 of the information processing apparatus 13 determines whether or not the selection operation (user input) of the candidate learning model by the user has been performed.

In the case of FIG. 10A, when the user selects the candidate learning model having the model name “M005”, the radio button 100 in the top row is selected by using the user interface 15. Further, when the user selects the candidate learning model having the model name "M023", the user selects the radio button 100 in the middle row using the user interface 15. Further, when the user selects the candidate learning model having the model name "M014", the user selects the radio button 100 in the lower row by using the user interface 15. In FIG. 10A, since the radio button 100 corresponding to the model name “M005” is selected, the frame 104 indicating that the candidate learning model having the model name “M005” is selected is displayed.

Then, when the user operates the user interface 15 to instruct the decision button 101, the CPU 131 determines that "the user has performed the candidate learning model selection operation (user input)" and corresponds to the selected radio button 100. Select the candidate learning model as the selective learning model.

As a result of this determination, when the user has performed the candidate learning model selection operation (user input), the process proceeds to step S8336, and the user has not performed the candidate learning model selection operation (user input). The process proceeds to step S8334.

In step S8336, the CPU 131 of the information processing apparatus 13 finally confirms whether or not the learning model is selected by "the number desired by the user". Then, when the learning model is finally selected by "the number desired by the user", the process proceeds to step S84, and finally the learning model is selected by "the number desired by the user". If it is not in the state, the process proceeds to step S8332.

Here, the "number desired by the user" is mainly determined according to the time (tact time) that can be spent on the visual inspection. For example, when the "number desired by the user" is two, the tendency of the detection target is changed by detecting a low frequency abnormal region with one learning model and detecting a high frequency defect with the other learning model. May enable a wide range of detection.

Here, if the user cannot narrow down to the "number desired by the user" just by looking at the display of FIG. 10A, a plurality of candidate learning models can be selected by selecting a plurality of radio buttons 100. You may choose. For example, when the "number desired by the user" is "1" and the number of selected radio buttons 100 is 2, the process proceeds to step S8332 via step S8336 with N = 2. In this case, in step S8332 and subsequent steps, the same processing is performed for N = 2 and F = 4. In this way, the process is repeated until the number of learning models finally selected reaches the "number desired by the user".

Further, the user may select the learning model by using the GUI of FIG. 10 (B) instead of the GUI of FIG. 10 (A). The GUI of FIG. 10A is a GUI that allows the user to directly select which learning model is better. On the other hand, in the GUI of FIG. 10B, a check box 102 is provided for each captured image, and the user can use the captured images in the captured image row for each captured image row arranged in a vertical row. The check box 102 of the captured image for which the result of the object detection process is determined to be preferable is specified by operating the user interface 15 and turned on (a check mark is added). Then, when the user operates the user interface 15 to specify the enter button 1015, the CPU 131 of the information processing apparatus 13 has the check box 102 among the candidate learning models whose model names are "M005", "M023", and "M014". The candidate learning model with the largest number of captured images turned on is selected as the selection learning model. In the example of FIG. 10B, two check boxes 102 of the four captured images of the candidate learning model having the model name “M005” are selected, and the candidate learning model having the model name “M023” is selected. One of the four captured images has the check box 102 turned on, and one of the four captured images of the candidate learning model whose model name is "M014" has the check box 102 turned on. In this case, the candidate learning model whose model name is "M005" is selected as the selective learning model. Such a method of selecting a selection learning model by GUI is effective, for example, when the value of F increases and it is difficult for the user to determine which candidate learning model is the best.

In the GUI of FIG. 10B, the easiest way to finally narrow down to the learning model of "the number desired by the user" is to "the number desired by the user" from the top in descending order of the number of checked boxes. It is a method to select up to.

If there is a candidate learning model with the same number or a small difference in "the number of captured images with the check box 72 selected", in step S8336, "finally, the learning model is only the" number desired by the user ". It is determined that the state is not selected, and the process proceeds to step S8332. Then, in step S8332 and subsequent steps, processing is performed on a candidate learning model having the same number or a small difference in "the number of captured images in which the check box 102 is selected". Even in such a case, the process is repeated until the number of learning models finally selected becomes "the number desired by the user".

Then, the CPU 131 of the information processing apparatus 13 notifies the cloud server 12 of information indicating the selective learning model (for example, the model name of the selective learning model) regardless of the method of selecting the selective learning model.

In step S84, the CPU 191 of the cloud server 12 uses a captured image (a captured image transmitted by the camera 10 to the cloud server 12 and the information processing device 13) using a selective learning model specified by the information notified from the information processing device 13. ) Is subject to object detection processing. Then, the CPU 191 of the cloud server 12 sets the final inspection device from the detection area obtained as a result of the object detection process. The inspection is executed when the production line is actually started up by the learning model and various parameters set here.

Although the learning model according to this embodiment is a model learned by deep learning, various object detection techniques such as rule-based detectors defined by various parameters, fuzzy inference, and genetic algorithms are learned as learning models. You may use it as.

<Modification example>
Each of the above embodiments is for reducing the cost of learning the learning model and adjusting the settings each time the detection / identification process for a new target is performed in the task of performing the target detection / identification process. It is an example of technology. Therefore, the techniques described in each of the above embodiments are not limited to being applied to the prediction of the yield of agricultural products, the detection of repair areas, the detection of abnormal areas of industrial products to be inspected, and the like. The subject applies to agriculture, industry, fisheries and a wide range of other fields.

In addition, the above radio buttons and check boxes are displayed as an example of a selection unit for the user to select a target, and if they have the same effect, other display items are displayed instead. Is also good. Further, the configuration in which the learning model used for the object detection process in the above-described embodiment is selected based on the user operation has been described (S24). However, the present invention is not limited to this, and a learning model used for the object detection process may be automatically selected. For example, the candidate learning model with the highest score may be automatically selected as the learning model used for the object detection process.

Further, the subject of each process in the above description is an example. For example, the CPU 131 of the information processing apparatus 13 may perform a part or all of the processing described as being performed by the CPU 191 of the cloud server 12. Further, the CPU 191 of the cloud server 12 may perform a part or all of the processing described as being performed by the CPU 131 of the information processing apparatus 13.

Further, in the above description, it has been described that the system of each embodiment performs the analysis process. However, the subject of the analysis process is not limited to the system of each of the above-described embodiments, and for example, the analysis process may be performed by another device / system.

In addition, the numerical values, processing timing, processing order, processing subject, data (information) configuration / destination / source, etc. used in each of the above embodiments and modifications are given as examples for specific explanation. It is just an example and is not intended to be limited to such an example.

In addition, some or all of the embodiments and modifications described above may be used in combination as appropriate. In addition, some or all of the embodiments and modifications described above may be selectively used.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiment, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to publicize the scope of the invention.

10: Camera 11: Communication network 12: Cloud server 13: Information processing device 14: Display device 15: User interface 131: CPU 132: RAM 133: ROM 134: Output I / F 135: Input I / F 191: CPU 192: RAM 193: ROM 194: Operation unit 195: Display unit 196: External storage device 197: I / F 198: System bus

Claims (14)

A first-choice means for selecting one or more learning models as candidate learning models from a plurality of learning models learned in different learning environments based on information related to object photography.
A second selection means for selecting one or more candidate learning models from the candidate learning model based on the result of object detection processing by the candidate learning model selected by the first selection means.
An information processing apparatus including a detection unit that performs object detection processing on a captured image of the object by using at least one of the candidate learning models selected by the second selection means. .. The first selection means generates a query parameter based on the information, and selects one or more learning models trained in an environment similar to the environment indicated by the query parameter from the plurality of learning models as a candidate learning model. The information processing apparatus according to claim 1. The second selection means obtains a score based on the result of object detection processing by the candidate learning model for each candidate learning model selected by the first selection means, and the first selection means selects based on the score. The information processing apparatus according to claim 1 or 2, wherein one or more candidate learning models are selected from the candidate learning models. In addition,
The information processing apparatus according to claim 1, further comprising display control means for displaying the result of object detection processing by the candidate learning model selected by the second selection means. The display control means has a large score as the result of the object detection process differs greatly among the candidate learning models for each of the plurality of captured images for which the candidate learning model selected by the second selection means has performed the object detection process. Is determined, and for each candidate learning model selected by the second selection means, a GUI including the result of object detection processing by the candidate learning model for a specified number of captured images from the top in descending order of score is displayed. The information processing apparatus according to claim 4. The GUI includes a selection unit for selecting a candidate learning model.
5. The detection means is characterized in that a candidate learning model corresponding to a selection unit selected according to a user operation in the GUI is used as a selection learning model, and object detection processing is performed using the selection learning model. The information processing device described in. The detection means uses the candidate learning model having the largest number of object detection processing results selected according to the user operation among the object detection processing results displayed for each candidate learning model by the display control means as the selection learning model. The information processing apparatus according to claim 5, wherein the object detection process is performed using the selective learning model. In addition,
6. The information processing device described. The information according to any one of claims 1 to 8, wherein the information includes Exif information of the photographed image, information relating to the field in which the photographed image was photographed, and information relating to an object included in the photographed image. The information processing device described in. In addition,
The object according to any one of claims 1 to 7, further comprising means for setting a device for photographing and inspecting the appearance of the product based on the detection area of the object obtained as a result of the object detection process. The information processing device described. The information processing apparatus according to any one of claims 1 to 7, 10, wherein the information includes information relating to an object included in the captured image. The detection means is characterized in that the object detection process is performed on the captured image of the object by using the candidate learning model selected based on the user operation among the candidate learning models selected by the second selection means. The information processing apparatus according to any one of claims 1 to 11. It is an information processing method performed by an information processing device.
The first selection means of the information processing apparatus selects one or more learning models as candidate learning models from a plurality of learning models learned in different learning environments based on the information related to the photographing of the object. ,
A second selection step in which the second selection means of the information processing device selects one or more candidate learning models from the candidate learning model based on the result of the object detection process by the candidate learning model selected in the first selection step. When,
A detection step in which the detection means of the information processing apparatus uses at least one of the candidate learning models selected in the second selection step to perform object detection processing on the captured image of the object. An information processing method characterized by being prepared. A computer program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 12.

Images