JP7025989B2 - Learning equipment, learning methods and learning programs - Google Patents

Description

The present invention relates to a learning device, a learning method and a learning program.

In recent years, with the decrease in the working population due to the declining birthrate and aging population, opportunities for artificial intelligence and robots to play an active role in factories, warehouses, etc. are increasing. For example, workers engaged in product inspection work (determination of non-defective products and defective products) are being replaced by robots. The robot visually recognizes the product, grasps the product, recognizes whether or not the product is defective, and if the product is defective, removes the product from the inspection line or the like. In order to make the robot perform such work, it is necessary to learn the software (artificial intelligence) that controls the robot in advance with a huge amount of learning data.

When artificial intelligence is a multi-layered neural network, so-called deep learning is performed. In deep learning using teacher data, in addition to the man-hours for preparing a huge number of learning data, the man-hours for attaching a correct answer label to the learning data are also required. For example, an image of tens of thousands of scales is acquired on the inspection line, and then a label indicating whether the product of the image is a good product or a defective product is attached to the image. The data processing device of Patent Document 1 acquires the movement of a person as a time-series image. The device newly extracts an image between the two images based on the degree of change between the image at the first time point and the image at the second time point thereafter. When the degree of change is smaller than a predetermined threshold value, the apparatus has the same label as the label attached to the image at the first time point or the label attached to the image at the second time point on the newly extracted image. With the same label as.

Japanese Unexamined Patent Publication No. 2016-76073

However, the data processing apparatus of Patent Document 1 manually inputs a user to determine which label should be attached to the image that is a candidate to be labeled (paragraph 0068 of Patent Document 1, FIG. 3). .. That is, in Patent Document 1, although the effect of acquiring an image that is a valid candidate for teacher data is recognized, a separate measure is required in order to reduce the man-hours for labeling.
Therefore, an object of the present invention is to automate the labeling work.

The learning device of the present invention has an inference unit that acquires a recognition result by an inference model from an inference model that inputs an image of a product and outputs a recognition result for the product, and an image of the product and an image of human behavior toward the product. It is equipped with a correct answer label information creation unit that acquires the recognition result by a person and a learning unit that learns an inference model based on the recognition result by the inference model and the recognition result by the person. The correct answer label information creation unit is an image of the product. And in the image of human behavior toward the product, as a result of the person touching the product on the inspection line that transports the product and then releasing the product, the product flows on the inspection line in a predetermined direction and disappears from the screen. When it is detected, the product is determined to be a non-defective product .
Other means will be described in the form for carrying out the invention.

According to the present invention, the labeling work can be automated.

It is a figure explaining the environment in which a learning device operates. It is a figure explaining the structure of a learning device. It is a figure explaining the correct answer label information. It is a figure explaining the gripping position coordinates and flameout. It is a figure explaining the inference result information. It is a flowchart of the whole processing procedure. It is a flowchart of the product ID creation processing procedure. It is a flowchart of a correct answer label information creation processing procedure. It is a flowchart of the inference result information creation processing procedure.

Hereinafter, a mode for carrying out the present invention (referred to as “the present embodiment”) will be described in detail with reference to figures and the like. This embodiment is an example in which the robot determines whether the product is good or bad and eliminates the defective product in the product inspection process. However, the present invention is generally applicable to an example in which a robot operates on an object based on the result of visually recognizing the appearance of the object and grasping the object.

(environment)
An environment in which the learning device of the present embodiment operates will be described with reference to FIG. FIG. 1 is a diagram of an inspection line 6 of a factory. Belt conveyors 4 are laid in the left-right direction of FIG. The belt conveyor 4 conveys a plurality of products 5 from the left to the right in FIG. The product 5 may be a finished product immediately before shipment or an intermediate product immediately before going to the next process. The shape and the like of the product 5 are not uniform and are different for each product 5. Worker 3 is standing in front of the belt conveyor 4.

The worker 3 visually observes the product 5 conveyed from the left side and recognizes the shape, color, and the like of the product. At the same time, the worker 3 grasps a certain part of the product 5 by hand and recognizes the weight, the position of the center of gravity, the fixed state of the internal structure, and the like. As a result of the recognition, when the worker 3 determines that the shape or the like of the product 5 does not meet the predetermined criteria, the worker 3 determines that the product 5 is a defective product, and determines that the defective product is a defective product outside the belt conveyor 4. Eliminate to the defective collection yard (not shown). When the shape of the product 5 or the like satisfies a predetermined criterion, the worker 3 determines that the product 5 is a non-defective product and forgoes the product 5 to be conveyed as it is on the belt conveyor 4.

A robot (not shown) performs such determination and operation of the worker 3 in exactly the same manner. The robot may be a humanoid robot having a human whole body shape, or may be an arm robot having only human arms and hands. The image pickup device 2 has acquired the images of the product 5 and the worker 3. The image pickup apparatus 2 is, for example, a three-dimensional RGB (Red Green Blue) camera. The 3D RGB camera can reproduce the captured product 5 and the hands of the worker 3 as a color image in a 3D virtual space in chronological order.

The learning device of the present embodiment creates correct answer label information (detailed later) from the image acquired by the image pickup device 2, and learns an inference model (detailed later) using the correct label information as learning data. In addition, the learning device uses the trained inference model to control the robot. That is, the criteria for determining the quality / defect of the product by the robot and the position of the product held by the robot are determined by the inference model.

(Inference model)
The inference model 31 (FIG. 2) of the present embodiment is, for example, a neural network (not shown) having an input layer, an output layer, and a plurality of intermediate layers. The input layer receives the image of the product. The output layer outputs the good / bad determination result and the grippable position coordinates of the product. As is known, the input layer, the intermediate layer and the output layer have a plurality of nodes (corresponding to human nerve cells). When a node receives a certain amount of information, a part of the amount of information is propagated to a node in the next layer. Each node has information (weight vector) as to how much information is propagated to which node. Learning the inference model 31 corresponds to optimizing the components of the weight vector for each node. Hereinafter, in the present embodiment, the weight vectors of the respective nodes are collectively referred to as “parameters”.

(Configuration of learning device)
The configuration of the learning device 1 will be described with reference to FIG. The learning device 1 is a general computer and includes a central control device 11, an input device 12 such as a keyboard and a touch panel, an output device 13 such as a display, a main storage device 14, an auxiliary storage device 15, and an image acquisition device 16. These are interconnected by a bus. The auxiliary storage device 15 stores the inference model 31, the correct label information 32, and the inference result information 33 (all of which are described in detail later).

The inference unit 21, the correct label information creation unit 22, the learning unit 23, the product detection unit 24, the error calculation unit 25, the learning continuation determination unit 26, and the inference model output unit 27 in the main storage device 14 are programs. In the following, when the operation subject is described as "○○ part is", it is the process described later after the central control unit 11 reads "○○ part" from the auxiliary storage device 15 and loads it into the main storage device 14. Means to execute. The image pickup apparatus 2 takes an image of the product 5 and the worker 3 (not shown in FIG. 2) on the belt conveyor 4 of the inspection line 6. The image pickup device 2 transmits the captured image to the learning device 1 via the wired or wireless network 7. The image acquisition device 16 of the learning device 1 receives an image.

(Correct label information)
The correct label information 32 will be described with reference to FIG. The correct label information 32 is associated with the imaging time point stored in the imaging time point column 101, the image storage position is in the image storage position column 102, the product ID is in the product ID (Identifier) column 103, and the product position coordinate column 104. The product position coordinates are stored in, the determination result is stored in the determination result column 105, and the grippable position coordinates are stored in the grippable position coordinate column 106. In the field of machine learning, specifying data (teacher data) that should be used as a model is generally expressed as "attaching a correct answer label to the data". This is the basis for naming the "correct label information" here.

The imaging time point in the imaging time point column 101 is the hour, minute, and second at the time when the image pickup apparatus 2 acquires the image of the product. The time of imaging may include the date of time when the image of the product was acquired.
The image storage position in the image storage position column 102 is information indicating a position where the image of the product is stored. The image of the product may be stored in the auxiliary storage device 15 of the learning device 1 or may be stored in another external device.
The product ID in the product ID column 103 is an identifier that uniquely identifies the product.

The product position coordinates in the product position coordinate column 104 are coordinate values indicating the positions of the products on the screen of the image pickup apparatus 2. For example, "(X1, Y1, X2, Y2)" means a rectangle having a line segment connecting the points (X1, Y1) and the points (X2, Y2) as diagonal lines. Here, the product position coordinates are made two-dimensional for the sake of simplification of the explanation, and the product position coordinates may be three-dimensional.
The determination result in the determination result column 105 is either "0" or "1". “0” indicates a defective product, and “1” indicates a non-defective product. The determination result here is the result of the determination by the worker. If the worker excludes the product, the determination result will be "0", and if the worker does not exclude the product, the determination result will be "1" (details will be described later). The determination result here corresponds to the "pass / fail of inspection" in the claim.

The grippable position coordinates in the grippable position coordinate column 106 are coordinate values indicating the grippable position of the product on the partial screen from which the product is cut out from the screen of the image pickup apparatus 2. That is, when the product position coordinates are global coordinates, it can be said that the grippable position coordinates are local coordinates. Of course, other than this example, the grippable position coordinates may be the same global coordinates as the product position coordinates. The grippable position is the position where the worker grips the product by hand for the purpose of inspecting the product. There are usually two or more grippable positions for a product. These grippable positions correspond to the joint positions of the hands (see reference numeral 41 in FIG. 4). The grippable position coordinates here are two-dimensional for the sake of simplification of the explanation, and the grippable position coordinates may be three-dimensional. The grippable position coordinates here are the result of the worker's movement.
The "recognition result by a person" in the claim corresponds to the determination result and the grippable position coordinates in FIG.

The product has a considerable weight. Therefore, the worker must grip the product in a position where it is difficult to drop it. Incomplete gripping may also increase the number of defective parts of the product. Therefore, the worker must grip the product in a structurally robust position. Further, the worker is required to inspect the required part of the product in a short time and in one operation. Therefore, the worker must hold the product in a position where it does not need to be changed. Skilled workers know these positions empirically.

From FIG. 3, for example, the following can be seen.
-The image pickup apparatus 2 acquired a total of four images, one at 12:00:00, 12:00:01, 12:00:02, and 12:00:03.
-The product was not shown in the image at 12:00:00.
-Product P001 was shown in the image at 12:00:01. Workers began to contact product P001 at two positions "(X3, Y3)" and "(X4, Y4)". "Contact" is the initial stage of "grasping".

-Products P001, P002 and P003 were shown in the image at 12:00:02.
-At 12:00:02, the worker in addition to the above-mentioned two positions, four more positions "(X7, Y7)", "(X8, Y8)", "(X9, Y9)" and " (X10, Y10) ”contacted (grasped) the product P001. The worker eliminated the product P001 in the defective collection yard.
At 12:00:02, the worker touched (picked) product P002 at two positions "(X13, Y13)" and "(X14, Y14)". The worker determined that the product P002 was a non-defective product.
At 12:00:02, the worker touched (picked) product P003 at two positions "(X17, Y17)" and "(X18, Y18)". The worker determined that the product P003 was a non-defective product.

-The image pickup device 2 fixes the image pickup screen by fixing its optical axis and angle of view. Then, the products are moving in chronological order on the image pickup screen. For example, between 12:00:01 and 12:00:02, the product P001 moves from "(X1, Y1, X2, Y2)" to "(X5, Y5, X6, Y6)". There is. From 12:00:02 to 12:00:03, the product P003 has moved from "(X15, Y15, X16, Y16)" to "(X19, Y19, X20, Y20)". During this time, the worker continues to hold the product P003 at the same positions "(X17, Y17)" and "(X18, Y18)" (local coordinates in column 106 have not changed).

(Gripable position coordinates)
The grippable position coordinates and the frameout will be described with reference to FIG. A worker puts his or her hand in contact with the product 5 (upper figure of FIG. 4). In this figure, the worker only touches the product 5, but the worker may grasp and lift the product 5. At this time, the position where the joint 41 comes into contact with the product 5 is the grippable position. When the worker determines that the product 5 is a non-defective product, the worker releases the product 5. Then, as shown in the lower figure of FIG. 5, the product 5 disappears to the right side 42 of the screen (frame out).

(Inference result information)
The inference result information 33 will be described with reference to FIG. The structure of the inference result information 33 is the same as that of the correct label information 32 (FIG. 3). However, the data in each column of the inference result information 33 differs from the data in the same column of the correct answer label information 32 in the following points.

The determination result (column 105) in FIG. 3 is based on the determination by the worker as described above. On the other hand, the determination result (column 115) in FIG. 5 is based on the output value output by the inference model in response to the inference unit 21 inputting the image of the product into the inference model 31.
The grippable position coordinates (column 106) in FIG. 3 are based on the operation by the operator as described above. On the other hand, the grippable position coordinates (column 116) in FIG. 5 are based on the output value output by the inference model in response to the inference unit 21 inputting the image of the product into the inference model.
The "recognition result by the inference model" in the claim corresponds to the determination result and the grippable position coordinates in FIG.

(Overall processing procedure)
The whole processing procedure will be described with reference to FIG.
In step S201, the inference unit 21 of the learning device 1 prepares an unlearned inference model 31. Specifically, the inference unit 21 prepares an inference model 31 whose parameters have not been learned yet.

In step S202, the correct answer label information creating unit 22 of the learning device 1 transmits an inspection instruction to the worker. Specifically, the correct answer label information creation unit 22 transmits an instruction to start the inspection work to the terminal device or the like carried by the worker.

In step S203, the learning unit 23 of the learning device 1 inputs the initial parameters to the inference model 31. Specifically, the learning unit 23 inputs, for example, a parameter randomly created using a random number table into the inference model 31.
At the stage when step S203 is completed, the image pickup apparatus 2 transmits the image captured by the product to the learning apparatus 1 frame by frame. The image acquisition device 16 of the learning device 1 receives images frame by frame.

In step S204, the product detection unit 24 of the learning device 1 creates a product ID and the like. Details of step S204 will be described later as a product ID creation processing procedure (FIG. 7). Here, as a result, it is assumed that the product detection unit 24 has acquired the product ID, the image storage position, the imaging time point, and the product position coordinates given to the product flowing on the belt conveyor 4. At the stage of step S204, one record of the correct label information 32 in FIG. 3 (columns 105 and 106 are blank) is newly created, and one record of the inference result information 33 in FIG. 5 (column 115). And column 116 is blank) is newly created.

In step S205, the correct answer label information creating unit 22 of the learning device 1 creates the correct answer label information 32. Details of step S205 will be described later as a procedure for creating correct label information (FIG. 8). Here, as a result, the correct answer label information creating unit 22 has completed one record (all columns are filled) of the correct answer label information 32 in FIG.

In step S206, the inference unit 21 of the learning device 1 creates the inference result information 33. Details of step S206 will be described later as an inference result information creation processing procedure (FIG. 9). Here, as a result, the inference unit 21 has completed one record (all columns are filled) of the inference result information 33 in FIG.

In step S207, the learning unit 23 of the learning device 1 determines whether or not the number of conveyed products exceeds the first reference value. The first reference value is the number of learning data that causes the inference model to estimate the determination result and the grippable position coordinates in a state where the parameters of the inference model are fixed. For example, the first reference value corresponds to the number of questions in one test that the teacher imposes on the students.

The learning unit 23 counts the number of product IDs (first counter value) created in step S204. When the first counter value reaches the first reference value (step S207 “Yes”), the learning unit 23 stores the inference result information 33 and the correct answer label information 32 accumulated up to this point in the auxiliary storage device 15. After storing and returning the first counter value to "0", the process proceeds to step S208. The inference result information 33 and the correct answer label information 32 correspond to the test answer and the answer, respectively. In other cases, the learning unit 23 returns to step S204 (step S207 “No”).

Now, it is assumed that the first reference value is "100". Then, when the cumulative number of transported products exceeds 100, the learning unit 23 proceeds to step S208. After that, the learning unit 23 continues to return to step S204 for a while. Next, the learning unit 23 proceeds to step S208 when the cumulative number of transported products exceeds 200.

In step S208, the learning unit 23 determines whether or not the number of times the parameter has changed reaches the second reference value. The learning unit 23 counts the number of times (second counter value) that has passed through steps S203 or S209. When the second counter value reaches the second reference value (step S208 “Yes”), the learning unit 23 returns the second counter value to “0” and proceeds to step S210. In other cases (step S208 “No”), the learning unit 23 proceeds to step S209. For example, the second reference value corresponds to the number of tests the teacher imposes on the student. Now, it is assumed that the second reference value is "50". Then, the learning unit 23 proceeds to step S209 until the cumulative number of the transported products exceeds 5,000. In the meantime, the parameters take 50 different values, including the initial values.

In step S209, the learning unit 23 creates a new parameter. Specifically, the learning unit 23 inputs, for example, a parameter randomly created using a random number table into the inference model. At this time, the learning unit 23 may create a parameter by using a genetic algorithm (details will be described later). After that, the learning unit 23 returns to step S203.

While the number of repetitions of the loop in steps S203 to S209 is small, the error between the determination result and the grippable position coordinates in the inference result information 33 and the determination result and the grippable position coordinates in the correct label information 32 is large. However, at this stage, the robot is not actually inspecting using the inference model, so there is no actual harm.

In step S210, the error calculation unit 25 of the learning device 1 calculates the error of the plurality of inference result information 33. At the stage of step S210, the auxiliary storage device 15 stores 50 inference result information 33 and 50 correct label information 32 corresponding to each of them. Therefore, the error calculation unit 25 creates 50 combinations of data "(s _i , _{Pi, e1 i, e2 i ) "} . “S _i ” means the i-th inference result information (i = 1, 2, 3, ..., 50). “P _i ” indicates the parameter of the i-th inference model. “ _Pi ” has, for example, a matrix type of the number of nodes × the number of parameters of each node.

“E1 _i ” is a frequency (number of products) different from each other when the determination result of the i-th inference result information is compared with the determination result of the i-th correct answer label information. It should be noted that 0 ≦ e1 _i ≦ 100 is established. “E2 _i ” is a frequency different from each other when the grippable position coordinates of the i-th inference result information are compared with the grippable position coordinates of the i-th correct answer label information. It should be noted that 0 ≦ e2 _i ≦ 100 is established. There may be multiple grippable position coordinates. Therefore, the error calculation unit 25 considers that all of the grippable position coordinates of the inference result information are different from each other when they are separated from any of the grippable position coordinates of the correct label information by a predetermined threshold value or more. As a matter of course, out of the 50 combinations, " _Pi " that minimizes e1 _i and e2 _i is suitable as a parameter of the inference model. “E1 _i ” and “e2 _i ” are called errors.

In step S211 the error calculation unit 25 specifies the parameter of the inference model having the minimum error. Specifically, the error calculation unit 25 has "E _i = w ₁ x e1 _i + w ₂ x e2 _i " from "P _i " (i = 1, 2, 3, ..., 50). Identify the smallest one. “W ₁ ” and “w ₂ ” are weights for error evaluation. It should be noted that "0 ≤ w ₁ ≤ 1, 0 ≤ w ₂ ≤ 1, w ₁ + w ₂ = 1" is established.

In step S212, the learning continuation determination unit 26 of the learning device 1 determines whether or not the error is equal to or less than the threshold value. Specifically, the learning continuation determination unit 26 proceeds to step S213 when the error Ei of the parameter of the inference model specified in step _S211 is equal to or less than a predetermined threshold value (step S212 “Yes”). In other cases, the learning continuation determination unit 26 returns to step S209 (step S212 “No”).

In step S213, the inference model output unit 27 of the learning device 1 transmits the inference model to the robot. Specifically, the inference model output unit 27 transmits the inference model and the parameters of the inference model specified in step S211 to the robot. In some cases, the robot does not need to perform an immediate inspection. In that case, the inference model output unit 27 may store the parameters of the inference model specified in step S211 in the auxiliary storage device 15, or may transmit the parameters to any external device.

In step S214, the inference model output unit 27 transmits an inspection instruction to the robot. Specifically, the inference model output unit 27 transmits an instruction to the robot to perform the inspection using the inference model and the parameters of the inference model transmitted in step S213.

In step S215, the correct answer label information creation unit 22 of the learning device 1 transmits a pause instruction to the worker. After that, the whole processing procedure is terminated.

(Genetic algorithm, etc.)
When returning to step S209 via step S212 “No”, it means that a plurality of combinations “( _{si, Pi, e1 i , e2 i ) ”} have been acquired. Therefore, the learning unit 23 of the learning device 1 selects the parameter P _i (parents) of the combination having the smallest value of “E _i ” and the parameter P _i (parents) of the second smallest combination. Parents may be multiplied to create a new parameter _Pi (child).

(Product ID creation processing procedure)
The product ID creation processing procedure will be described with reference to FIG. 7. The product ID creation processing procedure is the details of step S204.
In step S301, the product detection unit 24 of the learning device 1 determines whether or not the product is present in the image. Specifically, first, the product detection unit 24 acquires an image of the product from the image pickup device 2.
Second, the product detection unit 24 proceeds to step S302 when at least one product is present in the image acquired by the image pickup device 2 (step S301 “Yes”), and proceeds to the other case (step S301 “No”). ), End the product ID creation processing procedure.

In step S302, the product detection unit 24 determines whether or not the detected product exists in the immediately preceding image. Specifically, the product detection unit 24 proceeds to step S303 when the product is present in the immediately preceding (one frame before) image (step S302 “Yes”), and proceeds to other cases (step S302 “No”). , Step S304. The product detection unit 24 may use an area-based tracking method such as template matching or a Mean-shift method when determining product identity, or may use a feature point-based tracking method such as SURF Tracking. You may use it.

In step S303, the product detection unit 24 assigns the same product ID. Specifically, the product detection unit 24 assigns the product ID already attached to the product to the product.

In step S304, the product detection unit 24 assigns a new product ID. Specifically, the product detection unit 24 assigns a new product ID to the product after assigning a new product ID.

In step S305, the product detection unit 24 transmits the product ID and the like. Specifically, first, the product detection unit 24 cuts out an image of each product from the image acquired in "first" of step S301, and the product position coordinates (coordinate values of both ends of the diagonal line) of the cut out image. ).
Second, the product detection unit 24 creates a new record of the correct label information 32 (FIG. 3), and stores the time of imaging, the image storage position, the product ID, and the product position coordinates in the new record. The product detection unit 24 leaves the determination result column 105 and the grippable position coordinate column 106 of the new record blank.

Third, the product detection unit 24 creates a new record of the inference result information 33 (FIG. 5), and stores the time of imaging, the image storage position, the product ID, and the product position coordinates in the new record. The product detection unit 24 leaves the determination result column 115 and the grippable position coordinate column 116 of the new record blank. After that, the product ID creation processing procedure is terminated.

The product detection unit 24 repeats the product ID creation processing procedure for all the products in the image of a certain frame. Further, the product detection unit 24 repeats the product ID creation processing procedure for each frame received from the image pickup apparatus 2.

(Procedure for creating correct label information)
The procedure for creating correct label information will be described with reference to FIG. The procedure for creating correct label information is the details of step S205.
In step S401, the correct label information creating unit 22 of the learning device 1 extracts the joint. Specifically, the correct answer label information creation unit 22 acquires the position coordinates of the knuckle of the worker.

In step S402, the correct label information creating unit 22 determines whether or not the joint is in contact with the product. Specifically, the correct label information creation unit 22 proceeds to step S403 when at least one of the joints is in contact with the product (step S402 “Yes”), and proceeds to step S403 in other cases (step S402 “No”). The process proceeds to step S404.

In step S403, the correct answer label information creating unit 22 stores the grippable position coordinates. Specifically, the correct answer label information creating unit 22 is in contact with the product in the grippable position coordinate column 106 of the new record of the correct answer label information 32 (FIG. 3) created in “second” of step S305. Memorize joint position information.

In step S404, the correct label information creation unit 22 determines whether or not all the products detected in the immediately preceding image are present in the current image. Specifically, the correct label information creation unit 22 ends the correct label information creation processing procedure when all the products detected in the immediately preceding image are present in the current image (step S404 “Yes”). .. In other cases (step S404 “No”), the correct answer label information creating unit 22 proceeds to step S405.

In step S405, the correct label information creation unit 22 identifies a product that no longer exists. Specifically, the correct label information creation unit 22 identifies a product that no longer exists in the current image among the products that existed in the immediately preceding image. The product specified here is called an "disappearing product".

In step S406, the correct label information creating unit 22 determines whether or not the product has been framed out. Specifically, first, the correct label information creation unit 22 determines whether or not the extinguished product is a frame-out product. "Flameout" means that the extinguished product hangs on the right side of the image frame (downstream side of the conveyor belt) in the previous image, or is within a predetermined distance from the right side even if it does not hang. (See reference numeral 42 in FIG. 4). When a product is framed out, it means that the worker has determined that the product is non-defective and has forgotten to flow to the right on the conveyor belt 4. The fact that the product does not frame out means that the worker has determined that the product is defective and has removed it from the belt conveyor 4.

Second, the correct label information creating unit 22 proceeds to step S407 when the extinguished product is out of frame (step S406 “Yes”), and proceeds to step S408 in other cases (step S406 “No”).

In step S407, the correct label information creating unit 22 determines that the product is a non-defective product. Specifically, the correct answer label information creating unit 22 stores “1” in the determination result column 105 of the new record of the correct answer label information 32 created in the “second” of step S305. After that, the procedure for creating correct label information is terminated.

In step S408, the correct label information creating unit 22 determines that the product is defective. Specifically, the correct answer label information creating unit 22 stores “0” in the determination result column 105 of the new record of the correct answer label information 32 created in the “second” of step S305. After that, the procedure for creating correct label information is terminated.
The correct label information creation unit 22 repeats the correct label information creation processing procedure for all the products in the image of a certain frame. Further, the correct answer label information creating unit 22 repeats the correct answer label information creating processing procedure for each frame received from the image pickup apparatus 2.

(Procedure for creating inference result information)
The procedure for creating inference result information will be described with reference to FIG. 9. The inference result information creation processing procedure is the details of step S206.
In step S501, the inference unit 21 of the learning device 1 estimates the determination result. Specifically, the inference unit 21 inputs an image of the product into the inference model 31. Then, the inference model outputs "1" or "0" as the determination result. The inference unit 21 receives the determination result.

In step S502, the inference unit 21 estimates the grippable position coordinates. Specifically, the inference unit 21 inputs an image of the product into the inference model 31. Then, the inference model outputs the grippable position coordinates. The inference unit 21 receives the grippable position coordinates.

In step S503, the inference unit 21 stores the determination result and the grippable position coordinates. Specifically, first, the inference unit 21 receives the determination result in step S501 in the determination result column 115 of the new record of the inference result information 33 (FIG. 5) created in the “third” of step S305. Remember.
Second, the inference unit 21 stores the grippable position coordinates received in step S502 in the grippable position coordinate column 116 of the new record.
After that, the inference result information creation processing procedure is terminated.
The inference unit 21 repeats the inference result information creation processing procedure for all the products in the image of a certain frame. Further, the inference unit 21 repeats the inference result information creation processing procedure for each frame received from the image pickup apparatus 2.

(Effect of this embodiment)
The effects of the learning device of this embodiment are as follows.
(1) The learning device can automate the labeling of the learning data.
(2) The learning device can determine whether the product is good or bad.
(3) The learning device can determine the position where the product is gripped.
(4) The learning device can create an inference model with a small error.
(5) The learning device can output the inference model to the robot or the like.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the configurations described. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above-mentioned configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.
In addition, the control lines and information lines indicate what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

1 Learning device 2 Imaging device 3 Worker 4 Belt conveyor 5 Product 6 Inspection line 11 Central control device 12 Input device 13 Output device 14 Main memory device 15 Auxiliary storage device 16 Image acquisition device 21 Reasoning unit 22 Correct label information creation unit 23 Learning Part 24 Product detection part 25 Error calculation part 26 Learning continuation judgment part 27 Inference model output part 31 Inference model 32 Correct label information 33 Inference result information

Claims (5)

An inference unit that acquires the recognition result by the inference model from the inference model that inputs the image of the product and outputs the recognition result for the product.
A correct label information creation unit that acquires a recognition result by the person from the image of the product and the image of the person's behavior with respect to the product.
A learning unit that learns the inference model based on the recognition result by the inference model and the recognition result by the person.
Equipped with
The correct label information creation unit
In the image of the product and the image of the behavior of a person with respect to the product, as a result of the person touching the product and then releasing the product on the inspection line for transporting the product, the product is on the inspection line. When it is detected that the product has flowed in a predetermined direction and disappeared from the screen, it is judged that the product is a good product.
A learning device featuring. An error calculation unit that calculates an error between the recognition result by the inference model and the recognition result by the person,
Based on the calculated error, a learning continuation determination unit that determines whether or not to continue learning the inference model, and a learning continuation determination unit.
The learning device according to claim 1 , further comprising. Provided with an inference model output unit that outputs the learned inference model to an external device.
2. The learning device according to claim 2 . The reasoning part of the learning device is
The recognition result by the inference model is acquired from the inference model that inputs the image of the product and outputs the recognition result for the product.
The correct label information creation unit of the learning device
The recognition result by the person is acquired from the image of the product and the image of the behavior of the person with respect to the product.
The learning unit of the learning device is
The inference model is learned based on the recognition result by the inference model and the recognition result by the person.
The correct label information creation unit
In the image of the product and the image of the behavior of a person with respect to the product, as a result of the person touching the product and then releasing the product on the inspection line for transporting the product, the product is on the inspection line. When it is detected that the product has flowed in a predetermined direction and disappeared from the screen, it is judged that the product is a good product.
A learning method of a learning device characterized by. For the inference part of the learning device
From the inference model that inputs the image of the product and outputs the recognition result for the product, the process of acquiring the recognition result by the inference model is executed.
For the correct label information creation unit of the learning device
A process of acquiring a recognition result by the person from the image of the product and the image of the person's behavior with respect to the product is executed.
For the learning unit of the learning device
Based on the recognition result by the inference model and the recognition result by the person, the process of learning the inference model is executed.
For the correct label information creation unit
In the image of the product and the image of the behavior of a person with respect to the product, as a result of the person touching the product and then releasing the product on the inspection line for transporting the product, the product is on the inspection line. When it is detected that the product has flowed in a predetermined direction and disappeared from the screen, the process of determining that the product is non-defective is executed.
A learning program for operating a learning device characterized by.

Images