JP2023172115A - Object detection model training device, object detecting device, and object detection model training method - Google Patents

Abstract

To provide an object detection model training device capable of improving training accuracy of an object detection model.SOLUTION: In an object detection model training device, a training data generating unit 100 generates training data by using computer graphics to synthesize images of a detection target object, and generates a first teaching signal related to the detection target object, and a second teaching signal corresponding to a state of the detection target object during imaging. An object detection model comprises a first inferring unit 161 for outputting a first inferred result relating to the detection target object, and a second inferring unit 162 for outputting a second inferred result relating to the state of the detection target object during imaging. The first inferring unit 161 obtains the first inferred result using an input image and the second inferred result. An object detection training unit 150 trains the second inferring unit 162 by providing the training data to the second inferring unit 162, and trains the first inferring unit 161 by providing the training data and the second inferred result to the first inferring unit 161.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object detection model learning device, an object detection device, and an object detection model learning method.

Conventionally, computer graphics (CG) has been used to generate learning data when learning object detection using image recognition. For example, Patent Document 1 states, ``In order to obtain a trained model used when performing object detection processing, posture detection processing, etc., a large amount of learning data required during learning processing can be obtained in a short time. Provides a learning data generation system.", "The learning data generation system acquires a background image obtained by imaging a three-dimensional space.The learning data generation system also acquires a background image obtained by imaging a three-dimensional space. Acquire CG object generation data, which is processing data. Generate a CG object image based on the acquired CG object generation data.So that the CG object is placed at a predetermined position in three-dimensional space. , a rendered image obtained by combining a CG object image with a background image is acquired as a learning image.''

Japanese Patent Application Publication No. 2020-119127

According to the conventional technology, it is possible to achieve learning accuracy equivalent to the case where an actual image obtained by actually capturing an object to be detected is used as learning data. However, conventional techniques do not achieve higher learning accuracy than learning using real images.

An object of the present invention is to improve the learning accuracy of an object detection model.

In order to achieve the above object, one of the typical object detection model learning devices of the present invention includes an object detection learning section that learns an object detection model that detects an object from an input image, and a learning device used for the learning. a learning data generating section that generates data, the learning data generating section generates the learning data by synthesizing an image of an object to be detected using computer graphics, and , generates a first teacher signal regarding the object to be detected, and generates a second teacher signal corresponding to the state of the object to be detected at the time of imaging with respect to the learning data, and the object detection model: A first inference unit that outputs a first inference result regarding the object to be detected with respect to the input image, and a second inference result regarding the state of the object to be detected at the time of imaging with respect to the input image. a second inference unit, the first inference unit uses the input image and the second inference result to obtain the first inference result, and the object detection learning unit , a second error calculation unit that calculates, as a second error, a difference between a second inference result obtained by applying the learning data to the second inference unit and the second teacher signal; a first error calculation unit that calculates, as a first error, a difference between a first inference result obtained by applying the first inference data and the second inference result to the first inference unit and the first teacher signal; and an inference parameter updating unit that updates the parameters of the second inference unit based on the second error and updates the parameters of the first inference unit based on the first error. It is characterized by
Further, one of the representative object detection devices of the present invention includes a first inference unit that outputs a first inference result regarding an object to be detected with respect to an input image; a second inference unit that outputs a second inference result regarding the state at the time of imaging, the first inference unit using the input image and the second inference result to determine the first inference result. It is characterized by obtaining inference results.
An object detection learning method for learning an object detection model that detects an object from an input image, the method comprising: generating the training data by synthesizing images of objects to be detected using computer graphics; for the data, generating a first teacher signal related to the object to be detected; for the learning data, generating a second teacher signal corresponding to the state of the object to be detected at the time of imaging; providing the learning data to a second inference unit included in the object detection model to obtain a second inference result regarding the state of the object to be detected at the time of imaging; providing the learning data and the second inference result to the inference unit of the invention to obtain a first inference result regarding the object to be detected; and the first inference result and the first teacher signal. a step of obtaining the difference between the second inference result and the second teacher signal as a second error; and a step of obtaining the difference between the second inference result and the second teacher signal as a second error, and The method is characterized by comprising the steps of: updating parameters of the inference section of the inference section; and updating parameters of the first inference section based on the first error.

According to the present invention, the learning accuracy of an object detection model can be improved.

FIG. 1 is a configuration diagram of an object detection model learning device. FIG. 2 is a configuration diagram of the object detection device. FIG. 3 is an explanatory diagram of CG synthesis parameter constraints. FIG. 4 is a specific example of the first teacher signal selection condition. FIG. 5 is a specific example of the second teacher signal selection condition. FIG. 6 is a flowchart showing the learning processing procedure. FIG. 7 is a flowchart showing the processing procedure for object detection. FIG. 8 shows a specific example of the teacher signal. FIG. 9 is a specific example of a CG composite image. FIG. 10 is an explanatory diagram of the inference results. FIG. 11 is a configuration diagram of an object detection model learning device according to a modified example. FIG. 12 is a configuration diagram of an object detection device according to a modification.

Embodiments of the present invention will be described below with reference to the drawings. The embodiments described below do not limit the claimed invention, and all of the elements and combinations thereof described in the embodiments are essential to the solution of the invention. is not limited. Additionally, illustrations and explanations of well-known configurations that are essential to the configuration of the invention may be omitted.

In the following description, information such as an "xxx table" may be used to describe information from which an output is obtained in response to an input, but this information may be data having any structure. Therefore, the "xxx table" can be called "xxx information."

In addition, in the following explanation, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. good.

In addition, in the following description, processing may be explained using "program" as the subject. The program is executed by the processor unit to carry out predetermined processing using the storage unit and/or interface unit as appropriate, so the subject of the processing is the processor unit (or the controller that has the processor unit). devices such as ).

The program may be installed on a device such as a computer, or may be located on, for example, a program distribution server or a computer-readable (eg, non-transitory) recording medium. Furthermore, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

Further, the "processor section" is one or more processors. The processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). Furthermore, the processor may be single-core or multi-core. Further, the processor may be a processor in a broad sense such as a hardware circuit (eg, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

In addition, in the following explanation, when the same type of elements are explained without distinguishing them, reference numerals (or common numerals among the reference numerals) are used, and when the same kind of elements are explained separately, the element An identification number (or reference number) may be used. Moreover, the number of each element shown in each figure is an example, and is not limited to the number shown in the figure.

FIG. 1 is a configuration diagram of an object detection model learning device. The object detection model learning device shown in FIG. 1 includes an object detection learning section 150 and a learning data generation section 100. The object detection learning unit 150 performs learning of an object detection model that detects objects from input images. The learning data generation unit 100 generates learning data that the object detection learning unit 150 uses for learning.

The learning data generation unit 100 generates learning data by synthesizing images of objects to be detected using computer graphics. In addition, the learning data generation unit 100 generates, with respect to the learning data, a first teacher signal related to the object to be detected, and a second teacher signal corresponding to the state of the object to be detected at the time of imaging.

The object detection learning section 150 includes an object detection section 160, a first error calculation section 170, a second error calculation section 171, and an inference parameter update section 180.
The object detection unit 160 is a processing unit corresponding to an object detection model, and includes a first inference unit 161 and a second inference unit 162.

The second inference unit 162 outputs a second inference result regarding the state of the object to be detected at the time of imaging with respect to the input image. The first inference unit 161 outputs a first inference result regarding the object to be detected with respect to the input image. At this time, the first inference unit 161 uses the input image and the second inference result to obtain a first inference result. For example, if the second inference unit 162 outputs the position of the camera, the direction of illumination, etc. as an inference result, the first inference unit 161 considers the position of the camera and the direction of illumination, and outputs the detection target. Infer the presence or absence of objects.

Known object detection techniques can be used for the first inference section 161 and the second inference section 162. Examples include SSD (Single Shot MultiBox Detector) and YOLO (You only look once: Unified, real-time object detection).
These technologies utilize neural networks, and when an image is input, they output the type of detection target and the range of the detection target on the image as numerical information. Since the numerical information that can be output is not limited to these types, it can be used not only in the first inference section 161 but also in the second inference section 162.

The second error calculation unit 171 calculates, as a second error, the difference between the second inference result obtained by giving the learning data as an input image to the second inference unit 162 and the second teacher signal. .
The first error calculation unit 170 calculates the difference between the first inference result obtained by giving the learning data and the second inference result to the first inference unit 161 and the first teacher signal. Obtained as an error.

The inference parameter update unit 180 updates the parameters of the second inference unit 162 based on the second error, and updates the parameters of the first inference unit 161 based on the first error, thereby updating the parameters of the object detection unit. 160 studies.

The learning data generation section 100 includes a condition input section 110, a CG synthesis parameter constraint storage section 120, a CG synthesis parameter generation section 121, a learning CG generation section 122, a teacher signal selection condition storage section 130, a teacher signal selection section 131, It has a first teacher signal generation section 132 and a second teacher signal generation section 133.

The condition input unit 110 is an interface that accepts input of various conditions related to learning. The various conditions related to learning include CG synthesis parameter constraints, which are constraints regarding computer graphics parameters, and teacher signal selection conditions, which are conditions for selecting a teacher signal. Further, the teacher signal selection conditions include a first teacher signal selection condition and a second teacher signal selection condition.

The CG synthesis parameter constraint storage unit 120 stores the input CG synthesis parameter constraints. The teacher signal selection condition storage unit 130 stores the input teacher signal selection condition. The CG synthesis parameter constraint condition storage section 120 and the teacher signal selection condition storage section 130 may be realized by any storage medium.

The CG synthesis parameter generation section 121 generates a plurality of CG synthesis parameters within the range of the CG synthesis parameter constraint conditions, and outputs them to the learning CG generation section 122.
The learning CG generation unit 122 generates learning data for each of the plurality of CG synthesis parameters generated by the CG synthesis parameter generation unit 121.

For example, when detecting a device, the CG synthesis parameter generation unit 121 generates a CG of the appearance of the device based on CG synthesis parameters that specify the type of device, orientation, lighting direction, etc., and superimposes it on the background image. This will generate training data. The background image may be a real image or a CG image.

The teacher signal selection unit 131 performs processing to select items to be included in the teacher signal based on the CG synthesis parameters and teacher signal selection conditions. The first teacher signal generation unit 132 generates a first teacher signal for the teacher signal item selected by the teacher signal selection unit 131. The second teacher signal generation unit 133 generates a second teacher signal for the teacher signal item selected by the teacher signal selection unit 131.

For example, if the CG synthesis parameter has a value that specifies the device type, orientation, illumination direction, etc., and the type of detection target is specified as the first teacher signal, the value of the device type is the first teacher signal. becomes. Similarly, if the illumination direction is designated as the second teacher signal, the value of the illumination direction becomes the second teacher signal.

Although details will be described later, it is preferable that the first teacher signal includes the type of the object to be detected and the position of the object to be detected in the learning data. Further, it is preferable that the second teacher signal includes at least one of the following: the positional relationship between the object to be detected and the camera, the state of illumination of the object to be detected, and the state of deformation of the object to be detected. .
In other words, the first teacher signal is information used as an output of object detection, such as what is where. On the other hand, the second teacher signal is auxiliary information that contributes to improving the accuracy of object detection.

Note that the second teacher signal may further include the type of the object to be detected and the position of the object to be detected in the learning data. That is, the second teacher signal may include the same information as the first teacher signal. In this case, the second inference unit outputs the information that the first inference unit should output as an inference result, and the first inference unit decides to perform inference again based on the second inference result. Become.

FIG. 2 is a configuration diagram of the object detection device. The object detection device has a configuration in which an image input section 200 and an inference result output section 210 are connected to the object detection section 160 shown in FIG. In this configuration, the image input section 200 provides the same input image to the first inference section 161 and the second inference section 162. The second inference unit 162 infers the state at the time of imaging from the input image and outputs it to the first inference unit 161.

The first inference unit 161 performs image recognition on the input image, taking into account the state at the time of imaging inferred by the second inference unit 162, and infers the presence or absence and position of the object to be detected. The result is output to the inference result output unit 210.
Note that in FIG. 2, the second inference section 162 also outputs to the inference result output section 210. External output from the second inference unit 162 is not essential, but output is possible if the state at the time of imaging is required as an output.

FIG. 3 is an explanatory diagram of CG synthesis parameter constraints. In the example shown in FIG. 3, the CG synthesis parameter constraint condition 300 indicates the range of values that an item can take by associating the minimum and maximum input values with the item name. Note that depending on the item name, a sub-item name is set, and in this case, the minimum value and maximum value of the input value are associated with each sub-item name.

The item names are: type of detection target, number of detection targets, position of detection target, orientation of detection target, deformation parameter of detection target, camera position, camera direction, number of lights, light position, light direction, light Includes range, illuminance, lighting color, etc.

The type of detection target is, for example, "device A".
The number of detection targets is the number of detection targets included in one learning data. In FIG. 3, a range of 1 to 3 is specified.

The position of the detection target indicates the range of spatial coordinates in which the object to be detected is placed. When indicating the position of a detection target using the XYZ coordinate system, X, Y, and Z are used as sub-item names. In FIG. 3, the X coordinate range is 50 to 200, the Y coordinate range is 30 to 100, and the Z coordinate range is 0 to 0.

The direction of the detection target indicates the range of orientations of the detection target. For example, using two subdivisions, horizontal angle φ and vertical angle θ, the range of φ is specified as 10 degrees to 30 degrees, and the range of θ is specified as 0 degrees to 0 degrees.
The deformation parameter of the detection target has the movable part of the detection target as a sub-item name. The deformation parameters differ depending on the type of detection target. For example, if the device A is equipped with a rotating arm as a deformable part, the specific name may be the arm angle, which is in the range of 10 degrees to 50 degrees. The deformation parameter may be a telescopic member, a sliding member, or the like. If there are multiple deformed parts, increase the number of sub-items and set a range for each.

The camera position indicates the spatial coordinates of the virtual camera, that is, the range of the viewpoint from which the CG composite object is viewed. When indicating the position of the camera using the XYZ coordinate system, X, Y, and Z are used as sub-item names. In FIG. 3, the X coordinate range is 0.0 m to 5.0 m, the Y coordinate range is 5.0 m to 10.0 m, and the Z coordinate range is 3.0 m to 3.5 m.

The camera direction indicates the range of camera orientations. For example, using two subdivisions, horizontal angle φ and vertical angle θ, the range of φ is specified as 0 degrees to 360 degrees, and the range of θ is specified as 110 degrees to 135 degrees.

The number of illuminations indicates the range of the number of illuminations in CG composition.
The items of lighting position, lighting direction, irradiation range, illuminance, and lighting color are specified for each number of lights. The position of the illumination corresponds to the position of the light source of the illumination applied to the CG-composed object, and is specified using XYZ coordinates, similar to the position of the detection target and the position of the camera. The direction of illumination is indicated by a horizontal angle φ and a vertical angle θ, similar to the position of the detection target and the direction of the camera.
The illumination range is the angle of the range illuminated by the illumination, and is, for example, 30 degrees to 45 degrees. Illuminance indicates the brightness of illumination, and is, for example, 1000 lm. The color of the illumination may be expressed, for example, in RGB.

FIG. 4 is a specific example of the first teacher signal selection condition. The first teacher signal selection condition 400 shown in FIG. 4 specifies whether or not to select an item that can be learned as the output of the first inference. Items that can be learned as the output of the first inference include the type of detection target and the range of the detection target on the image. If the selection presence/absence value for these items is "yes", a teacher signal for the item is generated.

Specifically, if the type of detection target is "present", the CG synthesis parameter may be referred to and the value of the type of detection target may be used as it is as a teacher signal.
If the value of the type of detection target range on the image is "Yes", the camera's angle of view is determined based on the spatial coordinates of the object to be detected, the spatial coordinates of the camera, and the orientation of the camera by referring to the CG composition parameters. A teacher signal is generated by projecting the image of the detection target onto the plane of the image.

FIG. 5 is a specific example of the second teacher signal selection condition. The second teacher signal selection condition 500 shown in FIG. 5 specifies whether or not to select an item that can be learned as the output of the second inference. Examples of items that can be learned as the output of the second inference are shown below.

“Type of Detection Target” This is the same as the first teacher signal, so the explanation will be omitted.
"Range of detection target on image" This is the same as the first teacher signal, so the explanation will be omitted.
"Deformation Parameter of Detection Target" It is sufficient to refer to the CG synthesis parameter and use the value of the corresponding item as it is as the teacher signal.
"Relative position and relative direction of detection target with respect to camera" This can be calculated based on the spatial coordinates of the object to be detected, the spatial coordinates of the camera, and the orientation of the camera with reference to the CG synthesis parameters.
"Lighting, relative position, and relative direction of illumination with respect to the camera" Can be calculated based on the spatial coordinates of the camera, camera direction, spatial coordinates of illumination, and direction of illumination with reference to CG composition parameters. .
"Illumination, relative position, and relative direction of illumination to the detection target" Can be calculated based on the spatial coordinates of the object to be detected, the spatial coordinates of the illumination, and the direction of the illumination by referring to the CG synthesis parameters. .
"Irradiation range" can be determined by integrating information regarding the camera and multiple lights by referring to the CG synthesis parameters.
"Illuminance" It can be determined by referring to the CG synthesis parameters and integrating information regarding the camera and multiple lights.
"Lighting color" It can be determined by referring to the CG composition parameters and integrating information regarding the camera and multiple lights.

FIG. 6 is a flowchart showing the learning processing procedure. First, the condition input unit 110 receives input of CG synthesis parameter constraint conditions, and stores them in the CG synthesis parameter constraint storage unit 120 (S600). Furthermore, the condition input unit 110 receives the input of the first teacher signal selection condition, and stores it in the teacher signal selection condition storage unit 130 (S601). Similarly, the condition input unit 110 receives the input of the second teacher signal selection condition, and stores it in the teacher signal selection condition storage unit 130 (S602).

The CG synthesis parameter generation unit 121 randomly generates each CG synthesis parameter according to the constraints of the CG generation parameters (S603).
The learning CG generation unit 122 generates a learning CG, which is learning data, using the generated CG synthesis parameters (S604).

The first teacher signal generation unit 132 generates a first teacher signal according to the first teacher signal selection condition using the CG synthesis parameters and the learning CG (S605). This teacher signal is a teacher signal directly related to object detection, and includes information such as the type of object to be detected and the range of the object to be detected on the image.
The second teacher signal generation unit 133 generates a second teacher signal according to the second teacher signal selection condition using the CG synthesis parameters and the learning CG (S606). This teacher signal includes a teacher signal that is not directly related to object detection.
For example, if it is a teacher signal related to a specimen object, the information includes deformation parameters of the detection object, relative position and relative direction of the detection object with respect to the camera, and the like. For example, in the case of a teacher signal related to illumination, the information includes the relative position of each illumination with respect to the camera, the relative direction of irradiation, the relative position of each illumination with respect to the detection target, the relative direction of irradiation, etc. It also includes the irradiation range, illuminance, illumination color, etc.
Furthermore, when the second teacher signal includes a teacher signal related to the detection target, it is desirable that the second teacher signal also includes the first teacher signal. For example, when the second teacher signal contains a deformation parameter of the detection target, the type of the detection target is also required as the second teacher signal because the deformation of the detection target differs depending on the type of detection target. Furthermore, since it is necessary as a teacher signal to know where on the image the influence of the deformation parameter occurs, the range on the image of the detection target is also necessary. Hereinafter, the second teacher signal excluding the portion that overlaps with the first teacher signal will be referred to as a second teacher signal in a narrow sense.

Subsequently, the second inference unit 162 calculates a second inference result using the first inference parameters held at that time and the learning CG (S607). Here, the second inference result is numerical information corresponding to the second teacher signal. Since the second inference result corresponds to the input image, it can also be output as a two-dimensional map. This output will be described later.

The first inference unit 161 calculates a first inference result using the second inference result and the learning CG (S608). Alternatively, the first inference result may be output using only the portion of the second inference result that corresponds to the narrowly defined second teacher signal and the learning CG. The first inference result is numerical information corresponding to the first teacher signal.

Next, the first error calculation unit 170 calculates a first error using the first inference result and the first teacher signal (S609). The second error calculation unit 171 calculates the second error using the second inference result and the second teacher signal (S610).

The inference parameter update unit 180 determines whether termination conditions are satisfied, such as whether the error has become sufficiently small or whether parameter updates, which will be described later, have been performed a predetermined number of times (S611). If it is satisfied, the process ends, and if not, the first inference unit 161 and the second The parameters of the inference unit 162 are updated (S612). After that, the process returns to S603.

FIG. 7 is a flowchart showing the processing procedure for object detection. First, the image input unit 200 receives an image to be inferred as an input image (S700).
The second inference unit 162 calculates a second inference result using the image to be inferred (S701).
The first inference unit 161 calculates a first inference result using the second inference result and the inference target image (S702).
The inference result output unit 210 outputs the first inference result. If the result is to be viewed by a person, it can be output as a screen, or if the inference result is to be used in a system, it can be output to that system via a network.
At this time, the second inference result may be output. Then, detailed information about the detection target can be provided to the above-mentioned people and systems.

FIG. 8 shows a specific example of the teacher signal. In the figure, the type of detection target and the range of the detection target on the image are generated as the first teacher signal. In addition, as a second teacher signal, the type of detection target, the range of the detection target on the image, the deformation parameter of the detection target, the relative position of the detection target with respect to the camera, the relative direction of the detection target with respect to the camera, Relative position of illumination 1 with respect to the camera, relative direction of illumination 1 with respect to the camera, relative position of illumination 1 with respect to the detection target, relative direction of irradiation of illumination 1 with respect to the detection target, illumination 1 The illuminance of , the illumination color of illumination 1, etc. are generated.

FIG. 9 is a specific example of a CG composite image. The image 900 shown in FIG. 9 includes a CG composite image 910 and a CG composite image 911. The CG composite image 910 and the CG composite image 911 are images generated by CG composition of the detection target.

The CG composite image 910 and the CG composite image 911 have the same type of detection target object, but their appearance differs depending on how the object is illuminated. The CG composite image 910 is dark, and the CG composite image 911 is bright.

It is difficult for the object detection unit 160 to learn that objects that look so different in appearance are of the same type. However, the difficulty level decreases when different ways of seeing are given as input. Therefore, the second inference unit 162 infers the difference in appearance and provides the result to the first inference unit 161, thereby improving the accuracy of object detection.
The same is true for the object's orientation with respect to the camera and the object's deformation parameters.

FIG. 10 is an explanatory diagram of the inference results. In the two-dimensional map 1000 of FIG. 10, the element at each position on the map consists of a plurality of fixed-length vectors corresponding to the second inference results. The inference results regarding the entire screen, such as the relative position of the light with respect to the camera and the relative direction of the light with respect to the camera, are stored in a certain dimension of the vector, which spans the entire two-dimensional map. Furthermore, if learning has progressed sufficiently, the inference result corresponding to the CG composite image 910 is stored in another dimension of the vector, and is limited to the area 1010 of the two-dimensional map. Similarly, the inference result corresponding to the CG composite image 911 is limited to the area 1011 of the two-dimensional map.

Next, a modification example in which the object detection section has a shared inference section will be described.
FIG. 11 is a configuration diagram of an object detection model learning device according to a modified example. 11 differs from FIG. 1 in that the object detection section 160 further includes a shared inference section 1100. The output of the learning CG generation section 122 is input to the shared inference section 1100 as learning data.

The shared inference unit 1100 performs predetermined inference processing on the input learning data, and then outputs the result to the first inference unit 161 and the second inference unit 162.
That is, in this configuration, the shared inference unit 1100 executes preprocessing that is common to the first inference unit 161 and the second inference unit 162, thereby making the processing more efficient.
The other configurations and operations are the same as those in FIG. 1, so explanations will be omitted.

FIG. 12 is a configuration diagram of an object detection device according to a modification. 12 differs from FIG. 1 in that the object detection section 160 further includes a shared inference section 1100.
The shared inference unit 1100 receives an input image from the image input unit 200, performs predetermined inference processing on the input image, and then outputs the result to the first inference unit 161 and the second inference unit 162.
That is, in this configuration, the shared inference unit 1100 executes preprocessing that is common to the first inference unit 161 and the second inference unit 162, thereby making the processing more efficient.
The other configurations and operations are the same as those in FIG. 2, so their explanation will be omitted.

As described above, the object detection model learning device disclosed in the embodiment includes an object detection learning unit 150 that performs learning of an object detection model that detects an object from an input image, and a learning unit that generates learning data used for the learning. data generation unit 100.
The learning data generation unit 100 generates the learning data by synthesizing images of the object to be detected using computer graphics, and generates a first teaching signal regarding the object to be detected with respect to the learning data. A second teacher signal corresponding to the state of the object to be detected at the time of imaging is generated for the learning data.
The object detection model includes a first inference unit 161 that outputs a first inference result regarding the object to be detected with respect to the input image, and a first inference result regarding the state of the object to be detected at the time of imaging regarding the input image. a second inference unit 162 that outputs a second inference result, and the first inference unit 161 obtains the first inference result using the input image and the second inference result.
The object detection learning section 150 provides a second inference result obtained by giving the learning data to the second inference section 162 and obtains a difference between the second teacher signal and the second error. The error calculation unit 171 calculates the difference between the first inference result obtained by giving the learning data and the second inference result to the first inference unit 161 and the first teacher signal. a first error calculation unit 170 that calculates the error as an error; updates the parameters of the second inference unit based on the second error; and updates the parameters of the first inference unit based on the first error. The invention is characterized by comprising an inference parameter updating unit 180 that updates the inference parameters.
According to this configuration and operation, since the state of the object to be detected at the time of imaging is used, the learning accuracy of the object detection model can be improved, and higher learning accuracy can be expected than learning using real images.

Further, the learning data generation unit 100 receives constraint conditions regarding the computer graphics parameters, selection conditions for the first teacher signal, and selection conditions for the second teacher signal, and Generate a plurality of computer graphics synthesis parameters within a range, generate the learning data based on the computer graphics synthesis parameters, and use the computer graphics synthesis parameters and the selection condition of the first teacher signal. The first teacher signal is generated using the computer graphics synthesis parameters and the selection conditions for the second teacher signal.
According to such a configuration and operation, it is possible to perform learning regarding the state at the time of imaging using parameters of CG synthesis, and improve the learning accuracy of the object detection model.

Note that it is preferable that the first teacher signal includes the type of the object to be detected and the position of the object to be detected in the learning data.
Further, the second teacher signal may include at least one of a positional relationship between the object to be detected and the camera, a state of illumination for the object to be detected, and a state of deformation of the object to be detected. is suitable.
By performing learning using these parameters as a teacher, the state at the time of imaging can be learned efficiently, and the learning accuracy of object detection can be more effectively improved.

Note that the second teacher signal may further include the type of the object to be detected and the position of the object to be detected in the learning data.
This is because the second parameter to be inferred may be influenced by the type and position of the object to be detected.

The object detection model further includes a shared inference unit 1100, and after the input image is subjected to inference processing by the shared inference unit, the inference result of the shared inference unit is subjected to the first inference process. Processing may be performed by the inference unit 161 and the second inference unit 162.
In this way, by executing preprocessing that is commonly effective for the first inference unit 161 and the second inference unit 162 in the shared inference unit 1100, processing efficiency can be improved.

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible not only to delete such a configuration but also to replace or add a configuration.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing, for example, an integrated circuit. Further, the present invention can also be realized by software program codes that realize the functions of the embodiments. In this case, a recording medium on which a program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored on the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the embodiments described above, and the program code itself and the recording medium storing it constitute the present invention. Examples of recording media for supplying such program codes include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, A non-volatile memory card, ROM, etc. are used.

Furthermore, the program code that implements the functions described in this embodiment can be implemented using a wide range of program or script languages, such as assembler, C/C++, Perl, Shell, PHP, and Java (registered trademark).

In the above-described embodiments, the control lines and information lines are those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. All configurations may be interconnected.

100: Learning data generation unit, 110: Condition input unit, 120: CG synthesis parameter constraint storage unit, 121: CG synthesis parameter generation unit, 122: Learning CG generation unit, 130: Teacher signal selection condition storage unit, 131 : Teacher signal selection unit, 132: First teacher signal generation unit, 133: Second teacher signal generation unit, 150: Object detection learning unit, 160: Object detection unit, 161: First inference unit, 162: First Second inference unit, 170: First error calculation unit, 171: Second error calculation unit, 180: Inference parameter update unit, 200: Image input unit, 210: Inference result output unit, 1100: Shared inference unit

Claims (8)

an object detection learning unit that learns an object detection model that detects objects from input images;
a learning data generation unit that generates learning data used for the learning,
The learning data generation unit includes:
Generate the learning data by synthesizing images of the object to be detected using computer graphics,
generating a first teacher signal regarding the object to be detected for the learning data;
For the learning data, generate a second teacher signal corresponding to the state of the object to be detected at the time of imaging;
The object detection model is
a first inference unit that outputs a first inference result regarding the object to be detected with respect to the input image;
a second inference unit that outputs, regarding the input image, a second inference result regarding the state of the object to be detected at the time of imaging;
The first inference unit determines the first inference result using the input image and the second inference result,
The object detection learning section includes:
a second error calculation unit that calculates, as a second error, a difference between a second inference result obtained by applying the learning data to the second inference unit and the second teacher signal;
a first error that is determined as a first error between a first inference result obtained by giving the learning data and the second inference result to the first inference unit and the first teacher signal; calculation section and
An inference parameter updating unit that updates parameters of the second inference unit based on the second error and updates parameters of the first inference unit based on the first error. An object detection model learning device. The object detection model learning device according to claim 1,
The learning data generation unit includes:
accepting constraints regarding the computer graphics parameters, selection conditions for the first teacher signal, and selection conditions for the second teacher signal;
generating a plurality of computer graphics synthesis parameters within the range of the constraint conditions, and generating the learning data based on the computer graphics synthesis parameters;
generating the first teacher signal using the computer graphics synthesis parameter and the selection condition for the first teacher signal;
An object detection model learning device characterized in that the second teacher signal is generated using the computer graphics synthesis parameter and the selection condition for the second teacher signal. The object detection model learning device according to claim 1,
The object detection model learning device is characterized in that the first teacher signal includes a type of the object to be detected and a position of the object to be detected in the learning data. The object detection model learning device according to claim 1,
The second teacher signal includes at least one of a positional relationship between the object to be detected and a camera, a state of illumination for the object to be detected, and a state of deformation of the object to be detected. An object detection model learning device. The object detection model learning device according to claim 4,
The object detection model learning device is characterized in that the second teacher signal further includes a type of the object to be detected and a position of the object to be detected in the learning data. The object detection model learning device according to claim 1,
The object detection model further includes a shared inference unit, and after the input image is subjected to inference processing by the shared inference unit, the first inference unit and the inference process are performed on the inference result of the shared inference unit. An object detection model learning device characterized in that processing is performed by the second inference section. a first inference unit that outputs a first inference result regarding the object to be detected with respect to the input image;
a second inference unit that outputs, regarding the input image, a second inference result regarding the state of the object to be detected at the time of imaging;
The object detection device is characterized in that the first inference unit obtains the first inference result using the input image and the second inference result. An object detection learning method for learning an object detection model that detects an object from an input image, the method comprising:
generating training data by synthesizing images of objects to be detected using computer graphics;
generating a first teacher signal regarding the object to be detected with respect to the learning data;
generating, for the learning data, a second teacher signal corresponding to the state of the object to be detected at the time of imaging;
providing the learning data to a second inference unit included in the object detection model to obtain a second inference result regarding the state of the object to be detected at the time of imaging;
providing the learning data and the second inference result to a first inference unit included in the object detection model to obtain a first inference result regarding the object to be detected;
obtaining a difference between the first inference result and the first teacher signal as a first error;
obtaining a difference between the second inference result and the second teacher signal as a second error;
updating parameters of the second inference unit based on the second error, and updating parameters of the first inference unit based on the first error;
An object detection model learning method characterized by comprising:

Images