JP2021089476A - Information processing device, imaging device, control method, and program - Google Patents

Abstract

To provide an information processing device capable of executing accurate deduction to image data of various brightness.SOLUTION: An information processing device 100 includes a learning model 20 being a neural network for executing a deduction process by an arithmetic operation using a weighting value 240 for image data, an exposure specification unit 250 for specifying brightness of the image data, and a weight coefficient determination unit 260 which determines a weight coefficient α to be multiplied by the weighting value 240 on the basis of the specified brightness.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device, an imaging device, a control method, and a program having a learning model for executing inference processing.

A process of recognizing a subject included in image data (object recognition process) is used. When the subject recognition process is executed for dark image data, the recognition accuracy is low because the feature amount cannot be acquired appropriately. On the other hand, there is a need for recognition processing for image data having various brightness including dark image data.

In recent years, machine learning methods have been widely used in the field of object recognition processing. Generally, the input image used in the learning stage of machine learning is subjected to preprocessing such as offset processing and gain processing according to the bit width of the image. However, it is difficult to improve the recognition accuracy because the amount of information does not increase even if the image that is dark as a whole and has low contrast is scaled.

In an environment with abundant information processing resources, a learning model with a wide calculation bit width is used to suppress digit loss, and multiple learning models according to the brightness of the image are used to support various brightnesses. be able to. On the other hand, in embedded systems and the like, resources are not abundant, so inference processing should be executed with high accuracy using limited resources.

Patent Document 1 proposes a technique for improving inference accuracy by reducing the gradient disappearance caused by the activation function of a deep neural network.

JP-A-2019-67062

In the technique of Patent Document 1 in which gradient disappearance is reduced by multiplying weight parameters, information processing resources such as circuit scale are increased by providing a plurality of weight parameters in each neuron. There is no particular mention of setting parameters according to the input data.

Further, in an environment where information processing resources are limited, it is difficult to switch a plurality of learning models and adopt a resource usage that corresponds to various brightnesses.

In view of the above circumstances, it is an object of the present invention to provide an information processing device, an imaging device, a control method, and a program capable of performing highly accurate inference on image data having various brightnesses.

In order to achieve the above object, the information processing apparatus of the present invention is a learning model which is a neural network that executes inference processing by calculation using weight values for image data, and exposure specification for specifying the brightness of the image data. It is characterized by having a means and a weight coefficient determining means for determining a weighting coefficient to be multiplied by the weight value based on the specified brightness.

According to the present invention, highly accurate inference can be performed on image data having various brightnesses.

It is a block diagram which shows the structure of the information processing apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing of the learning model (inference device) which concerns on 1st Embodiment of this invention. It is a figure explaining the adjustment of the weight included in the learning model which concerns on 1st Embodiment of this invention. It is explanatory drawing which illustrates the inference processing (object recognition processing) by a prior art. It is explanatory drawing which illustrates the inference processing (object recognition processing) in 1st Embodiment of this invention. It is a flowchart of the process concerning determination of a weighting coefficient in 1st Embodiment of this invention. It is a figure which shows the relationship between the brightness and the weighting coefficient in the 1st Embodiment of this invention. It is explanatory drawing which illustrates the inference processing (object recognition processing) in the 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Each embodiment described below is merely an example of a configuration in which the present invention can be realized. Each of the following embodiments can be appropriately modified or modified according to the configuration of the apparatus to which the present invention is applied and various conditions. Therefore, the scope of the present invention is not limited by the configurations described in each of the following embodiments. For example, a configuration in which a plurality of configurations described in the embodiment are combined can be adopted as long as there is no mutual contradiction.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration of an information processing device 100 according to a first embodiment of the present invention. Generally, in the first embodiment, the accuracy of estimation by the inference device provided in the information processing apparatus 100 is improved by adjusting the weighting coefficient of the neural network according to the brightness of the input image.

The information processing device 100 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, an input unit 105, a display unit 106, and a system bus 107. The information processing device 100 may be an image pickup device having an image pickup optical system and an image pickup element to generate image data, or a terminal (personal computer or the like) that acquires image data from the outside via a network or the like. It may be.

The CPU 101 is a processor capable of executing various arithmetic processes, and functions as a control unit that integrally controls elements provided in the information processing apparatus 100.

The ROM 102 is a non-volatile storage medium, and is composed of elements such as a flash memory and EEPROM, and stores a program used for controlling the information processing apparatus 100.

The RAM 103 is a volatile storage medium and functions as a working memory used by the CPU 101 for calculation.

The HDD 104 is an internal storage of the information processing device 100 and stores various data and control information. The HDD 104 may store a program used for controlling the information processing apparatus 100.

The input unit 105 is an interface for inputting instructions from the user and data (for example, image data) from another device.

The display unit 106 is a display unit that displays various information acquired and generated by the operation of the information processing device 100, and is composed of, for example, a liquid crystal display.

The system bus 107 is a transmission line that interconnects the above-mentioned elements of the information processing apparatus 100.

The learning model 20, the functional block, and various processes according to the present embodiment described below are realized by the CPU 101 expanding the program stored in the ROM 102 or the HDD 104 into the RAM 103 and executing the program. Further, various processes according to the present embodiment are started by, for example, a user operating the input unit 105 to instruct the CPU 101, and the results of the various processes are output to the display unit 106.

FIG. 2 is an explanatory diagram of the learning model 20 (inference device) according to the first embodiment of the present invention. The learning model 20 is a neural network having an input layer 210, an intermediate layer 220, and an output layer 230 (hereinafter, may be abbreviated as NN), and is a trained model learned by supervised learning. In the trained model of the present embodiment, the input data is image data, and the teacher data is information indicating the type of subject (human, animal, tree, etc.) reflected in the image, for example, the trained model is trained according to an algorithm of a convolutional neural network. To do. The learning model 20 is defined by a program indicating the algorithm described below and parameters used during the inference process.

The input layer 210 is a layer including a plurality of nodes u1 and u2 into which image data is input, and outputs image data to the intermediate layer 220. More specifically, the input layer 210 is input with data obtained by converting the pixel values in the image data into a matrix. The nodes u1 and u2 of the input layer 210 output the input data to the intermediate layer 220, respectively.

The intermediate layer 220 is a layer including a plurality of neurons v1 and v2 (nodes), and executes arithmetic processing (product-sum operation, non-linear operation by activation function, etc.) on the input data supplied from the input layer 210. , Output to the output layer 230. That is, the intermediate layer 220 multiplies the input data by the weight values 240 (uv11, uv12, uv21, uv22) set for each edge, which is the path between the input layer 210 and the intermediate layer 220, and the obtained weighted sum. Is converted by the activation function and output to the output layer 230. As the activation function in the intermediate layer 220, for example, a sigmoid function or a ReLU function is used. The learning model 20 may include a plurality of intermediate layers. That is, the learning model 20 may be configured by a deep neural network.

The output layer 230 is a layer including a plurality of nodes y1 and y2 that execute arithmetic processing (product-sum operation, non-linear operation by activation function, etc.) on the input data supplied from the intermediate layer 220 and output the data. More specifically, the output layer 230 is weighted by multiplying the input data by the weight values 240 (by11, by12, by21, by22) set for each edge, which is the path between the intermediate layer 220 and the output layer 230. The sum is converted by the activation function and output to the output layer 230. The output layer 230 may convert the output value into a probability value and output it. When node y1 is an output representing a person and node y2 is an output representing a tree, the probability that the main subject included in the input image data is a person is output from node y1, and the probability that the main subject is a tree. Is output from the node y2. As the activation function in the output layer 230, for example, a softmax function is used.

The above-mentioned weight values 240 (uv11, uv12, uv21, uv22, by11, by12, by21, by22) are adjusted by the exposure specifying unit 250 and the weighting coefficient determining unit 260.

The exposure specifying unit 250 detects the brightness L of the image data input to the input layer 210 of the learning model 20 and outputs it to the weighting coefficient determining unit 260. The "brightness" is generally a value that can be expressed based on various units such as a luminous flux (lumen), a luminous intensity (candela), and an illuminance (lux). The exposure specifying unit 250 of the present embodiment specifies one index value L indicating the "brightness" of the image data for each image data.

The weighting coefficient determining unit 260 determines the weighting coefficient to be multiplied by the weighting value 240 (that is, adjusts the weighting value 240) based on the brightness L of the image data detected by the exposure specifying unit 250. As shown in FIG. 2, the same weight coefficient α may be multiplied by all the weight values 240, and as shown in FIG. 3, a plurality of weight coefficients α, β, γ, ... Are selectively present. May be multiplied by the weight value 240 of. For example, as shown in FIG. 3A, the weighting factors α and β, which are different for each layer, may be multiplied by the weight value 240. As shown in FIG. 3B, different weighting factors α and β may be multiplied by a plurality of edges included in the layer. As shown in FIG. 3C, there may be layers (edges) that are not multiplied by the weighting factors.

Image recognition according to the first embodiment of the present invention will be described with reference to FIGS. 4 and 5. FIG. 4 shows image recognition according to the prior art, and FIG. 5 shows image recognition according to the configuration of the present embodiment. In each of the figures, the recognition processing is performed on the bright images 401 and 501 (images in which the brightness index value L is relatively large) and the dark images 402 and 502 (images in which the brightness index value L is relatively small). It is contrasted. As shown in FIGS. 4 and 5, the main subject in this example is a “person”.

The learning model 20 in this example outputs an inferred value for inferring what the main subject shown in the input image data is. A plurality of nodes (neurons) included in the output layer 230 correspond to the inferred object. The output layer 230 of the learning model 20 includes, for example, a node y1 that outputs the probability that the main subject is a person and a node y2 that outputs the probability that the main subject is a tree. The learning model 20 of this example is shown as a general three-layer NN for simplicity of explanation, but the learning model 20 is based on a convolutional neural network (CNN) suitable for image processing including image recognition. May be configured.

In the learning model 20'by the conventional technique as shown in FIG. 4, the subject can be recognized accurately when bright image data is input, but the subject cannot be recognized accurately when dark image data is input. There is.

For example, in FIG. 4A in which bright image data 401 is input to the learning model 20', the probability that the main subject is a person is 99% and the probability that the main subject is a tree is 1 in the learning model 20'. It is output as%. That is, it is accurately recognized that the main subject shown in the image data is a person. On the other hand, in FIG. 4B in which dark image data 402 is input to the learning model 20', the probability that the main subject is a person is 33% and the probability that the main subject is a tree is 67 in the learning model 20'. It is output as%. That is, it is not accurately recognized that the main subject shown in the image data is a person. The reason is as follows.

In dark image data, the pixel value in the image data tends to be small. Therefore, when dark image data is input to the learning model 20', digit loss occurs in the process of inference calculation in NN, so that the value is often lost (becomes 0) in the middle of the calculation. As a result, the accuracy of inference by the learning model 20'decreases. The loss of the above value (data) occurs more remarkably when the calculation bit width of the learning model 20'is less than or equal to the bit width of the input image data.

Therefore, in the present embodiment, as described above, the weighting coefficient determining unit 260 determines the weighting coefficient α based on the brightness L of the input image data specified by the exposure specifying unit 250, and the weighting value included in the learning model 20. Multiply 240 by the weighting factor α. In determining the weighting coefficient α, the maximum value of the operation bit width of the NN of the learning model 20 is taken into consideration, as will be described later. By adjusting the weighting coefficient α as described above, digit loss (data loss) of digital data (calculated value) in inference processing for input image data is suppressed, and thus the recognition accuracy of the subject is improved.

For example, in FIG. 5A in which bright image data 501 is input to the learning model 20, the weighting coefficient α is set to a value that does not affect inference (for example, 1), and the same result as in FIG. 4A is output. Will be done. Further, in FIG. 5B in which dark image data 502 is input to the learning model 20, the probability that the main subject is a person is 83% and the probability that the main subject is a tree is 17% in the learning model 20. It is output that there is. That is, as compared with the conventional technique shown in FIG. 4, according to the technique of the present embodiment shown in FIG. 5, the recognition accuracy when dark image data is input to the learning model 20 can be remarkably improved.

The determination of the weighting coefficient α in the first embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart of a process in which the exposure specifying unit 250 specifies the brightness L and a process in which the weighting coefficient determining unit 260 determines the weighting coefficient α. Roughly speaking, the weighting coefficient α is determined based on the brightness L of the input image data and the calculation bit width of the learning model 20. This flow is executed, for example, after the image data as the inference target is input and prior to the image recognition by the learning model 20.

In step S601, the exposure specifying unit 250 specifies the brightness L of the image data input to the learning model 20. The brightness L of the above image data may be acquired by analyzing the input image data, or may be acquired based on information from a sensor (exposure meter, illuminance sensor, etc.) at the time of imaging. Further, the brightness L may be specified by the combination of both of the above.

The brightness L in this example is, for example, an exposure value (EV value). The above exposure values are relative values based on the state where the ISO sensitivity is ISO 100, the aperture is F1.0, and the shutter speed is 1 second (EV0). When the exposure value is increased by one step (for example, when it is increased from EV1 to EV2), it becomes twice as bright. The exposure value and the illuminance are related to the following equation (1). For example, when the exposure value is 0 (EV0), the illuminance is 2.5 lux.
(Illuminance (lux)) = 2.5 x 2 ^{(exposure value)} …… (1)

In step S602, the weighting coefficient determining unit 260 calculates the coefficient C based on the brightness L specified by the exposure specifying unit 250. More specifically, it is as follows.

First, the weighting coefficient determining unit 260 determines the brightness ratio A (= L / R), which is the ratio between the brightness L of the input image data and the reference value R calculated in advance in the learning stage of the learning model 20. calculate. The reference value R is a statistical representative value (average value, median value, mode value, etc.) of brightness L in a plurality of image data input to the learning model 20 in the learning stage. Since the brightness L is a value acquired for each image data, the brightness ratio A also changes when the image data changes.

Next, the weighting coefficient determining unit 260 calculates the coefficient C according to the following equation (2) using the calculated brightness ratio A.
C = (1 / square (A) x (b / (1 / square (Aev)))-(b / (1 / square (Aev))-d)) x c-e ... (2)

In the above equation (2), the value Av is the brightness (limit brightness) corresponding to the imaging limit of the image sensor that has acquired the image data, and is a value that is the minimum unit of the brightness in the image data. Since the value Aev is a value determined according to the specifications of the image pickup device, it is a predetermined value common to the image data acquired by the same image pickup device. The value b is the maximum value (for example, 256 = 8 bits) of the calculation bit width used in the NN of the learning model 20. The correction coefficients c, d, and e are parameters for adjusting the degree of change (slope of the graph) of the coefficient C with respect to the brightness ratio A, and the manufacturer or user of the information processing apparatus 100 responds to the inference result. It is an adjustable value.

FIG. 7 is an explanatory diagram showing the relationship between the brightness ratio A and the coefficient C (weighting coefficient α) in the first embodiment of the present invention. In the example of FIG. 7, a said value Aev (limit brightness) is 0.0078125 (= ^1/27), the value b (the maximum value of the operation bit width of the NN) 256 (= 8 bits) Is assumed to be. When the brightness ratio A is equal to the value Aev corresponding to the limit luminance, when the calculation is performed according to the above equation (2), the coefficient C (weight coefficient α) is a value equal to the maximum value b of the calculation bit width of NN. 256 is obtained. When the brightness ratio A changes, the coefficient C (weighting coefficient α) changes as shown in the table of FIG. 7 (a) and the graph of FIG. 7 (b). Generally, the smaller the brightness ratio A, the larger the coefficient C (weighting coefficient α).

In the equation (2) described above, the calculation bit width that can be used in the learning model 20 is wider based on the value Aev (limit brightness) corresponding to the detection limit (imaging limit) of the image sensor, and the NN calculation is performed. It is an expression which calculates a coefficient C as used in. Therefore, any mathematical formula capable of calculating the coefficient C (weighting coefficient α) as described above can be used in step S602. For example, in the above equation (2), the coefficient C takes the maximum value when the brightness ratio A is equal to the value Aev, but the correction coefficients c, d, and e are adjusted so that the brightness ratio A sets another value. An equation in which the coefficient C takes the maximum value at the time of taking may be adopted. Further, as described above with reference to FIG. 3, the coefficients C (weighting coefficients α, β, γ, ...) Are calculated by applying different correction coefficients c, d, and e for each layer or edge of the NN. You may.

In step S603, the weighting coefficient determining unit 260 sets the coefficient C calculated in step S602 in the learning model 20 as the weighting coefficient α of NN.

According to the configuration of the present embodiment described above, since the weighting coefficient determined according to the brightness of the image data input to the learning model 20 in the inference stage is multiplied by the weight of the learning model 20, various brightnesses are obtained. Highly accurate inference can be performed on the image data of.

In particular, since the weighting coefficient is determined based on the maximum value of the calculation bit width of the learning model 20, even if dark image data, that is, image data having a small pixel value is input to the learning model 20, the digit at the time of calculation Since the drop is suppressed, the accuracy of inference can be maintained.

<Second Embodiment>
Hereinafter, the second embodiment of the present invention will be described. In each of the embodiments illustrated below, for elements whose actions and functions are equivalent to those of the premise example or the first embodiment, the reference numerals referred to in the above description will be used and the respective description will be omitted as appropriate.

In the first embodiment, one brightness L is specified for each of the image data and used for adjusting the weighting coefficient of the learning model 20. In the second embodiment, the brightness L is specified for each of the plurality of regions included in the image data, and is used for adjusting the weighting coefficient of the learning model 80.

The learning model 80, the functional blocks, and various processes according to the present embodiment described below are realized by the CPU 101 of the information processing apparatus 100 expanding the program stored in the ROM 102 or the HDD 104 into the RAM 103 and executing the program. Will be done. Further, various processes according to the present embodiment are started by, for example, a user operating the input unit 105 to instruct the CPU 101, and the results of the various processes are output to the display unit 106.

It is explanatory drawing which illustrates the inference processing (object recognition processing) in the 2nd Embodiment of this invention with reference to FIG. Similar to the learning model 20 of the first embodiment, the learning model 80 of the present embodiment outputs an inferred value for inferring what the main subject is shown in the input image data. The image data 801 of FIG. 8 includes a dark portion (dark portion) created by the shadow of a person who is the main subject. The image data 801 is divided into a plurality of regions and is input to a plurality of nodes of the input layer of the learning model 80 and the exposure specifying unit 850, respectively.

The exposure specifying unit 850 detects the brightness L of the image data input to the learning model 80 for each region and outputs it to the weighting coefficient determining unit 860. The weighting coefficient determining unit 860 determines the weighting coefficients α, β, γ, ... To be multiplied by the weight value corresponding to each region based on the brightness L of each region of the image data detected by the exposure specifying unit 850. (That is, adjust the weight value).

For example, as shown in FIG. 3C, the weight coefficients α to be multiplied by the weight values uv11 and uv12 for the data output by the node u1 and the weights uv21 and uv22 to be multiplied by the weight values uv21 and uv22 for the data output by the node u2. The coefficient β is set separately. That is, the weighting coefficient α is determined according to the brightness L1 of the region of the image data 801 input to the node u1, and the weighting coefficient β is determined according to the brightness L2 of the region of the image data 801 input to the node u2. Will be done.

According to the configuration of the present embodiment described above, the same technical effect as that of the first embodiment is achieved. In addition, since the brightness L is specified for each image data area and the weighting coefficients α, β, γ, ... Are determined, highly accurate inference can be made even for image data having both a bright area and a dark area. Can be executed.

<Modification example>
Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications and modifications can be made within the scope of the gist thereof.

In the above embodiment, the learning models 20 and 80 recognize the main subject of the image data, but the learning models 20 and 80 may be used for other inferences. For example, the learning models 20 and 80 may estimate the scene (landscape, portrait, etc.) corresponding to the image data.

In the above-described embodiment, the exposure specifying units 250 and 850 and the weighting coefficient determining units 260 and 860 are configured as software-like functional blocks, but they are hardened by an electrical configuration (circuit or the like) capable of executing the above-described processing. It may be configured as a piece of clothing.

The image data to be inferred may be image data that has already been captured, or may be real-eye image data output by the image sensor of the image pickup device.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors of the computer of the system or device reads the program. It can also be realized by the processing to be executed. The present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The above computers may have one or more processors or circuits. Multiple separate computers, or a network of separate processors or circuits, may read and execute instructions that can be executed by a computer.

Instead of the CPU 101 in the above embodiment, the information processing apparatus 100 may adopt any processor or circuit. For example, the following elements may be used as a processor or circuit.

-Micro processing unit (MPU)
-Graphics processing unit (GPU)
-Application-specific integrated circuit (ASIC)
-Field Programmable Gateway (FPGA)
-Digital Signal Processor (DSP)
-Data Flow Processor (DFP)
-Neural processing unit (NPU)

20 Learning model 80 Learning model 100 Information processing device 250 Exposure identification part (exposure identification means)
260 Weight coefficient determination unit (weight coefficient determination means)
850 Exposure identification part (exposure identification means)
860 Weight coefficient determination unit (weight coefficient determination means)

Claims (10)

A learning model, which is a neural network that executes inference processing by operations using weight values for image data,
An exposure specifying means for specifying the brightness of the image data and
An information processing apparatus comprising: a weighting coefficient determining means for determining a weighting coefficient to be multiplied by the weighting value based on the specified brightness. The information processing apparatus according to claim 1, wherein the calculation bit width in the calculation executed by the learning model is equal to or less than the bit width of the image data. The information processing apparatus according to claim 1 or 2, wherein the weighting coefficient determining means increases the weighting coefficient as the brightness of the image data decreases. The learning model has a plurality of layers including a plurality of nodes, respectively.
The information processing according to any one of claims 1 to 3, wherein the weighting coefficient determining means changes the weight value corresponding to at least one of the layers based on the brightness. apparatus. The weighting coefficient determining means determines the weighting coefficient based on the ratio of the brightness to a representative value of brightness in a plurality of image data input to the learning model in the learning stage. The information processing apparatus according to any one of claims 1 to 4. Any of claims 1 to 5, wherein the weighting coefficient determining means determines the weighting coefficient based on an imaging limit in an image pickup device that has acquired the image data that is the target of the inference processing. The information processing apparatus according to item 1. The exposure specifying means identifies the brightness for each of a plurality of regions included in the image data.
Any one of claims 1 to 6, wherein the weighting coefficient determining means determines, respectively, a plurality of the weighting coefficients based on the plurality of brightnesses specified for each of the plurality of the regions. The information processing device described in the section. The information processing apparatus according to any one of claims 1 to 7.
An image pickup apparatus comprising: an image pickup means including an image pickup element for acquiring the image data. A control method for an information processing device equipped with a learning model, which is a neural network that executes inference processing by calculation using weight values for image data.
Identifying the brightness of the image data and
A control method comprising: determining a weighting coefficient to be multiplied by the weight value based on the specified brightness. A program that causes a computer to function as a learning model and means for the information processing apparatus according to any one of claims 1 to 7.

Images