JP7398938B2 - Information processing device and its learning method - Google Patents

Description

The present invention relates to pattern recognition processing using a neural network.

Hierarchical calculation methods represented by convolutional neural networks (hereinafter abbreviated as CNN) are attracting attention as a method that enables pattern recognition that is robust against changes in recognition targets. For example, Non-Patent Document 1 discloses various application examples and implementation examples of a pattern recognition method based on deep learning technology.

In addition, as a method to cope with large fluctuations in the photographing environment (lighting, subject condition, etc.) of the recognition target, Patent Document 1 proposes a method that changes the photographing conditions of the photographing device at predetermined intervals to increase the face detection probability in the image. A method for improving this is disclosed. Further, Patent Document 2 discloses a method of controlling the gain and exposure time of an imaging device based on the result of face detection and re-acquiring image data under conditions suitable for attribute recognition processing of a detected person.

Japanese Patent Application Publication No. 2014-127999 JP2017-098746A

Yann LeCun, Koray Kavukcuoglu and Clement Farabet, "Convolutional Networks and Applications in Vision", Proc. International Symposium on Circuits and Systems (ISCAS'10), IEEE, 2010

However, since the method described in Patent Document 1 is configured to change the imaging conditions at predetermined intervals, it is not always possible to obtain appropriate images. Furthermore, the method described in Patent Document 2 has a problem in that it is difficult to determine in advance an optimal photographing condition change table for various changes in the photographing environment. Furthermore, it is not possible to deal with cases where the optimal photographing conditions differ for each region of the same frame image.

The present invention has been made in view of such problems, and an object of the present invention is to provide a technique that enables more robust pattern recognition against various fluctuations in data to be processed.

In order to solve the above-mentioned problems, an information processing device according to the present invention has the following configuration. That is, an information processing device that can be connected to a sensing device,
Setting means for setting data acquisition conditions in the sensing device;
a first processing means for performing hierarchical feature extraction processing on data obtained by the sensing device using a first neural network (NN);
a second processing means for generating regression data indicative of data acquisition conditions to be used in subsequent data acquisition by the sensing device using the feature map in the intermediate layer of the first NN;
has
The setting means sets data acquisition conditions indicated in the regression data to the sensing device.

According to the present invention, it is possible to provide a technique that enables more robust pattern recognition against various fluctuations in data to be processed.

It is an operation flowchart of learning processing in a 1st embodiment. 1 is a diagram showing a schematic configuration of an image processing system and a learning device. It is a figure explaining the processing in a recognition processing part. FIG. 3 is a diagram showing a detailed configuration of a hierarchical feature extraction processing section. It is a figure explaining the operation timing in a pattern recognition device. FIG. 2 is a diagram illustrating the configuration of a stacked device. FIG. 2 is a diagram showing a detailed configuration of a pattern recognition device. It is a figure explaining a regression map. FIG. 3 is a diagram illustrating the operation of learning processing of each network. It is a figure showing a specific example of learning processing in a 1st embodiment. FIG. 3 is a diagram showing a relationship between a sensor gain control value and an output signal. It is a figure showing a specific example of learning processing in a 2nd embodiment. It is an operation flowchart of learning processing in a 3rd embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

(First embodiment)
As a first embodiment of the information processing device according to the present invention, an image processing system using a pattern recognition device will be described below as an example.

<System and device configuration>
FIG. 2 is a diagram showing a schematic configuration of an image processing system and a learning device. FIG. 2A shows a configuration example of an image processing system using the pattern recognition device 201. The system has the ability to detect a specific object region from image data. On the other hand, FIG. 2(b) shows an example of the configuration of the learning device. The learning results (weighting coefficients) obtained by the learning device will be used by the pattern recognition device 201. Note that although the image processing system and the learning device are described here as separate devices, they may be configured as an integrated device.

The image processing system includes a pattern recognition device 201, a CPU (Central Processing Unit) 205, a ROM (Read Only Memory) 206, a RAM (Random Access Memory) 207, and a DMAC (Direct Memory Access Controller). Furthermore, the pattern recognition device 201 includes an imaging device 202, a recognition processing section 203, and a RAM 204.

The imaging device 202 includes an optical system, a photoelectric conversion device, a driver circuit/AD converter, and the like. As the photoelectric conversion device, a CCD (Charge-Coupled Devices) or a CMOS (Complimentary Metal Oxide Semiconductor) sensor may be used. The recognition processing unit 203 controls the imaging device 202 to perform predetermined recognition processing on the acquired image data. The RAM 204 is used as a calculation work buffer for the recognition processing unit 203. Here, in order to reduce data transmission delay, the pattern recognition apparatus 201 is described as having a configuration including an imaging device 202 and a recognition processing unit 203, but it may also be configured as a separate unit if it can be connected to the imaging device. good.

The CPU 205 controls the entire image processing system. The ROM 206 stores instructions and parameter data that define the operation of the CPU 205. The RAM 207 is a memory necessary for the operation of the CPU 205. The DMAC 208 controls data transfer between the pattern recognition processing device 201 and the RAM 207. Data bus 209 is a data transfer path between devices.

The pattern recognition processing device 201 executes imaging and recognition processing according to instructions from the CPU 205, and stores the results in the RAM 207. The CPU 205 provides various applications using the recognition results.

The learning device includes a calculation device 210, an interface device 212, and a storage device 213, and can be realized by, for example, a general-purpose computer device. The arithmetic unit 210 includes computer devices such as a CPU and a memory, and executes a learning process that will be described later with reference to FIG. The storage device 213 is a large-capacity data storage device such as a hard disk drive, and stores programs executed by the arithmetic device 210, image data used for learning, teacher data, and the like. The interface device 212 is an interface for retrieving data obtained through learning, and is a communication interface or a portable storage device interface. The learning results by the learning device are taken out via the interface device 212 and stored in the ROM 206 or the like of the image processing system.

FIG. 7 is a diagram showing the detailed configuration of the pattern recognition device 201. As shown in FIG. 2 is a diagram illustrating the configuration of a recognition processing unit 203 in more detail. FIG.

The feature extraction processing unit 701 repeatedly executes hierarchical feature extraction processing while retaining intermediate results of the hierarchical calculation in the memory 703, and outputs recognition processing results and control data using the extracted feature amounts.

The imaging device 704 corresponds to the imaging device 202 and includes an optical system, a photoelectric conversion device, a driver circuit/AD converter, and the like. The imaging control processing unit 705 controls the operation (imaging conditions, etc.) of the imaging device 704 according to the control data provided from the feature extraction processing unit 701. Specifically, the photographing conditions include the gain for the signal after photoelectric conversion, the accumulation time (exposure time) of the photoelectric conversion device (photodiode, etc.), the characteristics of A/D conversion, and the like. The imaging control processing unit 705 is configured to be able to control these imaging conditions on a block-by-block basis on the sensor surface. For example, with the recent development of semiconductor stacking technology, it has become possible to stack control logic on the sensor surface, which makes it possible to implement readout control on a block-by-block or pixel-by-pixel basis.

FIG. 6 is a diagram illustrating the configuration of a stacked device. FIG. 6(a) schematically shows the physical configuration of a stacked device. Here, an example is shown in which a sensor layer 61 on which a photoelectric conversion element is mounted, a logic layer 62 on which a read control logic is mounted, and a memory layer 63 on which a large-scale memory and its control section are mounted are stacked. The sensor layer 61 corresponds to the imaging device 704, the logic layer 62 corresponds to the imaging control processing section 705, and the memory layer 63 corresponds to the memory 703 and the like. Signals are transmitted between each layer using through vias or the like.

FIG. 6(b) schematically shows the configuration of the logic layer 62. Here, in the logic layer 62, n×n control circuits for controlling the photoelectric conversion elements of the sensor layer 61 are arranged. The control circuit ct(n, n) controls reading of one or more photoelectric conversion elements of the sensor layer 61 located at the corresponding position. Therefore, with the above configuration, readout conditions (gain, exposure time, etc.) can be controlled for each block of n×n blocks. In other words, the imaging characteristics can be controlled for each n×n portion in the image.

Note that in the first embodiment, it is assumed that the feature extraction processing unit 701 is also implemented in the logic layer 62 and the memory layer 63. For example, by stacking the sensor layer 61, control data can be fed back with less delay. When the photographing environment or object changes rapidly, it is desirable to control the imaging device 704 with less image frame delay, so stacking the sensor layer 61 is preferable.

FIG. 3 is a diagram illustrating processing in the recognition processing unit 203. The recognition processing unit 203 includes a recognition network 302 and a sensor control network 313, which are the logical processing structure of the feature extraction processing unit 701. The recognition network 302 is an arithmetic network in which the imaging device 704 recognizes the position of a predetermined object within the imaging target 301 using CNN. The sensor control network 313 is a calculation network that uses CNN to extract information for controlling the imaging conditions of the imaging device.

Here, an example is shown in which the recognition network 302 is configured by a five-layer CNN. The calculation processes 303 to 307 are calculation processes including convolution calculations, activation function calculations, pooling calculations, etc., and are specifically realized by the configuration shown in FIG. 4, which will be described later.

Feature maps 308 to 312 are called intermediate layers (feature maps 308 to 311) or final layers (feature map 312) of CNN calculation processing, and correspond to the results of calculation processing 303 to 307, respectively. Feature maps 308-312 are stored in memory 703. The feature maps 308 to 312 are two-dimensional data obtained by performing feature extraction processing on image data output by an imaging device.

Here, details of the two-dimensional CNN calculation processing on image data will be explained. When the kernel (filter coefficient matrix) size of the convolution operation is columnSize×rowSize and the number of feature maps in the previous layer is L, one feature map is calculated by the product-sum operation shown in Equation (1) below.

input (x, y): reference pixel value at two-dimensional coordinates (x, y) output (x, y): calculation result at two-dimensional coordinates (x, y) weight (column, row): coordinate (x+column, y+row) L: Number of feature maps in the previous layer ColumnSize, rowSize: Horizontal and vertical size of the two-dimensional convolution kernel In CNN calculation processing, multiple convolution kernels are calculated in pixel units according to formula (1). A feature map is calculated by repeating product-sum calculations while scanning and non-linearly transforming (activation processing) the final product-sum calculation results. Furthermore, the generated feature map may be reduced by pooling processing and referred to in the next layer. The feature maps 308 to 312 have a plurality of maps in one hierarchy, and feature maps with different characteristics are generated in response to different weighting coefficient groups.

FIG. 4 is a diagram showing the detailed configuration of the feature extraction processing unit 701. The feature extraction processing unit 701 is a concrete implementation configuration of the calculation processes 303 to 307.

The reference data buffer 401 is a circuit that acquires from memory all or part of the data of the feature map of the previous layer (input (x, y) in formula (1)), which serves as reference data for the convolution operation, and buffers it. .

The multiplier 402 and the cumulative adder 403 are circuits that execute the calculation of formula (1).

The coefficient data buffer 404 is a circuit that transfers and buffers all or part of the weighting coefficient data (weight (column, row) in formula (1)) obtained through learning in advance from the memory 703 in a predetermined unit. .

The activation processing circuit 405 is a circuit that processes a nonlinear function such as ReLU (Rectified Linear Unit, Rectifier) on the convolution operation result (output (x, y)) shown in equation (1).

The pooling processing circuit 406 is a circuit that reduces the feature map using a spatial filter such as a maximum value filter. If pooling processing is not performed, the result of activation processing 405 is stored in memory 703, and if pooling processing is performed, the result of pooling processing 406 is stored in memory 703. The data stored here becomes the feature map of the current layer.

When the calculation of the feature map of the current layer is completed, calculation of the feature map of the next layer is similarly processed using the calculated feature map as the feature map of the previous layer. In this way, feature maps of a plurality of layers are calculated while sequentially referring to the feature maps stored in the memory 703. The control unit 102 (not shown in FIG. 4) controls the operation of each component in FIG. 4, thereby realizing hierarchical feature extraction processing (CNN calculation processing).

CNN achieves recognition processing that is robust to changes in the identification target by repeating feature extraction across multiple layers in this way. The CNN determines the existence of a desired pattern in calculation 307 of the final layer according to the feature extraction results of each layer. The feature map 312 in the final layer expresses the recognition result 320, and is, for example, a confidence map that expresses the existence probability of an object in an image as two-dimensional information. Note that the calculation 307 in the final layer may be configured by a fully connected neural network or a linear discriminator instead of the above-mentioned convolution calculation.

In addition, the feature maps 308 to 311 of each layer express the feature extraction results for input data, and generally, lower layers (layers close to the layer that inputs the data to be processed) indicate low-level features such as edges, The upper layer (layer close to the recognition result) shows features with a high degree of abstraction. Each feature map has different characteristics depending on the target of pattern recognition and the learning method.

Next, the sensor control network 313 will be explained. The sensor control network 313 is a calculation network that returns data for sensor control. In other words, the sensor data acquisition conditions are determined using CNN. Arithmetic processes 314 and 315 are similar to arithmetic processes 303 to 307, and are processed by the circuit shown in FIG.

The sensor control network 313 regresses the control signal using the feature map 308 in the lower layer of the recognition network 302. In other words, the feature values obtained during the calculation process of the recognition network are used. By sharing the feature map with the recognition network, it is expected to improve regression performance and make learning easier, as well as reduce the overall calculation cost. Further, here, since the sensor control network 313 is configured by network calculation processing (CNN) similar to the recognition network, control data can be generated using the feature extraction processing unit 701. That is, the advantage is that a dedicated circuit or the like is not required.

Feature maps 316 to 317 are feature maps in the sensor control network 313, and a feature map 317 (hereinafter referred to as regression map 317) that regresses the control data of the imaging conditions is generated in the final layer calculation 315. The regression map 317 is control data that specifies imaging conditions corresponding to the spatial position of the image sensor, and is data that corresponds to, for example, designation of the gain and exposure time of the image sensor corresponding to the position of the map. Only one regression map 317 is required when there is only one type of control target and the control is performed using scalar values. If there are multiple control conditions or if the control parameters are vector data, multiple regression maps will exist.

FIG. 8 is a diagram illustrating a regression map. Specifically, an example of a regression map 317 corresponding to the control logic shown in FIG. 6(b) is schematically shown. rg(n, n) corresponds to pixel data of the regression map. That is, the size of the regression map here is n×n. The value of the regression data is expressed in shading, and corresponds to, for example, the gain of data after photoelectric conversion.

The imaging control processing unit 705 controls the photoelectric conversion element according to the control signal data returned by the sensor control network 313, and acquires image data suitable for recognition processing. The image data obtained here is different from image data for humans to observe, understand and appreciate the content, and is image data suitable for improving the accuracy of recognition processing.

Note that in the sensor control network 313, the calculation process 314 includes a pooling process to reduce the size of the feature map. Therefore, the size of the regression map is smaller than the image size of the sensor output. That is, the readout conditions are controlled for each block, which has a plurality of pixels as a unit. The pooling ratio and the like are determined in consideration of the block size that can be controlled by the imaging control processing unit 705.

FIG. 5 is a timing chart illustrating the operation timing of pattern recognition processing in the pattern recognition device 201. The horizontal axis represents the passage of time and exemplarily shows the timing at which each process of the recognition network (recognition network 302), control network (sensor control network 313), and condition setting is executed. Here, a state in which recognition processing is continuously performed on three temporally consecutive frames (first to third frames) of image data is shown.

At timing 501, imaging of the first frame and recognition network processing for the first frame are executed, and in parallel, at timing 504, control network processing for the subsequent second frame is executed. At timing 507, processing for setting the operating conditions (gain, exposure time, etc.) of the imaging device for the second frame is executed.

At timing 502, imaging of the second frame and recognition network processing for the second frame are executed, and in parallel, at timing 505, control network processing for the subsequent third frame is executed. At timing 508, processing for setting the operating conditions (gain, exposure time, etc.) of the imaging device for the third frame is executed.

At timing 503, imaging of the third frame and recognition network processing for the third frame are executed, and in parallel, at timing 506, control network processing for the subsequent fourth frame (not shown) is executed. At timing 509, processing for setting operating conditions (gain, exposure time, etc.) of the imaging device for the fourth frame (not shown) is executed.

In this way, the control network sequentially sets imaging conditions suitable for recognition in response to changes in the situation of the object to be photographed, and the recognition network performs imaging in accordance with the imaging conditions and executes pattern recognition processing.

<Operation of learning device>
Next, learning processing of the recognition network and control network in the learning device will be explained. As described above, the learning results (weighting coefficients) by the learning device are used by the pattern recognition device 201.

FIG. 9 is a diagram illustrating the operation of the learning process of the recognition network and the control network. Learning here means a process of determining the weighting coefficients of the respective neural networks of the recognition network 302 and the sensor control network 313 so that the pattern recognition process has more suitable (or best) performance. Note that in FIG. 9, the calculation processes 303 to 307 and 314 to 315 in FIG. 3 are omitted.

A sensor model 901 is a sensor model required to generate data used for learning each network. For example, the sensor model 901 generates pseudo sensor data 902 (pseudo data). In other words, the pseudo sensor data 902 is image data that simulates the output of a sensor when it is assumed that the image data 301 is the object to be photographed.

The sensor model 901 can be created theoretically according to the physical characteristics of the sensor. However, it may be created using a learning method such as GAN (Generative Adversarial Network) using data obtained by actual sensors. Further, the sensor model 901 has a function of simulating output fluctuations in response to a control signal that controls sensor readout conditions. For example, if a high gain value is set as the control signal, the output will also be a high value.

That is, the sensor model 901 is a model that defines both the relationship between the image data 301 captured by another imaging device and the pseudo sensor data 902, and the relationship between the control signal and the pseudo sensor data. It is assumed that the sensor model is created in advance before learning the recognition network and sensor control network.

The memory 903 is a memory that holds intermediate calculation results of the recognition network necessary for learning the sensor control network 313. The teacher data 905 is teacher data created in a pair with the image data 301, and is prepared in advance. The teacher data here is the data distribution of the recognition result 320 expected as the recognition result. For example, when detecting a face in an image, it is assumed that map data is in a normal division format with a peak at the center of the face. Processing 904 is processing for calculating the difference between recognition result 320 and teacher data 905.

FIG. 9(a) is a diagram illustrating main processing when learning a recognition network. Image data 301 is converted using a sensor model 901 to obtain pseudo sensor data 902. A pattern recognition process is executed using the generated pseudo sensor data 902 and a recognition result 320 is obtained. A recognition network is trained by a backpropagation method using the difference between the recognition result 320 and the teacher data 905 as error data. That is, the weighting coefficients of the convolution operations for generating the feature maps 308 to 312 are sequentially updated.

FIG. 9(b) is a diagram illustrating the main processing when learning the sensor control network 313. The sensor control network 313 realizes sensor control signal generation processing. During learning of the sensor control network 313, error information, which is difference information between the recognition result 320 and the teacher data 905, is back-propagated. At this time, no learning is performed in the recognition network 302 (that is, the coefficients are fixed). Furthermore, error information for learning the sensor control network 313 is obtained via the inverse function of the sensor model. The sensor model 901 holds data generated as pseudo sensor data and table information for realizing an inverse function in order to calculate a correct value for sensor control.

The sensor control network is learned by the backpropagation method using the obtained error information and the data of the feature map 308 (stored in the memory 903) obtained through the recognition process. That is, the coefficients of the convolution operation that generates the feature maps 316 to 317 are sequentially updated. The backpropagation method is processed using a conventionally proposed method. This process is characterized in that the recognition network is fixed and the error in the recognition result is given to the sensor control network 313 via an inverse function of the sensor model. The inverse function of the sensor model is determined in advance according to the characteristics of the sensor.

FIG. 10 is a diagram showing a specific example of the operation in the learning process in the first embodiment. FIGS. 10A and 10B show operation patterns of pattern recognition processing and learning processing when learning a recognition network. FIGS. 10C and 10D show operation patterns of pattern recognition processing and learning processing when learning the sensor control network. FIG. 10 shows an example in which the number of nodes in the neural network of the recognition network 302 is two (nodes 1003 and 1004), and the number of nodes in the neural network of the sensor control network 313 is one (node 1005).

Furthermore, in order to simplify the explanation, an example of a neural network having an MLP (Multi Layer Perceptron) configuration will be used instead of a CNN. When viewed as a CNN, this corresponds to a case where the kernel size of the convolution operation is 1x1. Further, here, the sensor model is shown divided into a sensor imaging model 1001 and a sensor control model 1002. In FIG. 10, the recognition result 320 is the recognition result for the image data 301 of the learning data set, and the teacher data 905 and calculation processing 904 are the same as those in FIG.

FIG. 1 is an operational flowchart of learning processing in the first embodiment. In S101, the learning device executes initialization processing. Specifically, various initialization processes such as initialization of sensor models (sensor imaging model 1001 and sensor control model 1002) are executed.

In S102, the learning device selects learning data to be used for learning processing. For example, the image data 301 and the teacher data 905 for learning are selected from the learning data set stored in the storage device 213 and read into a memory (not shown) of the arithmetic device 210 .

In S103, the learning device converts the image data according to the control conditions of the sensor model. Here, the image data 301 is converted according to conditions set in the sensor control model 1002 (for example, sensor control conditions such as sensitivity, gain, exposure time, etc.) to generate pseudo sensor data that is pseudo sensor data.

In S104, the learning device performs a predetermined pattern recognition process on the pseudo sensor data generated in S103 (operation pattern 1006). The pattern recognition processing here is, for example, pattern recognition processing such as detecting a face in an image. Recognition processing is performed on the pseudo sensor data 902 converted by the sensor imaging model 1001 at the calculation nodes n ₁ and n ₂ of the neural network, and a recognition result 320 is obtained. By performing recognition processing on the two-dimensional data of the image data 301 in raster order, the recognition result 320 also becomes a two-dimensional map.

In S105, the learning device performs learning of pattern recognition processing based on the recognition result 320 obtained in S104 (operation pattern 1007). Here, the weighting coefficients of the recognition network are learned by the backpropagation method using the training data selected in S102. The coefficient W _n2 for the node n ₂ (node 1004) and the coefficient W _n1 for the node n ₁ (node 1003) are sequentially updated using the difference value between the recognition result 320 and the teacher data 905 as an error. Note that the coefficient _Wr for node r (node 1005) is not updated during the calculation in S105.

A specific example of learning using the backpropagation method will be described below. In the backpropagation method, the coefficients W ₁ and W ₂ are adjusted so that the error between the recognition result 320 and the teacher data 905 in their respective image positions is minimized.

Let y be the output value corresponding to a certain pixel position included in the recognition result 320, let _yt be the teacher data value corresponding to that position, and define the error E between the teacher data and the output value as shown in Equation (2) below. . Note that the notation of coordinate data is omitted here for the sake of simplicity.

When the output of the node n ₁ is n ₁ and α is the learning coefficient, the coefficient W ₂ is updated to W' ₂ using the following equation (3).

If the nonlinear function of nodes n ₁ and n ₂ is a ReLU function f _ReLU (), then y=f _ReLU (W ₂ ×n ₁ ). In addition, when W ₂ ×n ₁ >0, the differential of the fReLU function is 1, so the formula (3) becomes the following formula (4).

Next, when the output of the sensor model at the corresponding pixel position is s and the learning coefficient is α, W ₁ is updated to W′ ₁ using the following equation (5).

Here, n ₁ =f _ReLU (W ₁ ×s). Therefore, when W ₁ ×s>0, Equation (5) becomes Equation (6) below, and W′ ₁ after updating can be calculated.

In S106, the learning device performs the pattern recognition process again on the recognition network of the weighting coefficients learned (updated) in S105, and outputs the recognition result 320. At the same time, the calculation result n ₁ of the node n ₁ (node 1003) is stored in the memory 903 (operation pattern 1008).

In S107, the learning device executes learning of the sensor control network 313 based on the result of the pattern recognition process in S106. Here, the sensor control network 313 is learned using the teacher data 905 selected in S102 and the error E' of the recognition result 320. The error E' here is an error with respect to the pattern recognition processing result after the recognition network update calculated in S106.

First, the coefficients of the recognition network 302 are fixed and the error E' is back-propagated. The correct sensor control value rt is estimated from the error Es (=E'×W' ₂ ×W' ₁ ) calculated by back-propagating the recognition network 302 and the sensor control value r stored in the sensor model. Here, estimation is performed according to the inverse function f _rev (back propagation error, control value) of the sensor model 904 (Equation (7)).

The f _rev function is an inverse function of the sensor model (sensor imaging model 1001 and sensor control model 1002). The correct value r _t of the control parameter is calculated backward from the error value Es of the sensor data and the current control value r.

FIG. 11 is a diagram showing the relationship between the sensor gain control value and the output signal. More specifically, it is a diagram schematically showing an inverse function of a sensor model using gain control as an example. A straight line 1101 indicates a function for realizing an inverse function, and is a function expressing the relationship between the sensor gain control value and the output signal. Although shown as a linear function, it is actually an arbitrary function determined based on logical analysis or experimentation, and is retained as an approximate function or table information. The straight line 1101 corresponds to a combination of the inverse function of the sensor imaging model 1001 and the inverse function of the sensor control model 1002.

Point 1102 is a point indicating the relationship between the control value r of gain control and the output signal at that time. A point 1103 indicates a point at which a correct value for gain control is determined according to the sensor output error Es. The correct value r _t (point 1103) for gain control is calculated using the output signal r (point 1102) during pseudo sensor data generation stored in the memory in the model and the sensor output error signal Es back-propagated from the recognition network. .

Using the obtained gain control correct value r _t , the weighting coefficient W _r of the node r (node 1005) is updated by the back propagation method (operation pattern 1009). More specifically, it is updated based on the output data of node n ₁ (node 1003) stored in memory 903.

The weighting coefficient W _r of the sensor control network 313 is updated using the following formula (8), where β is the learning coefficient.

If E _r is the error data for learning the sensor control network 313, r is the output value of the node r of the sensor control network 313, and r _t is the correct value of the sensor control value, Equation (9) is satisfied.

Therefore, Equation (8) can be transformed into Equation (10) below.

As a result, the coefficients of the sensor control network can be updated using the following equation (11) from equations (7) and (10).

The above processing is executed for all or a plurality of selected positions included in the image data 301. That is, the coefficient W' _r is learned so that a map 317 corresponding to regression information for appropriately controlling the sensor is generated.

In S108, the learning device regresses the sensor control parameters using the sensor control network with the updated weighting coefficient _W'r .

In S109, the learning device sets the regressed parameters as control parameters of the sensor model. That is, in S103 for the next image data (next loop), the image data 301 is converted into pseudo sensor data 902 using the control parameters set here. The sensor control unit is a partial region unit determined according to the size of the regression data map 318 corresponding to the control parameter.

In S110, the learning device determines whether a predetermined termination condition is satisfied. If the conditions are satisfied, the process proceeds to S111; if the conditions are not satisfied, the process returns to S102. The predetermined termination condition is, for example, the completion of learning processing for a plurality of image data specified in advance.

In S111, the learning device retrieves the learning results. The learning results here become weighting coefficients for the recognition network 302 and sensor control network 313. That is, the obtained weighting coefficients are stored in the RAM 204 of the pattern recognition device 201. This allows the pattern recognition device 201 to more appropriately perform pattern recognition processing.

As described above, according to the first embodiment, the sensor control network 313 is trained using the sensor model 901 including image data and the sensor control model 1002. This allows the pattern recognition device 201 to perform pattern recognition that is more robust against various fluctuations in the data to be processed.

Furthermore, the sensor control network 313 regresses the control signal using the feature map 308 in the lower layer of the recognition network 302. That is, the sensor control network 313 shares with the recognition network 302 the feature amount obtained in the calculation process of the recognition network 302. This is expected to improve regression performance and facilitate learning in the pattern recognition device 201, as well as reduce the overall calculation cost.

(Second embodiment)
In the second embodiment, a mode will be described in which a part of the recognition network 302 is also learned when the sensor control network 313 is learned. That is, in the first embodiment, a case has been described in which the recognition network 302 is not trained when the sensor control network 313 is trained, but the learning method is not limited to this.

<Operation of learning device>
FIG. 12 is a diagram showing a specific example of learning processing in the second embodiment. More specifically, it shows an operation pattern in learning of the control network, and corresponds to the operation pattern 1009 of the first embodiment. Other processes are the same as those in the first embodiment (FIGS. 1 and 10), so description thereof will be omitted.

Similar to the first embodiment, the sensor model is shown divided into a sensor imaging model 1201 and a sensor control model 1202. Further, nodes 1203 to 1204 are nodes of the recognition network 302, and node 1205 is a node of the sensor control network 313.

As described above, in the second embodiment, the coefficient W' ₁ of the recognition network 302 is updated to W'' ₁ when the sensor control network 313 learns.More specifically, as in the first embodiment, equation (10 ) is used to update W' _r , and W' ₁ is updated using the following equation (12).

By performing such a learning process, the feature quantity output by the node n ₁ (node 1203) becomes a feature quantity suitable for the sensor control network 313 as well.

As explained above, according to the second embodiment, more robust pattern recognition is possible compared to the first embodiment.

Further, similarly to the first embodiment (FIG. 1), the recognition network 302 and the sensor control network 313 can be learned alternately (coevolutionary learning). In that case, suitable coefficients can be learned for each network. Therefore, the influence on the performance of the recognition network 302 due to learning of the sensor control network 313 can be reduced. Furthermore, by setting the learning coefficient β in Equation (12) to a small value, it is possible to further reduce the influence on the performance of the recognition network 302.

(Third embodiment)
In the third embodiment, a mode will be described in which the recognition network 302 and the sensor control network 313 are learned independently. That is, in the first and second embodiments, the case where the recognition network 302 and the sensor control network 313 are learned alternately has been described, but the learning method is not limited to this.

<Operation of learning device>
FIG. 13 is an operational flowchart of learning processing in the third embodiment. Note that S1301 to S1305 and S1310 to S1313 are the same as S101 to S105 and S106 to S109 in FIG. 1, so a description thereof will be omitted.

In S1306, the learning device determines whether a predetermined termination condition is satisfied. If the conditions are met, the process advances to S1307; if not, the process returns to S1302. The predetermined termination condition is, for example, the completion of learning processing for a plurality of image data specified in advance. In S1307, the learning device retrieves the learning results. The learning results here become weighting coefficients for the recognition network 302.

In S1308, similarly to S1302, the learning device selects learning data to be used in the learning process. In S1309, similarly to S1303, the learning device converts the image data according to the control conditions of the sensor model to generate pseudo sensor data.

In S1314, the learning device determines whether a predetermined termination condition is satisfied. If the conditions are met, the process advances to S1315; if not, the process returns to S1308. The predetermined termination condition is, for example, the completion of learning processing for a plurality of image data specified in advance. In S1315, the learning device retrieves the learning results. The learning result here becomes a weighting coefficient for the sensor control network 313.

As explained above, according to the third embodiment, the recognition network 302 and the sensor control network 313 are learned separately. With this configuration, the sensor control network 313 can be learned without affecting the learned recognition network 302.

(Modified example)
In the embodiments described above, the task of detecting a specific pattern in an image has been described as an example of the recognition network, but the present invention is not limited thereto. It can be applied to various recognition tasks, such as recognizing the attributes of a recognition target or understanding the content of an image. Furthermore, it is applicable not only to recognition tasks but also to various image processing tasks such as geometric image transformation, brightness/color correction, noise removal, and format conversion. This can be expected to improve the quality of generated images.

Although the above-described embodiment describes an example of a two-dimensional image sensor, the present invention is not limited to this. For example, it can be applied to various sensors with different data dimensions and modalities. Furthermore, it is applicable to systems that use voice data, radio wave sensor data, and various sensing devices.

Although the above embodiment describes the case where the gain is controlled in the sensor control network 313, the present invention is not limited thereto. For example, it can be applied to control various other readout parameters such as exposure time, frame rate, sensitivity, resolution, etc.

In the above-described embodiment, a case has been described in which the coupling coefficients (weighting coefficients) of a neural network are learned, but the present invention may also be applied to a method of learning the configuration of a network at the same time, such as the NeuroEvolution method.

In the above-described embodiment, a backpropagation method was used as the learning method, but the present invention is not limited thereto. For example, it is possible to apply various other metaheuristic techniques such as genetic algorithms. In this case, the present invention can be applied even when it is difficult to set the inverse function of the sensor model required for error backpropagation.

In the above-described embodiment, a case has been described in which the imaging control processing unit 705 controls the imaging conditions on a block-by-block basis, but the present invention is not limited to this. Control may be performed pixel by pixel, or the entire image may be controlled at once. When controlling the entire image at once, the configuration may be such that the data of the feature map in the final layer of the sensor control network 313 is passed through a linear discriminator to calculate the control data. Alternatively, a configuration may be adopted in which the control data is the result of performing global pooling processing on the feature map of the final layer.

In the above-described embodiment, the sensor control network 313 uses the control signal (regression) using the feature map 308 in the lower layer of the recognition network 302, but the present invention is not limited thereto. The feature map of the upper layer may be used, or the feature map of each layer may be selected and used. Further, the hierarchical structure (the number of layers and the number of feature maps in a layer) of the recognition network 302 and the sensor control network 313 may have any configuration depending on the recognition target, control target, etc. to which they are applied. Furthermore, an independent sensor control network may be constructed using the output of the imaging device 704 as input without using the feature map of the recognition network 302. However, even in that case, the recognition network 302 is used for learning when the sensor control network 313 is trained.

In the above-described embodiment, a case has been described in which reliability of pattern recognition and control conditions are generated in the final layer of hierarchical feature extraction processing, but the present invention is not limited to this. For example, a configuration may be adopted in which recognition and control data generation are realized by directly referring to the feature map of the intermediate layer.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

701 Feature extraction processing unit; 702 Control unit; 703 Memory; 704 Imaging device; 705 Imaging control processing unit; 706 Image correction processing unit; 302 Recognition network; 313 Sensor control network

Claims (12)

An information processing device connectable to a sensing device,
Setting means for setting data acquisition conditions in the sensing device;
a first processing means for performing hierarchical feature extraction processing on data obtained by the sensing device using a first neural network (NN);
second processing means for generating regression data indicative of data acquisition conditions to be used in subsequent data acquisition by the sensing device using the feature map in the intermediate layer of the first NN;
has
The information processing apparatus is characterized in that the setting means sets data acquisition conditions indicated in the regression data to the sensing device. The information processing apparatus according to claim 1, wherein the second processing means generates the regression data using a second NN. The information processing apparatus according to claim 2, wherein at least one of the first NN and the second NN is a convolutional neural network (CNN). The information processing apparatus according to any one of claims 1 to 3, wherein the second processing means generates the regression data using a feature map in a lower hierarchy of the first NN. . 5. The information processing apparatus according to claim 1, wherein a control unit of data acquisition conditions in the regression data is a partial area unit of the sensing device. The sensing device is an imaging device,
6. The information processing apparatus according to claim 1, wherein the data is image data. 3. A method for learning weighting coefficients of the second NN in the information processing device according to claim 2, comprising:
a generation step of generating pseudo data that simulates data output from the sensing device based on learning data and data acquisition conditions using a sensor model according to characteristics of the sensing device;
a first step of learning weighting coefficients of the first NN using the pseudo data;
a second step of learning a weighting factor of the second NN using the pseudo data and the weighting factor of the first NN learned in the first step;
method including. The first step is
a first recognition step of performing a recognition process using the first NN with the pseudo data as input;
A first method of learning weighting coefficients of the first NN by back-propagating the first NN with errors in teacher data prepared in advance corresponding to the recognition result of the first recognition step and the learning data. learning process and
8. The method according to claim 7, comprising: The second step is
a second recognition step of performing a recognition process using the first NN with the pseudo data as input;
Using the result of back-propagating the error of the teacher data prepared in advance corresponding to the recognition result of the second recognition step and the learning data through the first NN, and the inverse function of the sensor model. a second learning step of learning weighting coefficients of the second NN;
9. The method according to claim 7 or 8, comprising: 10. The method according to claim 9, wherein in the second learning step, some weighting coefficients included in the first NN are fixed. 11. The method according to claim 7, wherein the first step and the second step are performed alternately on a plurality of pieces of learning data. A program for causing a computer to execute the method according to any one of claims 7 to 11.

Images