JP2021089493A - Information processing apparatus and learning method thereof - Google Patents

Abstract

To allow for more robust pattern recognition for a diversity of variations of processing object data.SOLUTION: An information processing apparatus connectable to a sensing device includes setting means which sets a data acquisition condition in the sensing device, first processing means which uses a first neural network (NN) to perform hierarchical feature extraction processing on data obtained by the sensing device, and second processing means which uses a feature map in an intermediate layer of the first NN to generate regression data indicative of a data acquisition condition for use in following data acquisition in the sensing device. The setting means sets the data acquisition condition indicated by the regression data to the sensing device.SELECTED DRAWING: Figure 3

Description

The present invention relates to a pattern recognition process using a neural network.

As a method that enables robust pattern recognition with respect to fluctuations in the recognition target, a hierarchical calculation method represented by a convolutional neural network (hereinafter abbreviated as CNN) is drawing attention. For example, Non-Patent Document 1 discloses various application examples and implementation examples of a pattern recognition method based on a deep learning technique.

Further, as a method for dealing with large fluctuations in the shooting environment (lighting, subject state, etc.) of the recognition object, in Patent Document 1, the shooting conditions of the shooting device are changed at predetermined intervals to determine the face detection probability in the image. Techniques for improvement are disclosed. Further, Patent Document 2 discloses a method of controlling the gain and exposure time of an imaging device based on the result of face detection and reacquiring image data under conditions suitable for attribute recognition processing of the detected person.

Japanese Unexamined Patent Publication No. 2014-127999 Japanese Unexamined Patent Publication No. 2017-098746

Yann LeCun, Koray Kavukcuoglu and Clement Farabet, "Convolutional Networks and Applications in Vision", Proc. International Symposium on Circuits and Systems (ISCAS'10), IEEE, 2010

However, the method described in Patent Document 1 has a configuration in which the photographing conditions are changed at predetermined intervals, so that it is not always possible to acquire an appropriate image. Further, the method described in Patent Document 2 has a problem that it is difficult to determine in advance a change table of optimum shooting conditions for various fluctuations in the shooting environment. In addition, it is not possible to deal with cases where the optimum shooting conditions are different for each region of the same frame image.

The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique capable of more robust pattern recognition for various fluctuations of data to be processed.

In order to solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configurations. That is, it is an information processing device that can be connected to a sensing device.
A setting means for setting data acquisition conditions in the sensing device, and
A first processing means for executing a hierarchical feature extraction process using a first neural network (NN) on the data obtained by the sensing device, and
A second processing means that uses the feature map in the middle layer of the first NN to generate regression data indicating the data acquisition conditions used in subsequent data acquisition by the sensing device.
Have,
The setting means sets the data acquisition conditions shown in the regression data in the sensing device.

According to the present invention, it is possible to provide a technique that enables more robust pattern recognition for various fluctuations in the data to be processed.

It is an operation flowchart of the learning process in 1st Embodiment. It is a figure which shows the schematic structure of an image processing system and a learning apparatus. It is a figure explaining the processing in a recognition processing part. It is a figure which shows the detailed structure of the hierarchical feature extraction processing part. It is a figure explaining the operation timing in a pattern recognition apparatus. It is a figure explaining the structure of a laminated device. It is a figure which shows the detailed structure of the pattern recognition apparatus. It is a figure explaining a regression map. It is a figure explaining the operation of the learning process of each network. It is a figure which shows the specific example of the learning process in 1st Embodiment. It is a figure which shows the relationship between the gain control value of a sensor, and an output signal. It is a figure which shows the specific example of the learning process in 2nd Embodiment. It is an operation flowchart of the learning process in 3rd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

(First Embodiment)
As a first embodiment of the information processing apparatus according to the present invention, an image processing system using a pattern recognition apparatus will be described below as an example.

<System and device configuration>
FIG. 2 is a diagram showing a schematic configuration of an image processing system and a learning device. FIG. 2A shows a configuration example of an image processing system using the pattern recognition device 201. The system has a function of detecting a region of a specific object from image data. On the other hand, FIG. 2B shows a configuration example of the learning device. The learning result (weighting coefficient) by the learning device will be used for the pattern recognition device 201. Although the image processing system and the learning device are described here as individual devices, they may be configured as an integrated device.

The image processing system includes a pattern recognition device 201, a CPU (Central Prosessing Unit) 205, a ROM (Read Only Memory) 206, a RAM (Random Access Memory) 207, and a DMAC (Direct Memory Access Controller). Further, the pattern recognition device 201 includes an image pickup device 202, a recognition processing unit 203, and a RAM 204.

The image pickup device 202 includes an optical system, a photoelectric conversion device, a driver circuit / AD converter, and the like. As the photoelectric conversion device, a CCD (Charge-Coupled Devices) or CMOS (Complimentary Metal Oxide Semiconductor) sensor or the like can be used. The recognition processing unit 203 executes a predetermined recognition process on the image data acquired by controlling the image pickup device 202. The RAM 204 is used as a calculation work buffer of the recognition processing unit 203. Here, in order to reduce the data transmission delay, the pattern recognition device 201 is described as a configuration including the image pickup device 202 and the recognition processing unit 203, but if it can be connected to the image pickup device, it may be a separate configuration. Good.

The CPU 205 controls the entire image processing system. The ROM 206 stores instructions and parameter data that define the operation of the CPU 205. The RAM 207 is a memory required for the operation of the CPU 205. The DMAC 208 controls data transfer and the like between the pattern recognition processing device 201 and the RAM 207. The data bus 209 is a data transfer path between each device.

The pattern recognition processing device 201 executes imaging and recognition processing according to an instruction from the CPU 205, and stores the result in the RAM 207. The CPU 205 provides various applications by using the recognition result.

The learning device includes an arithmetic unit 210, an interface device 212, and a storage device 213, and can be realized by, for example, a general-purpose computer device. The arithmetic unit 210 has a computer device such as a CPU and a memory, and executes a learning process described later with reference to FIG. The storage device 213 is a large-capacity data storage device such as a hard disk drive, and stores a program executed by the arithmetic unit 210, image data, teacher data, and the like used for learning. The interface device 212 is an interface for taking out the data obtained by learning, and is an interface of a communication interface or a portable storage device. The learning result by the learning device is taken out via the interface device 212 and stored in the ROM 206 or the like of the image processing system.

FIG. 7 is a diagram showing a detailed configuration of the pattern recognition device 201. It is a figure which described the structure of the recognition processing part 203 in more detail.

The feature extraction processing unit 701 repeatedly executes the hierarchical feature extraction processing while holding the intermediate result of the hierarchical calculation in the memory 703, and outputs the recognition processing result and the control data by using the extracted feature amount.

The image pickup device 704 corresponds to the image pickup device 202, and is composed of an optical system, a photoelectric conversion device, a driver circuit / AD converter, and the like. The image pickup control processing unit 705 controls the operation (shooting conditions, etc.) of the image pickup device 704 according to the control data provided by the feature extraction processing unit 701. Specifically, the shooting conditions include the gain for the signal after photoelectric conversion, the accumulation time (exposure time) of the photoelectric conversion device (photodiode, etc.), the characteristics of A / D conversion, and the like. The image pickup control processing unit 705 is configured to be able to control these shooting conditions in block units on the sensor surface. For example, with the recent development of semiconductor stacking and mounting technology, it has become possible to stack and mount control logic on a sensor surface, which makes it possible to realize read control in block units or pixel units.

FIG. 6 is a diagram illustrating a configuration of a laminated device. FIG. 6A schematically shows the physical configuration of the laminated device. Here, an example in which a sensor layer 61 for mounting a photoelectric conversion element, a logic layer 62 for mounting a read control logic, and a memory layer 63 for mounting a large-scale memory and its control unit are shown is shown. The sensor layer 61 corresponds to the image pickup device 704, the logic layer 62 corresponds to the image pickup control processing unit 705, and the memory layer 63 corresponds to the memory 703 and the like. A signal is transmitted between each layer by a penetrating via or the like.

FIG. 6B schematically shows the configuration of the logic layer 62. Here, in the logic layer 62, n × n control circuits for controlling the photoelectric conversion element of the sensor layer 61 are arranged. The control circuit ct (n, n) controls the reading of one or more photoelectric conversion elements of the sensor layer 61 existing at the corresponding positions. Therefore, in the above configuration, the reading conditions (gain, exposure time, etc.) can be controlled for each block of n × n blocks. That is, the imaging characteristics can be controlled for each of n × n parts in the image.

In the first embodiment, it is assumed that the feature extraction processing unit 701 is also mounted on the logic layer 62 and the memory layer 63. For example, by laminating and mounting on the sensor layer 61, control data can be fed back with less delay. When the shooting environment or the object changes rapidly, it is desired to control the image pickup device 704 with a smaller image frame delay, so that the laminated mounting on the sensor layer 61 is preferable.

FIG. 3 is a diagram illustrating processing in the recognition processing unit 203. The recognition processing unit 203 includes a recognition network 302 and a sensor control network 313, which are logical processing structures of the feature extraction processing unit 701. The recognition network 302 is an arithmetic network in which the image pickup device 704 recognizes the position of a predetermined object in the image pickup target 301 by the CNN. The sensor control network 313 is an arithmetic network that extracts information for controlling the imaging conditions of the imaging device by CNN.

Here, an example in which the recognition network 302 is composed of five layers of CNN is shown. The arithmetic processes 303 to 307 are arithmetic operations including a convolution operation, an activation function operation, a pooling operation, and the like, and are specifically realized by the configuration shown in FIG. 4 to be described later.

The feature maps 308 to 312 are called the intermediate layer (feature map 308 to 311) or the final layer (feature map 312) of the CNN arithmetic processing, and correspond to the results of the arithmetic processing 303 to 307, respectively. The feature maps 308 to 312 are stored in the memory 703. The feature maps 308 to 312 are two-dimensional data obtained by feature extraction processing on the image data output by the imaging device.

Here, the details of the two-dimensional CNN calculation processing for the image data will be described. When the kernel (filter coefficient matrix) size of the convolution operation is volumeSize × lowSize and the number of feature maps in the previous layer is L, one feature map is calculated by the multiply-accumulate operation shown in the following formula (1).

input (x, y): Reference pixel value in two-dimensional coordinates (x, y) output (x, y): Calculation result in two-dimensional coordinates (x, y) perpendicular (collect, low): Coordinates (x + volume, Weight coefficient in y + low) L: Number of feature maps in the previous layer coordinate, lowSize: Horizontal and vertical size of the two-dimensional convolution kernel In CNN arithmetic processing, multiple convolution kernels are processed in pixel units according to formula (1). The feature map is calculated by repeating the product-sum calculation while scanning and performing a non-linear conversion (activation processing) of the final product-sum calculation result. In addition, the generated feature map may be reduced by pooling processing and referred to in the next layer. The feature maps 308 to 312 have a plurality of maps in one hierarchy, and maps of features of different characteristics are generated corresponding to different weight coefficient groups.

FIG. 4 is a diagram showing a detailed configuration of the feature extraction processing unit 701. The feature extraction processing unit 701 is a concrete implementation configuration of the arithmetic processing 303 to 307.

The reference data buffer 401 is a circuit that acquires all or a part of the feature map data (input (x, y) in the mathematical expression (1)) of the previous layer, which is the reference data of the convolution operation, from the memory and buffers it. ..

The multiplier 402 and the cumulative adder 403 are circuits that execute the calculation of the mathematical expression (1).

The coefficient data buffer 404 is a circuit that transfers all or part of the weight coefficient data (weight (collect, low) in the mathematical formula (1)) obtained by learning in advance from the memory 703 in a predetermined unit and buffers it. ..

The activation processing circuit 405 is a circuit that processes a non-linear function such as ReLU (Rectified Linear Unit, Rectifier) with respect to the convolution operation result (output (x, y)) shown in the mathematical formula (1).

The pooling processing circuit 406 is a circuit that reduces the feature map by using a spatial filter such as a maximum value filter. When the pooling process is not performed, the result of the activation process 405 is stored in the memory 703, and when the pooling process is performed, the result of the pooling process 406 is stored in the memory 703. The data stored here becomes the feature map of the current hierarchy.

When the calculation of the feature map of the current layer is completed, the calculated feature map is used as the feature map of the previous layer, and the calculation of the feature map of the next layer is processed in the same manner. While sequentially referring to the feature maps stored in the memory 703 in this way, the feature maps of a plurality of layers are calculated. A hierarchical feature extraction process (CNN arithmetic process) is realized by controlling the operation of each component in FIG. 4 by a control unit 102 (not shown in FIG. 4).

By repeating feature extraction over a plurality of layers in this way, CNN realizes a recognition process that is robust to fluctuations in the identification target. The CNN determines the existence of a desired pattern by the calculation 307 of the final layer according to the feature extraction result of each layer. The feature map 312 of the final layer expresses the recognition result 320, and is, for example, a reliability map that expresses the existence probability of an object in the image as two-dimensional information. The operation 307 of the final layer may be composed of a fully connected neural network or a linear discriminator instead of the above-mentioned convolution operation.

In addition, the feature maps 308 to 311 of each layer express the feature extraction results for the input data, and generally, the lower layer (the layer close to the layer for inputting the data to be processed) shows low-level features such as edges. The upper layer (the layer closer to the recognition result) shows features with a high degree of abstraction. The characteristics of each feature map differ depending on the target of pattern recognition and the learning method.

Next, the sensor control network 313 will be described. The sensor control network 313 is an arithmetic network that regresses data for sensor control. That is, CNN is used to determine the data acquisition conditions of the sensor. The arithmetic processes 314 and 315 are the same arithmetic processes as the arithmetic processes 303 to 307, and are processed by the circuit shown in FIG.

In the sensor control network 313, the control signal is regressed by using the feature map 308 in the lower layer of the recognition network 302. That is, the features obtained in the calculation process of the recognition network are used. By sharing the feature map with the recognition network, it is expected that the regression performance will be improved and learning will be facilitated, and the overall calculation cost can be reduced. Further, since the sensor control network 313 is configured by network arithmetic processing (CNN) similar to the recognition network here, control data can be generated by using the feature extraction processing unit 701. That is, it is an advantage that a dedicated circuit or the like is not required.

The feature maps 316 to 317 are feature maps in the sensor control network 313, and generate a feature map 317 (hereinafter, referred to as a regression map 317) that regresses the control data of the shooting conditions by the calculation 315 of the final layer. The regression map 317 is control data for designating shooting conditions corresponding to the spatial position of the image sensor, and is, for example, data corresponding to the designation of the gain and exposure time of the image sensor corresponding to the position of the map. Only one regression map 317 is required when there is only one type of control target and the scalar value is used for control. If there are multiple control conditions or if the control parameters are vector data, there will be multiple regression maps.

FIG. 8 is a diagram illustrating a regression map. Specifically, an example of the regression map 317 corresponding to the control logic shown in FIG. 6B is schematically shown. rg (n, n) corresponds to the pixel data of the regression map. That is, the size of the regression map here is n × n. The value of the regression data is represented by shading, and corresponds to, for example, the gain of the data after photoelectric conversion.

The image pickup control processing unit 705 controls the photoelectric conversion element according to the control signal data regressed by the sensor control network 313, and acquires image data suitable for the recognition process. The image data obtained here is different from the image data for human observation to understand and appreciate the contents, and is suitable for improving the accuracy of the recognition process.

The sensor control network 313 has a pooling process in the arithmetic process 314 to reduce the size of the feature map. Therefore, the size of the regression map is smaller than the image size of the sensor output. That is, the read condition is controlled for each block having a plurality of pixels as a unit. The pooling ratio and the like are determined in consideration of the block size that can be controlled by the image pickup control processing unit 705.

FIG. 5 is a timing chart for explaining the operation timing of the pattern recognition process in the pattern recognition device 201. The horizontal axis represents the passage of time, and exemplifies the timing at which each process of the recognition network (recognition network 302), the control network (sensor control network 313), and the condition setting is executed. Here, the state in which the recognition process is continuously executed for the image data for three frames (first to third frames) that are continuous in time is shown.

At the timing 501, the shooting of the first frame and the processing of the recognition network for the first frame are executed, and at the timing 504, the processing of the control network for the subsequent second frame is executed. At the timing 507, the setting process of the operating conditions (gain, exposure time, etc.) of the imaging device for the second frame is executed.

At the timing 502, the shooting of the second frame and the processing of the recognition network for the second frame are executed, and at the timing 505, the processing of the control network for the subsequent third frame is executed. At the timing 508, the setting process of the operating conditions (gain, exposure time, etc.) of the imaging device for the third frame is executed.

At the timing 503, the shooting of the third frame and the processing of the recognition network for the third frame are executed, and at the timing 506, the processing of the control network for the subsequent fourth frame (not shown) is executed. At timing 509, a process of setting operating conditions (gain, exposure time, etc.) of the imaging device for the fourth frame (not shown) is executed.

In this way, the control network sequentially sets shooting conditions suitable for recognition according to changes in the situation of the shooting target, and the recognition network performs shooting according to the shooting conditions and executes pattern recognition processing.

<Operation of learning device>
Next, the learning process of the recognition network and the control network in the learning device will be described. As described above, the learning result (weighting coefficient) by the learning device is used for the pattern recognition device 201.

FIG. 9 is a diagram illustrating the operation of the learning process of the recognition network and the control network. The learning here means a process of determining the weighting coefficients of the neural networks of the recognition network 302 and the sensor control network 313 so that the pattern recognition process has more preferable (or best) performance. In FIG. 9, the arithmetic processes 303 to 307 and 314 to 315 of FIG. 3 are omitted.

The sensor model 901 is a sensor model necessary for generating data used for learning each network. The sensor model 901 is, for example, a pseudo sensor data 902 (pseudo) that simulates the output data of the sensor according to the control conditions and the characteristics of the sensor from the image data 301 (two-dimensional data in the image format) taken by another imaging device. Used to generate data). That is, the pseudo sensor data 902 is image data that simulates the output of the sensor when it is assumed that the shooting target is the image data 301.

The sensor model 901 can be theoretically created according to the physical characteristics of the sensor. However, it may be created by a learning method such as GAN (Generative Adversarial Network) using the data obtained by the actual sensor. Further, the sensor model 901 has a function of simulating the fluctuation of the output with respect to the control signal for controlling the reading condition of the sensor. For example, when a high gain value is set as a control signal, the output also outputs a high value.

That is, the sensor model 901 is a model that defines both the relationship between the image data 301 and the pseudo sensor data 902 taken by another imaging device and the relationship between the control signal and the pseudo sensor data. It is assumed that the sensor model is created in advance before learning the recognition network and the sensor control network.

The memory 903 is a memory that holds the intermediate calculation result of the recognition network necessary for learning the sensor control network 313. The teacher data 905 is teacher data created in pairs with the image data 301 and is prepared in advance. The teacher data here is the data distribution of the recognition result 320 expected as the recognition result. For example, when detecting a face in an image, it is assumed that the map data is in the normal minute format with the center of the face as the peak. The process 904 is a process for calculating the difference between the recognition result 320 and the teacher data 905.

FIG. 9A is a diagram illustrating a main process when learning the recognition network. The image data 301 is converted by the sensor model 901 to obtain pseudo sensor data 902. The pattern recognition process is executed using the generated pseudo sensor data 902, and the recognition result 320 is acquired. The recognition network is learned by the back propagation method using the difference between the recognition result 320 and the teacher data 905 as error data. That is, the weighting coefficients of the convolution operation for generating the feature maps 308 to 312 are sequentially updated.

FIG. 9B is a diagram illustrating a main process when learning the sensor control network 313. The sensor control network 313 realizes the sensor control signal generation process. When learning the sensor control network 313, the error information, which is the difference information between the recognition result 320 and the teacher data 905, is back-propagated. At this time, learning is not performed in the recognition network 302 (that is, the coefficient is fixed). Further, the error information for training the sensor control network 313 is acquired via the inverse function of the sensor model. The sensor model 901 holds data generated as pseudo sensor data and table information for realizing an inverse function in order to calculate a correct value for sensor control.

The sensor control network is learned by the backpropagation method using the obtained error information and the data of the feature map 308 (stored in the memory 903) obtained by the recognition process. That is, the coefficients of the convolution operation that generate the feature maps 316 to 317 are sequentially updated. The backpropagation method uses a conventionally proposed method for processing. The feature of this process is that the recognition network is fixed and an error of the recognition result is given to the sensor control network 313 via an inverse function of the sensor model. As for the inverse function of the sensor model, the inverse function according to the characteristics of the sensor is determined in advance.

FIG. 10 is a diagram showing a specific example of the operation in the learning process according to the first embodiment. 10 (a) and 10 (b) show the pattern recognition process and the operation pattern of the learning process when learning the recognition network. 10 (c) and 10 (d) show the operation patterns of the pattern recognition process and the learning process when learning the sensor control network. FIG. 10 shows an example in which the number of nodes of the neural network of the recognition network 302 is two (nodes 1003 and 1004) and the number of nodes of the neural network of the sensor control network 313 is one (node 1005).

Further, for the sake of simplicity, an example of a neural network having an MLP (Multi Layer Perceptron) configuration will be described instead of CNN. When it is regarded as CNN, it corresponds to the case where the kernel size of the convolution operation is 1x1. Further, here, the sensor model is divided into a sensor imaging model 1001 and a sensor control model 1002. In FIG. 10, the recognition result for the image data 301 of the learning data set is the recognition result 320, and the teacher data 905 and the arithmetic processing 904 are the same as those in FIG.

FIG. 1 is an operation flowchart of the learning process according to the first embodiment. In S101, the learning device executes the initialization process. Specifically, various initialization processes such as initialization of the sensor model (sensor imaging model 1001 and sensor control model 1002) are executed.

In S102, the learning device selects the learning data to be used for the learning process. For example, the image data 301 and the teacher data 905 for learning are selected from the learning data set stored in the storage device 213 and read into a memory (not shown) of the arithmetic unit 210.

In S103, the learning device converts the image data according to the control conditions of the sensor model. Here, the image data 301 is converted according to the conditions set in the sensor control model 1002 (for example, sensor control conditions such as sensitivity, gain, exposure time, etc.) to generate pseudo sensor data which is pseudo sensor data.

In S104, the learning device executes a predetermined pattern recognition process on the pseudo sensor data generated in S103 (operation pattern 1006). The pattern recognition process here is, for example, a pattern recognition process such as detecting a face in an image. This pseudo sensor data 902 converted by the sensor imaging model 1001, it executes the recognition process in operation node _n 1, _{n 2} of the neural network, obtaining a recognition result 320. By executing the recognition process on the two-dimensional data of the image data 301 in the raster order, the recognition result 320 also becomes a two-dimensional map.

In S105, the learning device executes learning of the pattern recognition process based on the recognition result 320 obtained in S104 (operation pattern 1007). Here, the weighting coefficient of the recognition network is learned by the backpropagation method using the teacher data selected in S102. Using the difference value between the recognition result 320 and the teacher data 905 as an error, the _{coefficient W n2 for the node n 2} (node 1004) and the _{coefficient W n1 for the node n 1} (node 1003) are sequentially updated. _{The coefficient W r} for the node r (node 1005) is not updated during the calculation of S105.

Hereinafter, a specific example of learning by the backpropagation method will be described. _{In the back propagation method, the coefficients W 1} and W ₂ are adjusted so that the error between the recognition result 320 and the teacher data 905 with respect to the respective image positions is minimized.

The output value corresponding to a pixel position included in the recognition result 320 y, the teacher data value corresponding to that position and y _t, which defines the error E teacher data and the output value by the following expression (2) .. The notation of coordinate data is omitted here for the sake of simplicity.

Assuming that the output of _{node n 1 is n 1} and α is the learning coefficient, the coefficient W ₂ is updated to W ' _{2 by the following mathematical formula (3).}

Assuming that the nonlinear function of the nodes n ₁ and n _{2 is the ReLU function f ReLU} (), y = f _ReLU (W ₂ × n ₁ ). Then, the mathematical formula (3) is the following mathematical formula (4) because the derivative of the fReLU function is _{1 when W 2} × n 1> 0.

Then, the output of the sensor model of the corresponding pixel position s, the learning coefficient is alpha, and updates the W ₁ W 'to ₁ by the following equation (5).

Here, n ₁ = f _ReLU (W ₁ × s). Therefore, Equation (5) in the case of _{W 1} × s> 0, can be calculated following Equation (6), and the W _'1 after updating.

In S106, the learning device executes the pattern recognition process again on the recognition network of the weighting coefficient learned (updated) in S105, and outputs the recognition result 320. At the same time, the operation result n _{1 of the node n 1} (node 1003) is stored in the memory 903 (operation pattern 1008).

In S107, the learning device executes learning of the sensor control network 313 based on the result of the pattern recognition process of S106. Here, the sensor control network 313 is learned by using the teacher data 905 selected in S102 and the error E'of the recognition result 320. The error E'here is an error with respect to the pattern recognition processing result after the recognition network update calculated in S106.

First, the coefficient of the recognition network 302 is fixed and the error E'is back-propagated. A recognition network 302 from the sensor control value r stored and the sensor model and the inverse propagate error was calculated Es (= E '× W' 2 × W '1), to estimate the correct value rt of the sensor control. _{Here, the estimation is performed according to the inverse function frev} (back propagation error, control value) of the sensor model 904 (mathematical expression (7)).

The f _rev function is an inverse function of the sensor model (sensor imaging model 1001 and sensor control model 1002). Calculated backward the correct value r _t of the control parameter from the error value Es and the current control value r of the sensor data.

FIG. 11 is a diagram showing the relationship between the gain control value of the sensor and the output signal. More specifically, it is a diagram schematically showing the inverse function of the sensor model by taking gain control as an example. The straight line 1101 shows a function for realizing an inverse function, and is a function expressing the relationship between the gain control value of the sensor and the output signal. Although it is shown as a linear function, it is actually an arbitrary function determined based on logical analysis or experiment, and is retained as an approximate function or table information. The straight line 1101 corresponds to a combination of the inverse function of the sensor imaging model 1001 and the inverse function of the sensor control model 1002.

Point 1102 is a point showing the relationship between the control value r of the gain control and the output signal at that time. Point 1103 indicates a point for obtaining the correct value of gain control according to the sensor output error Es. By using the sensor output error signal output signal during the pseudo sensor data generated to be stored in memory in the model r (the point 1102) is back propagation from the recognition network Es, obtains gain control of the correct value r _{t (point} 1103) ..

With correct value _{r t} of the resulting gain control, by a back propagation method, it updates the weight coefficient _{W r} of node r (Node 1005) (operation pattern 1009). More specifically, the update is performed based on the output data of _{the node n 1} (node 1003) stored in the memory 903.

Weight coefficient W _r of the sensor control network 313 is updated by the following equation and the learning coefficient and beta (8).

Error data for learning a sensor control network 313 to E _r, the output value of the node r of the r sensor control network 313, if the r _t a correct value of the sensor control values satisfy the equation (9).

Therefore, the mathematical formula (8) can be transformed into the following mathematical formula (10).

As a result, the coefficient of the sensor control network can be updated from the mathematical formula (7) and the mathematical formula (10) by the following mathematical formula (11).

The above processing is executed for all or a plurality of selected positions included in the image data 301. _{That is, the coefficient W'r} is learned so that the map 317 corresponding to the regression information that appropriately controls the sensor is generated.

In S108, the learning device uses the sensor control network of the updated weight coefficient W _'r regressing sensor control parameter.

In S109, the learning device sets the regressed parameters as control parameters of the sensor model. That is, in S103 for the next image data (next loop), the image data 301 is converted into pseudo sensor data 902 using the control parameters set here. The control unit of the sensor is a subregion unit obtained according to the size of the regression data map 318 corresponding to the control parameter.

In S110, the learning device determines whether or not the predetermined end condition is satisfied. If it is satisfied, the process proceeds to S111, and if it is not satisfied, the process returns to S102. The predetermined end condition is, for example, the completion of the learning process for a plurality of predetermined image data.

In S111, the learning device takes out the learning result. The learning result here becomes the weighting coefficient of the recognition network 302 and the sensor control network 313. That is, the acquired weighting coefficient is stored in the RAM 204 of the pattern recognition device 201. As a result, the pattern recognition device 201 can execute the pattern recognition process more appropriately.

As described above, according to the first embodiment, the sensor control network 313 is learned by using the sensor model 901 including the image data and the sensor control model 1002. As a result, the pattern recognition device 201 can perform more robust pattern recognition with respect to various fluctuations in the data to be processed.

Further, in the sensor control network 313, the control signal is regressed by using the feature map 308 in the lower layer of the recognition network 302. That is, the sensor control network 313 shares the feature amount obtained in the calculation process of the recognition network 302 with the recognition network 302. As a result, in the pattern recognition device 201, it is expected that the regression performance will be improved and learning will be facilitated, and the overall calculation cost can be reduced.

(Second Embodiment)
In the second embodiment, a mode in which a part of the recognition network 302 is also learned at the time of learning the sensor control network 313 will be described. That is, in the first embodiment, the case where the recognition network 302 is not learned at the time of learning the sensor control network 313 has been described, but the learning method is not limited to this.

<Operation of learning device>
FIG. 12 is a diagram showing a specific example of the learning process in the second embodiment. More specifically, it shows an operation pattern in learning of a control network, and corresponds to an operation pattern 1009 of the first embodiment. Since other processes are the same as those in the first embodiment (FIGS. 1 and 10), description thereof will be omitted.

Similar to the first embodiment, the sensor model is divided into a sensor imaging model 1201 and a sensor control model 1202. Further, the nodes 1203 to 1204 are the nodes of the recognition network 302, and the nodes 1205 are the nodes of the sensor control network 313.

As described above, in the second embodiment, More specifically. For updating the coefficients W _'1 of the recognition network 302 at the time of learning of the sensor control network 313 W _"1, similarly to the first embodiment Formula (10 'and it updates the _r, W by the following equation (12)' W in) to update _1.

By performing such learning processing, _{the feature amount output by the node n 1} (node 1203) becomes a feature amount suitable for the sensor control network 313.

As described above, according to the second embodiment, more robust pattern recognition is possible as compared with the first embodiment.

Further, similarly to the first embodiment (FIG. 1), the recognition network 302 and the sensor control network 313 can be alternately learned (co-evolutionary learning). In that case, it is possible to learn suitable coefficients for each network. Therefore, it is possible to reduce the influence on the performance of the recognition network 302 due to the learning of the sensor control network 313. Further, it is possible to reduce the influence on the performance of the recognition network 302 by setting the learning coefficient β of the mathematical formula (12) to a small value.

(Third Embodiment)
In the third embodiment, a mode in which the recognition network 302 and the sensor control network 313 are learned independently will be described. That is, in the first and second embodiments, the case where the recognition network 302 and the sensor control network 313 are alternately learned has been described, but the learning method is not limited to this.

<Operation of learning device>
FIG. 13 is an operation flowchart of the learning process according to the third embodiment. Since S1301 to S1305 and S131 to S1313 are the same as S101 to 105 and S106 to 109 in FIG. 1, description thereof will be omitted.

In S1306, the learning device determines whether or not the predetermined end condition is satisfied. If it is satisfied, the process proceeds to S1307, and if it is not satisfied, the process returns to S1302. The predetermined end condition is, for example, the completion of the learning process for a plurality of predetermined image data. In S1307, the learning device takes out the learning result. The learning result here becomes the weighting coefficient of the recognition network 302.

In S1308, similarly to S1302, the learning device selects the learning data to be used for the learning process. In S1309, similarly to S1303, the learning device converts the image data according to the control conditions of the sensor model to generate pseudo sensor data.

In S1314, the learning device determines whether or not the predetermined end condition is satisfied. If it is satisfied, the process proceeds to S1315, and if it is not satisfied, the process returns to S1308. The predetermined end condition is, for example, the completion of the learning process for a plurality of predetermined image data. In S1315, the learning device takes out the learning result. The learning result here becomes the weighting coefficient of the sensor control network 313.

As described above, according to the third embodiment, the recognition network 302 and the sensor control network 313 are learned separately. With this configuration, the sensor control network 313 can be learned without affecting the learned recognition network 302.

(Modification example)
In the above-described embodiment, the case of the task of detecting a specific pattern in the image has been described as an example of the recognition network, but the present invention is not limited thereto. It can be applied to various recognition tasks such as a task of recognizing the attributes of a recognition object and a task of understanding the contents of an image. Furthermore, it can be applied not only to recognition tasks but also to various image processing tasks such as geometric transformation of images, brightness / color correction, noise removal, and format conversion. This can be expected to improve the generated image quality.

In the above-described embodiment, an example for a two-dimensional image sensor has been described, but the present invention is not limited to this. For example, it can be applied to various sensors with different data dimensions and modality. In addition, voice data and radio wave sensor data can be applied to systems using various sensing devices.

In the above-described embodiment, the case where the gain is controlled in the sensor control network 313 has been described, but the present invention is not limited thereto. For example, it can be applied to control various other readout parameters such as exposure time, frame rate, sensitivity, and resolution.

In the above-described embodiment, the case of learning the coupling coefficient (weighting coefficient) of the neural network has been described, but it may be applied to a method of simultaneously learning the network configuration such as the NeuroEvolution method.

In the above-described embodiment, the case where the backpropagation method is used as the learning method has been described, but the present invention is not limited to this. For example, various other metaheuristic methods such as genetic algorithms can be applied. In this case, the present invention can be applied even when it is difficult to set the inverse function of the sensor model required for error back propagation.

In the above-described embodiment, the case where the image pickup control processing unit 705 controls the shooting conditions in block units has been described, but the present invention is not limited to this. It may be controlled in pixel units, or the entire image may be controlled collectively. When the entire image is collectively controlled, the control data may be calculated by passing the data of the feature map of the final layer of the sensor control network 313 through a linear discriminator. Alternatively, the control data may be the result of performing global pooling processing on the feature map of the final layer.

In the above-described embodiment, the sensor control network 313 uses (regresses) the control signal by using the feature map 308 in the lower layer of the recognition network 302, but the present invention is not limited to this. The feature map of the upper layer may be used, or the feature map of each layer may be selected and used. Further, the hierarchical structure (the number of layers and the number of feature maps in the hierarchy) of the recognition network 302 and the sensor control network 313 may have any configuration depending on the recognition target to be applied, the control target, and the like. Further, an independent sensor control network may be configured by using the output of the image pickup device 704 as an input without using the feature map of the recognition network 302. However, even in that case, the recognition network 302 is used for learning when learning the sensor control network 313.

In the above-described embodiment, the case where the reliability and control conditions of pattern recognition are generated in the final layer of the hierarchical feature extraction process has been described, but the present invention is not limited to this. For example, it may be configured to realize recognition and control data generation by directly referring to the feature map of the intermediate layer.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

701 Feature extraction processing unit; 702 control unit; 703 memory; 704 imaging device; 705 imaging control processing unit; 706 image correction processing unit; 302 recognition network; 313 sensor control network

Claims (12)

An information processing device that can be connected to a sensing device
A setting means for setting data acquisition conditions in the sensing device, and
A first processing means for executing a hierarchical feature extraction process using a first neural network (NN) on the data obtained by the sensing device, and
A second processing means that uses the feature map in the middle layer of the first NN to generate regression data indicating the data acquisition conditions used in subsequent data acquisition by the sensing device.
Have,
The information processing apparatus is characterized in that the setting means sets the data acquisition conditions shown in the regression data in the sensing device. The information processing apparatus according to claim 1, wherein the second processing means uses a second NN to generate the regression data. The information processing apparatus according to claim 2, wherein at least one of the first NN and the second NN is a convolutional neural network (CNN). The information processing apparatus according to any one of claims 1 to 3, wherein the second processing means generates the regression data by using the feature map in the lower layer of the first NN. .. The information processing apparatus according to any one of claims 1 to 4, wherein the control unit of the data acquisition condition in the regression data is a subregion unit of the sensing device. The sensing device is an imaging device.
The information processing apparatus according to any one of claims 1 to 5, wherein the data is image data. A method of learning the weighting coefficient of the second NN in the information processing apparatus according to claim 2.
A generation step of generating pseudo data that simulates the data output from the sensing device based on the learning data and the data acquisition conditions using the sensor model according to the characteristics of the sensing device.
The first step of learning the weighting coefficient of the first NN using the pseudo data, and
A second step of learning the weighting coefficient of the second NN by using the pseudo data and the weighting coefficient of the first NN learned in the first step, and
How to include. The first step is
A first recognition step of executing a recognition process using the first NN with the pseudo data as an input, and
A first that learns the weighting coefficient of the first NN by back-propagating the error between the recognition result by the first recognition step and the teacher data prepared in advance corresponding to the learning data by back-propagating the first NN. Learning process and
7. The method of claim 7. The second step is
A second recognition step of executing the recognition process using the first NN with the pseudo data as an input, and
The recognition result by the second recognition step, the result of back-propagating the error of the teacher data prepared in advance corresponding to the learning data by the first NN, and the inverse function of the sensor model are used. In the second learning step of learning the weighting coefficient of the second NN,
The method according to claim 7 or 8, wherein the method comprises. The method according to claim 9, wherein in the second learning step, a part of the weighting factors included in the first NN is fixed. The method according to any one of claims 7 to 10, wherein the first step and the second step are alternately executed on a plurality of learning data. A program for causing a computer to execute the method according to any one of claims 7 to 11.

Images