JP7401246B2 - Imaging device, method of controlling the imaging device, and program - Google Patents

Description

The present invention relates to a technique for controlling imaging parameters used during imaging.

As a technology for controlling imaging parameters that enables acquisition of a desired image in an imaging device, a technology that executes recognition processing on an image and controls imaging parameters based on the obtained recognition results is widely used. Proposed. Patent Document 1 discloses that based on the results of face recognition of a person using an image taken by a camera such as a monitoring device, the camera is adjusted in the vertical direction, horizontal direction, rotation direction, etc. to make it easier to confirm the person's face. A technique for determining at least one imaging parameter of a shooting direction and an angle of view is disclosed.

Japanese Patent Application Publication No. 2014-64083

In the case of the imaging parameter control technique based on the image recognition results described above, the image to which recognition processing is applied is an image that has already been acquired. Therefore, when the imaging parameters are controlled based on the image recognition results, the position of the recognition target on the image and the pixel-level brightness data have already changed in the newly acquired image after reflecting the imaging parameters. There is a possibility that In this case, the values of the imaging parameters controlled to obtain a desired image such as a face image may end up being inappropriate values for the newly captured image based on the imaging parameters. Possible. In other words, the imaging parameters may not be suitable for a newly captured image.

Therefore, an object of the present invention is to make it possible to determine imaging parameters that are more suitable for an image that is captured based on the imaging parameters.

The imaging device of the present invention includes an imaging sensor capable of setting imaging parameters for each pixel or region of an image, an imaging means for acquiring an image of a moving object, and an imaging means for acquiring an image of a moving object based on the acquired image. , an image generation unit that generates a predicted image that predicts an image that will be captured after the object moves ; an image recognition unit that performs image recognition processing on the predicted image generated by the image generation unit ; parameter determining means for determining the imaging parameters of the imaging sensor for each pixel or region based on the position of the object in the predicted image obtained by recognition processing; and parameter setting means for setting each pixel or region , and the imaging means acquires an image according to the parameters set by the parameter setting means .

According to the present invention, it is possible to determine imaging parameters that are more suitable for an image that is captured based on the imaging parameters.

1 is a diagram illustrating a configuration example of an imaging device according to a first embodiment; FIG. 3 is a processing flowchart according to the first embodiment. FIG. 2 is a diagram illustrating a configuration example for realizing predicted image generation processing according to the first embodiment. 3 is a diagram showing an example of a configuration for realizing image recognition processing according to the first embodiment; FIG. FIG. 3 is an explanatory diagram of image recognition processing according to the first embodiment. 7 is a processing flowchart according to Embodiment 2. 7 is a diagram illustrating a configuration example for realizing image recognition processing according to a second embodiment. FIG. FIG. 7 is an explanatory diagram of image recognition processing (object recognition processing) according to the second embodiment. FIG. 7 is an explanatory diagram of image recognition processing (object posture recognition processing) according to the second embodiment. FIG. 7 is an explanatory diagram of a method for calculating the center position of the human head according to the second embodiment. FIG. 3 is an explanatory diagram of an imaging area according to Embodiment 2. FIG. 12 is a processing flowchart according to Embodiment 3. FIG. 7 is an explanatory diagram of image recognition processing according to Embodiment 3; FIG. 7 is a diagram illustrating a configuration example of an imaging device according to a fourth embodiment. 12 is a processing flowchart according to Embodiment 4. 12 is a diagram illustrating a configuration example of an imaging device according to a fifth embodiment. FIG. 12 is a processing flowchart according to Embodiment 5.

Hereinafter, embodiments of the present invention will be described in detail based on the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.
<Embodiment 1>
First, Embodiment 1 according to the present invention will be described. FIG. 1 is a block diagram showing a configuration example of an imaging device according to this embodiment.

As shown in FIG. 1, the imaging apparatus of this embodiment includes an image sensor section 10, an image generation section 20, an image recognition section 30, a parameter determination section 40, a parameter setting section 50, a RAM 80, an image bus 120, and a CPU bus 170. has. The imaging device further includes a bridge 130, a DMAC (Direct Memory Access Controller) 70, a CPU (Central Processing Unit) 140, a ROM 150, and a RAM 160. The image sensor section 10, the image generation section 20, the image recognition section 30, the parameter determination section 40, the parameter setting section 50, and the DMAC 70 are connected to each other via an image bus 120. Further, the CPU 140, ROM 150, and RAM 160 are connected to each other via a CPU bus 170. Bridge 130 enables data transfer between image bus 120 and CPU bus 170.

The image sensor unit 10 includes an optical system, an image sensor, a driver circuit that controls the image sensor, an A/D conversion circuit that converts an image signal acquired by the image sensor into a digital signal, and an image sensor that converts the image signal acquired by the image sensor into a digital signal. It is equipped with a developing circuit that develops the signal as an image. The image sensor includes, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor.

Here, the image sensor unit 10 has a function of controlling imaging parameters that determine the operating characteristics of the imaging sensor. For example, a driver circuit within the image sensor unit 10 has the function of controlling the imaging parameters. The imaging parameters include exposure parameters, focus parameters, dynamic range parameters, gain parameters, frame rate parameters, imaging area parameters, resolution parameters, and the like. The exposure parameter is a parameter that controls the exposure time and the like in the image sensor. The focus parameter is a parameter that controls the operation of focus pixels and the like on the image sensor. The gain parameter is a parameter that controls the gain of the signal captured by the image sensor. The frame rate parameter is a parameter that controls the frame rate during image capturing in the image sensor. The imaging area parameter is a parameter that controls the area to be imaged on the image sensor. The resolution parameter is a parameter that controls the resolution of the image sensor. A specific example of the control function using the imaging parameters will be described later.

The image generation unit 20 has a function of generating a predicted image predicted as a future image from at least one piece of image data acquired by the image sensor unit 10. Details of the predicted image generation function in the image generation unit 20 will be described later.
The image recognition unit 30 has a function of performing predetermined image recognition processing on the predicted image generated by the image generation unit 20. In the first embodiment, a case will be described in which the image recognition unit 30 executes area division processing as image recognition processing. Details of the image recognition processing function (area division processing function) in the image recognition section 30 will be described later.

The parameter determination unit 40 determines the type and value of an imaging parameter for controlling the imaging sensor of the image sensor unit 10 based on the recognition result by the image recognition unit 30. Details of the imaging parameter determination function in the parameter determination unit 40 will be described later.
The parameter setting unit 50 sets the value of the imaging parameter determined by the parameter determining unit 40 for the image sensor unit 10. Details of the imaging parameter setting function in the parameter setting section 50 will be described later.

The DMAC 70 controls data transfer between the image sensor section 10, the image generation section 20, the image recognition section 30, the parameter determination section 40, the parameter setting section 50, and the CPU bus 170.
The ROM (Read Only Memory) 150 stores programs executed by the CPU 140, instructions that define the operation of the CPU 140, parameter data, weighting coefficients, and the like.
The CPU 140 reads the programs, instructions, parameter data, etc. from the ROM 150, and controls the overall operation of the imaging device and performs various calculations. Note that the CPU 140 can also access the RAM 80 on the image bus 120 via the bridge 130.
The RAM 160 is used as a work area for the CPU 140 when the CPU 140 controls the imaging device.

Next, the operation of the imaging apparatus having the configuration shown in FIG. 1 will be described.
FIG. 2 is a flowchart showing the operation of the imaging device according to this embodiment. Note that control of the entire processing flow shown in FIG. 2 is executed by the CPU 140 based on a preset program.

First, in step S180, the CPU 140 executes various initialization processes in the imaging device before starting the imaging process. For example, the CPU 140 transfers weighting coefficients necessary for the operations of the image recognition section 30 and the image generation section 20 from the ROM 150 to the RAM 160, and also sets various registers for defining the operations of the image recognition section 30 and the image generation section 20. conduct. Specifically, the CPU 140 sets predetermined values in a plurality of registers present in the control section in the image recognition section 30 and the image generation section 20. Similarly, the CPU 140 writes values necessary for operation into the image sensor section 10 as well. At this time, the value of the imaging parameter for controlling the imaging sensor of the image sensor unit 10 is set to a predetermined initial value determined in advance or an initial value set by the user.

Next, in step S181, the image sensor unit 10 performs imaging processing to acquire captured image data.
Subsequently, in step S182, the image generation unit 20 stores the image data acquired by the image sensor unit 10 in an internal frame buffer in units of frames. Here, in the case of the present embodiment, it is assumed that the image generation unit 20 starts generating and outputting one frame's worth of predicted images after receiving five frames' worth of image data. For this reason, the image generation unit 20 acquires image data frame by frame from the image sensor unit 10, and determines whether to start generating a predicted image in step S183. If the image generation unit 20 is able to acquire five frames of image data, it starts generating a predicted image in step S183, generates and outputs one frame of predicted image. On the other hand, if image data for five frames has not been acquired, the image generation unit 20 returns to the image acquisition process of step S181 by the image sensor unit 10 without starting generation of a predicted image. In this embodiment, the image generation unit 20 performs predetermined processing (such as feature extraction processing) on the image data already stored in the frame buffer before acquiring the next frame of image data. various processes necessary for generation). Details of the predicted image generation process in the image generation unit 20 will be described later.

When the predicted image generation process is started in step S183 and the predicted image is generated, the CPU 140 advances the process to step S184, which is performed by the image recognition unit 30.
Proceeding to step S184, the image recognition unit 30 performs image recognition processing using the predicted image generated by the image generation unit 20. In the case of this embodiment, the image recognition unit 30 executes image recognition processing and outputs image recognition results in units of divided regions obtained by dividing the predicted image generated by the image generation unit 20 into a plurality of regions. Then, the image recognition unit 30 outputs the coordinate values of the divided regions (coordinate values indicating the positions of pixels on the image) on the image (predicted image) as a result of the image recognition process.

Next, as an imaging parameter determination process in step S185, the parameter determining unit 40 determines the value of the aforementioned imaging parameter that determines the operating characteristics of the image sensor unit 10. In the case of the present embodiment, the parameter determination unit 40 uses information on the coordinate values of the divided regions on the image calculated as the recognition result by the image recognition process in step S184, and the predicted image generated in the image generation process in step S182. Based on this, the value of the imaging parameter is determined. A specific example of the method for determining the imaging parameter value will be described later.

Next, as imaging parameter setting processing in step S186, the parameter setting unit 50 sets the imaging parameter value determined in step S185 to the image sensor unit 10.
When the setting of the imaging parameters in step S186 is completed, the CPU 140 confirms the issuance of a termination notification (for example, an interrupt signal to the CPU 140 based on a termination instruction by the user) in step S187. If the end notification has not been issued, the CPU 140 returns the processing of the imaging device to the image acquisition processing of step S181, and causes the processing from step S181 to continue. On the other hand, upon confirming the issuance of the termination notification, the CPU 140 terminates the process of the flowchart of FIG. 2 .

Note that in the above description, the image generation unit 20 starts outputting one frame's worth of predicted images after five frames' worth of image data is input, but the present invention is not limited to this example. For example, the image generation unit 20 may start outputting a predicted image for one frame after inputting image data for any n frames in the past. In this case, "n" for any n frames in the past is a value of 1 or more. For example, when "n" is 1, the image generation unit 20 generates a predicted image based on image data for one past frame. Further, in the present embodiment, the image generation unit 20 outputs a predicted image for one frame, but it may generate and output predicted images for multiple frames. The predicted images for multiple frames may be predicted images for multiple frames in chronological order, or may be multiple predicted images corresponding to the same frame time. The plurality of predicted images corresponding to the same frame time may be, for example, predicted images of a plurality of patterns in the order of probabilities predicted from images acquired by imaging (for example, in order of highest prediction probability).

Next, the image generation process executed by the image generation section 20 described above will be described in detail using FIG. 3. FIG. 3 is a diagram schematically showing the configuration of a neural network (hereinafter referred to as NN) that implements image generation processing to generate a predicted image.
Regarding the NN (neural network) used in this embodiment, the one disclosed in detail in Reference 1 below can be applied, and the image generation unit 20 applies the NN to generate multiple frames. It has a function to generate predicted images based on image data. In this embodiment, it is assumed that the method disclosed in reference document 1 is applied as a method for realizing image generation processing, but the method is not limited to this method. For example, a method as disclosed in Reference 2 below may be used. Furthermore, the method for implementing the image generation process in this embodiment is not limited to the methods disclosed in Reference Document 1 and Reference Document 2, but may also use other methods.

Reference 1: Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting", arXiv 2015

Reference 2: William Lotter, Gabriel Kreiman, David Cox, "Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning", arXiv 2017

The image generation unit 20 temporarily stores the image data X _t input from the image sensor unit 10 in an internal frame buffer, and then inputs it to the convolution LSTM module as shown in FIG. The convolution LSTM module is a NN that executes processing including time-series information on time-series data. The convolutional LSTM module is an extension of the widely used LSTM (long short-term memory) to time-series image data. Reference 1 mentioned above provides a detailed explanation of convolutional LSTM.

Processing in convolution LSTM is expressed by the following equation (1). As described above, convolutional LSTM is a neural network (NN) that can obtain a desired output by learning in advance using time-series image data. In this embodiment, learning is performed in advance to generate the predicted image X ₅ of the next frame from the past five frames of time-series image data X ₀ to X ₄ based on equation (1). .

Here, the contents indicated by each symbol in formula (1) are as follows. Note that the Hadamard product is described in equation (1).
X: Input data C: Cell output data H: Hidden layer state i: Input gate f: Forgetting gate o: Output gate t: Time (frame)
W _xo : Input load matrix (output gate)
W _ho : State load matrix (output gate)
W _co : Cell output load matrix (output gate)
b _o : Bias (output gate)
W _xc : Input load matrix (cell output)
W _hc : State weight matrix (cell output)
b _c : Bias (cell output)
W _xf : Input load matrix (forgetting gate)
W _hf : state weight matrix (forgetting gate)
W _cf : Cell output load matrix (forgetting gate)
b _f : Bias (forgetting gate)
W _xi : Input load matrix (input gate)
W _hi : State weight matrix (input gate)
W _ci : Cell output load matrix (input gate)
b _i : Bias (input gate)
*: Convolution operation

Although the image generation unit 20 of this embodiment is configured using convolutional LSTM as described above, it is not limited to this. Regarding the method of generating a predicted image, various methods have been proposed in addition to the above-mentioned method, such as Non-Patent Document 2, and other methods may also be used.

In the configuration of the NN shown in FIG. 3, the convolution LSTM outputs a predicted image for one frame based on images for five past frames. In the example of FIG. 3 as well, the image generation process may output a predicted image for one frame based on images for any n frames in the past, or may output a predicted image for multiple frames. It may be something that is output. The image generated by the convolution LSTM is temporarily stored in a frame buffer and input to the image recognition process in step S184 in FIG.

Next, the image recognition process in step S184 executed by the image recognition unit 30 described above will be described in detail using FIG. 4. FIG. 4 is a diagram schematically showing a configuration example of a NN that implements predetermined image recognition processing on an input predicted image.
The NN shown in FIG. 4 is composed of a hierarchical convolutional NN (neural network) module 190 and a hierarchical deconvolutional NN module 191, and each layer is composed of convolution operation elements 401 to 408. The NN as shown in FIG. 4 is disclosed in detail in Reference 3 below. The image recognition unit 30 has a function of performing image recognition processing for each object in the image (processing to obtain divided regions as a recognition result) on input predicted image data using a NN as shown in FIG.

Reference 3: Hyeonwoo Noh, Seunghoon Hong, Bohyung Han, "Learning Deconvolution Network for Semantic Segmentation", ICCV 2015

The image recognition unit 30 once stores the input predicted image data in an internal frame buffer, and then inputs it to the hierarchical convolution NN module 190 shown in FIG. 4, and further inputs it to the hierarchical deconvolution NN module 191. A NN composed of a hierarchical convolutional neural network module 190 and a hierarchical deconvolutional neural network module 191 is widely used in image processing. In particular, an example in which area division is applied as in this embodiment is described in detail in Reference 3.

The processing in the hierarchical convolutional NN module 190 is expressed by the following equation (2).

Here, the contents indicated by each symbol in formula (2) are as follows.
x: Input data y: Output data w: Convolution filter v: Height of convolution filter h: Width of convolution filter m: Previous layer feature surface index n: Later layer feature surface index i: Previous layer calculation position y-coordinate j: Previous layer Layer calculation position x-coordinate i': Previous layer calculation position y-coordinate j': Subsequent layer calculation position x-coordinate k: Convolution filter y-coordinate l(t): Convolution filter x-coordinate

Furthermore, the hierarchical deconvolution NN module 191 is expressed by equation (3).

Here, the contents indicated by each symbol in formula (3) are as follows.
x: Input data (data expanded by inserting zero values into the feature plane of the previous layer)
y: Output data w: Deconvolution filter v: Height of deconvolution filter h: Width of deconvolution filter m: Previous layer feature surface index n: Later layer feature surface index i: Previous layer calculation position y-coordinate j: Previous layer Calculation position x-coordinate i': Previous layer calculation position y-coordinate j': Subsequent layer calculation position x-coordinate k: Deconvolution filter y-coordinate l(t): Deconvolution filter x-coordinate

In this embodiment, the NN composed of the hierarchical convolutional NN module 190 and the hierarchical deconvolutional NN module 191 is subjected to pre-learning using image data having training data regarding region segmentation, so that a desired result can be obtained. It becomes possible to obtain output. Regarding the learning process, a general learning method can be applied, and a detailed explanation of the learning method will be omitted.

Although the image recognition unit 30 in this embodiment is configured using a NN including the hierarchical convolutional NN module 190 and the hierarchical deconvolutional NN module 191 as described above, the present invention is limited to this example. It's not a thing. Regarding the method of realizing image recognition processing involving region division, various other methods have been proposed, including reference document 3, and other methods may also be used.

In the image recognition process of step S184 in FIG. 2 described above, the image recognition unit 30 executes region division processing on the input predicted image, and divides the divided regions defined for each object in the predicted image as a recognition result. Perform the calculation process. Specifically, the image recognition unit 30 outputs the coordinates of pixels corresponding to the boundary line 510 of each area of the object included in the predicted image 500, as shown in FIG.

Next, the imaging parameter determination process in step S185 executed by the parameter determination unit 40 described above will be described.
The parameter determination unit 40 first calculates the average brightness value in each region in the predicted image 500 shown in FIG. 5 based on the area information defined for each object in the predicted image and the data of the predicted image. Calculate the value.

Here, when determining, for example, an exposure parameter as an imaging parameter, the parameter determining unit 40 uses an LUT (lookup table) to calculate an appropriate exposure time value determined in advance corresponding to the average value range of brightness values in each area. ). Using this LUT, the parameter determination unit 40 determines an exposure parameter that determines the exposure time for each pixel circuit corresponding to each area of the image sensor of the image sensor unit 10, based on the average value of the calculated brightness values. .

Furthermore, when determining, for example, a dynamic range parameter as an imaging parameter, the parameter determination unit 40 determines the pixel circuits corresponding to each region of the image sensor of the image sensor unit 10 based on the average value of the luminance values calculated as described above. Determine the dynamic range for each. In this case, the parameter determination unit 40 holds, as an LUT, appropriate dynamic range values determined in advance corresponding to the average value range of luminance values in each region. Then, the parameter determination unit 40 uses the LUT to determine a dynamic range parameter that determines the dynamic range for each pixel circuit corresponding to each area of the image sensor of the image sensor unit 10 based on the average value of the calculated brightness values. Determine.

Further, when determining, for example, a gain parameter as an imaging parameter, the parameter determining unit 40 determines the output value of the image sensor of the image sensor unit 10 for each region based on the average value of the luminance values calculated as described above. A gain value is determined for each corresponding pixel. In this case, the parameter determination unit 40 holds as an LUT an appropriate gain value determined in advance corresponding to the range of the average value of brightness values in each region. Then, the parameter determination unit 40 uses the LUT to determine a gain parameter that determines a gain value for each pixel corresponding to each region of the image sensor of the image sensor unit 10 based on the average value of the calculated brightness values. do.

After that, in the imaging parameter determination process of step S185, the parameter determination unit 40 outputs the imaging parameters determined as described above to the imaging parameter setting process of step S186.
Note that the focus parameter, frame rate parameter, imaging area parameter, and resolution parameter will be explained in other embodiments to be described later.

Furthermore, although the present embodiment has been described using an example in which imaging parameters are determined for each divided region as described above, the present invention is not limited to this example. The imaging parameter determination process may be a process of determining imaging parameters determined for a specific area as imaging parameters to be used for the entire area. For example, in FIG. 5, the imaging parameters determined from the region of the car in the predicted image 500 may be used as the imaging parameters for the entire region. Furthermore, it is also possible to switch between the case where the imaging parameters are determined for each divided region as described above and the case where the imaging parameters to be used for the entire region are determined. For example, the imaging parameters to be used for the entire region may be determined only at a predetermined frame period, and the imaging parameters may be determined for each divided region at other times.

Next, the imaging parameter setting process in step S186 executed by the parameter setting unit 50 will be described.
The parameter setting unit 50 has an imaging parameter setting function that sets the imaging parameters input from the parameter determining unit 40 to the image sensor unit 10.
When an exposure parameter is input as an imaging parameter, the parameter setting unit 50 sets a register related to exposure time control for each pixel of the image sensor in the driver circuit based on the exposure time determined for each region by the exposure parameter. , set the exposure time.

Further, when a dynamic range parameter is input, the parameter setting unit 50 sets the dynamic range in the register related to dynamic range control for each pixel of the fixed image sensor of the driver circuit based on the dynamic range determined for each region. Set the value of

Further, when a gain parameter is input, the parameter setting unit 50 sets a gain value to a register related to gain value control for each pixel of the image sensor in the driver circuit based on the gain value determined for each region. do.

In the image sensor unit 10, the driver circuit controls the operation of the image sensor based on the values set in the register (the exposure time value, the dynamic range value, and the gain value) as described above.
Note that the image sensor in this embodiment has a configuration in which the exposure time, dynamic range, and gain value can be controlled for each pixel, and the exposure time, dynamic range, and gain value can be set. do. That is, the image sensor in this embodiment has a configuration in which the photodiode reset timing can be set for each pixel, thereby making it possible to control the exposure time for each pixel. Further, in the image sensor, by configuring an independent A/D conversion circuit for each pixel, it is possible to control the dynamic range and gain value for each pixel.

In addition, in the imaging parameter determination process, for example, if an imaging parameter determined from a specific area is determined as an imaging parameter to be used for the entire area, the parameter setting unit 50 sets the input imaging parameter to the entire area of the imaging sensor. Set for. Specifically, each imaging parameter is set in a register related to control of the entire area of the imaging sensor in the driver circuit.

As described above, the imaging apparatus of this embodiment generates a predicted image from past captured images, and controls imaging parameters based on the image recognition processing result for the predicted image. As a result, according to the imaging device of the embodiment, it is possible to prevent mismatches in imaging parameters that may occur when controlling imaging parameters for future images based on image recognition processing results estimated from past images. can. That is, according to this embodiment, it is possible to realize control of imaging parameters suitable for future images. For example, if the area of each object in an image changes from image frame to image frame, the area of each object obtained from past images may shift in future images, and as a result, the adjustment area of imaging parameters may also change. It is possible that it will shift. In contrast, in the imaging device of this embodiment, the imaging parameters for each object area are determined based on the image recognition processing results of the predicted image, so if the area for each object changes in each image frame within the image, However, it becomes possible to set more appropriate imaging parameters.

As described above, in the first embodiment, the imaging parameters of the image sensor are controlled based on the recognition result for the predicted image, but the method for generating the predicted image, the recognition processing method for the predicted image, and the imaging parameter based on the recognition processing result There are no particular limitations on the determination and setting method. In particular, regarding the determination and setting method of imaging parameters based on the recognition processing results, many methods based on the recognition processing results are disclosed as in Reference 4 below, and these are the recognition processing results for the predicted image in this embodiment. It is also applicable to methods based on .

Reference 4: Japanese Patent Application Publication No. 2011-130031

<Embodiment 2>
Next, a second embodiment will be described.
In the first embodiment, an example was given in which imaging parameters for divided regions (or regions of the entire image) are set based on the results of image recognition processing. On the other hand, in the case of the second embodiment, the image recognition unit 30 shown in FIG. 1 executes object recognition processing as the image recognition process, and the parameter determination unit 40 This embodiment differs from the first embodiment in that predetermined imaging parameters are determined for the object region. Hereinafter, only the parts that are different from Embodiment 1 will be explained, and the other parts will be omitted because they are the same as Embodiment 1.

In the imaging device of this embodiment, the image recognition unit 30 in FIG. 1 has an image recognition processing function that recognizes a specific object set in advance from the predicted image generated by the image generation unit 20. The specific object set in advance is a human body in this embodiment, but any other object that has been learned in advance, such as a car, a bicycle, or a traffic light, can be recognized.

FIG. 6 is a flowchart showing the operation of the imaging device in this embodiment.
In FIG. 6, the processes from step S180 to step S183 are the same as those described in the first embodiment, so detailed description thereof will be omitted.

In the case of the second embodiment, when the predicted image generation process is started in step S183 and the predicted image is generated, the CPU 140 advances the process to step S188, which is performed in the image recognition unit 30.
Proceeding to object recognition processing in step S188, the image recognition unit 30 performs human body recognition processing on the predicted image generated by the image generation unit 20. Then, the image recognition unit 30 outputs the coordinate values (upper left pixel position and lower right pixel position) on the image of the rectangular area estimated for the recognized human body as a result of the image recognition process. Details of the process for recognizing a human body from a predicted image will be described later. After the processing in step S188, the process advances to step S185.

Proceeding to the imaging parameter determination process in step S185, the parameter determination unit 40 uses information on the coordinate values of the rectangular region of the human body obtained in the object recognition process in step S188 and the predicted image generated in the image generation process in step S182. Based on this, the imaging parameter values are determined. Details of the method for determining the imaging parameter values will be described later.

Next, the image recognition process in step S188 executed by the image recognition unit 30 of the second embodiment will be described in detail using FIG. 7. FIG. 7 is a diagram schematically showing a configuration example of a NN that implements a predetermined image recognition process on an input predicted image (X).

The image recognition process in this embodiment is configured by a hierarchical convolutional neural network that is widely applied as a recognition processing technology, and includes the hierarchical convolutional neural network module 190 in FIG. 4 included in the neural network described in the first embodiment above. Executes similar arithmetic processing. Each layer of the hierarchical convolution NN module is composed of convolution operation elements 701 to 704.

The image recognition unit 30 temporarily stores the input predicted image data in an internal frame buffer, and then inputs it to a hierarchical convolutional neural network module to perform a process of recognizing a human body. Here, the processing in the hierarchical convolutional NN module is executed by sequentially scanning a predetermined region within an image, similar to general object recognition processing.

Processing in the hierarchical convolutional NN is expressed by equation (2) similarly to the processing in the hierarchical convolutional NN in the first embodiment. In the case of the second embodiment, it is possible to recognize a human body in an image by performing prior learning using image data having training data regarding the position and size of the human body on the image.

Although the image recognition unit 30 in this embodiment is configured using a hierarchical convolutional neural network module, the present invention is not limited to this, and various other methods have been proposed for recognizing a specific object. However, other methods may also be used.

In the object recognition process of step S188 in FIG. 6, the image recognition unit 30 performs a process of recognizing a human body on the input predicted image, and estimates the position and size of the human body in the predicted image. Specifically, as shown in FIG. pixel position 812).

In addition to the human body recognition process, the image recognition unit 30 can also perform a human body posture recognition process as the image recognition process in step S188. The human body posture recognition process can be performed using the above-described hierarchical convolutional neural network module. In this case, in the object recognition process of step S188, the image recognition unit 30 calculates the position of each part of the human body from the predicted image 900 as shown in FIG. 9, in addition to the coordinate values on the image of the rectangular area estimated as described above. Each coordinate value indicating 910 to 915 is calculated. In the example of FIG. 9, the coordinate values of the human body's head center position 911, torso center position 910, left hand position 912, right hand position 913, left foot position 914, and right foot position 915 of the predicted image 900 are output. . Note that the object to be recognized is not limited to the posture of a human body, and object posture recognition processing for recognizing the posture of any object may be performed. For example, when recognizing the posture of a car, each part such as the headlamp position, tire position, roof position, etc. can be recognized.

Next, the imaging parameter determination process in step S185 executed by the parameter determination unit 40 in the second embodiment will be described.
The parameter determination unit 40 uses information on the coordinate values (upper left pixel position 811 and lower right pixel position 812) on the image of the rectangular area 810 estimated for the human body in the predicted image 800 of FIG. 8, and the data of the predicted image 800. Based on this, the average value of the brightness values in each area is calculated.

When determining the exposure parameters, the parameter determination unit 40 holds, as a LUT, appropriate exposure times determined in advance corresponding to the average value range of the brightness values in each region. Then, the parameter determining unit 40 uses the LUT to determine the exposure time for each pixel circuit corresponding to each area of the image sensor of the image sensor unit 10 based on the average value of the calculated brightness values. Determine.

Further, when determining the dynamic range parameter, the parameter determination unit 40 determines the dynamic range of each pixel circuit corresponding to each area of the image sensor of the image sensor unit 10 based on the average value of the luminance values calculated as described above. Determine. In this case, the parameter determining unit 40 maintains an appropriate dynamic range determined in advance corresponding to the average value range of luminance values in each area as an LUT. Then, the parameter determination unit 40 uses the LUT to determine the dynamic range for each pixel circuit corresponding to each area of the image sensor of the image sensor unit 10 based on the average value of the calculated luminance values. Determine parameters.

Further, when determining the gain parameter, the parameter determination unit 40 determines the output value of the image sensor of the image sensor unit 10 based on the average value of the luminance values calculated as described above. Determine the gain value for each time. In this case, the parameter determination unit 40 holds as an LUT an appropriate gain value determined in advance corresponding to the range of the average value of brightness values in each region. The parameter determination unit 40 determines a gain parameter by determining a gain value for each pixel corresponding to each area of the image sensor of the image sensor unit 10 based on the calculated average value of the luminance values using the LUT. .

Also in the second embodiment, the parameter determining unit 40 sends each imaging parameter determined as described above to the parameter setting unit 50 as the imaging parameter setting process in step S186.

In the case of the second embodiment, the parameter determining unit 40 determines the imaging area parameter ( It is also possible to determine the area to be imaged by the image sensor. When the imaging area parameters are set by the parameter setting unit 50, the image sensor unit 10 acquires only images of the area defined by the imaging area parameters, as will be described later.

Further, in the second embodiment, the parameter determining unit 40 calculates the pixel position to be focused based on the coordinate values (upper left pixel position and lower right pixel position) on the image of the rectangular area estimated for the human body. You can also do that. For example, the parameter determination unit 40 calculates the center position of the human head, for example, the left and right center positions 1010 that are 1/10 from the top of the rectangular area 1011 estimated for the human body as shown in FIG. Based on this, determine the focus parameter that determines the focus position. Then, when the focus parameter is set by the parameter setting unit 50, the image sensor unit 10 can perform focusing at the focus position set by the focus parameter.

In the second embodiment, when performing human body posture recognition processing in addition to human body recognition processing, the image recognition unit 30 recognizes each input human body based on a preset part or a part specified by the user. It is also possible to select one of the coordinate values of the part. In this case, the parameter determination unit 40 can also determine a preset region or a region specified by the user (for example, the center position of the human body head) as the focus parameter for determining the focus position.

In the case of the second embodiment as well, the parameter setting section 50 sets the imaging parameters determined by the parameter determining section 40 for the image sensor section 10 as the imaging parameter setting process of step S186 in FIG.

For example, when setting imaging area parameters, the parameter setting unit 50 sets the registers related to imaging area control of the imaging sensor in the driver circuit of the image sensor unit 10 based on the imaging area parameters determined by the parameter determining unit 40. Set the imaging region (ROI).

Further, when setting a focus parameter, the parameter setting unit 50 sets the focus parameter in a register related to autofocus control of the image sensor of the driver circuit based on the focus parameter determined by the parameter determination unit 40.
Note that imaging parameter setting processing other than these executed by the parameter setting unit 50 is the same as that in the first embodiment described above, so a description thereof will be omitted.

In the image sensor unit 10, the driver circuit controls the operation of the image sensor based on the value set in the register, as in the first embodiment described above.
Note that the image sensor in this embodiment has a configuration that allows the exposure time, dynamic range, and gain value to be controlled for each pixel, as described above. Furthermore, the image sensor in this embodiment has a function of capturing an image of only the set imaging area based on the set imaging area. For example, as shown in FIG. 11, the image sensor in Embodiment 2 images only the human body region (area 1110 surrounded by a black rectangle in FIG. 11) recognized from the predicted image based on the control of the driver circuit. Obtain as data. Furthermore, the image sensor of this embodiment has a function of realizing autofocus processing including control of the optical system based on the set focus position.

As described above, the imaging apparatus of the second embodiment controls imaging parameters for future images based on image recognition results for predicted images generated from past images. Thereby, as in the first embodiment described above, it is possible to prevent mismatching of imaging parameters that may occur when controlling imaging parameters for future images based on recognition results for past images. When performing image recognition of a person as in Embodiment 2, if the position and size of an object such as a human body changes from image frame to image frame, the area of the human body etc. estimated from past images will shift in future images. There is a possibility that the adjustment range of the imaging parameters may also shift. In addition, if the size and posture of a human body etc. in an image change, the position of a specific part based on the area estimated from past images and the position of a specific part based on the posture of the object will also shift in future images. there is a possibility. In contrast, in this embodiment, the area and posture of the human body, etc. in future images can be estimated based on the predicted image, so even if the position, size, and posture of the human body, etc. change from image frame to image frame, the area and posture can be estimated. It becomes possible to set appropriate imaging parameters for the posture.

Note that in this embodiment, the imaging parameters of the image sensor unit are controlled based on the recognition results for the predicted image, but as in the first embodiment, the method for generating the predicted image, the recognition processing method for the predicted image, and the recognition processing method for the predicted image are controlled based on the recognition processing results. The method for determining and setting imaging parameters is not limited. Regarding the imaging parameter determination and setting method based on the recognition processing result, various methods based on the recognition processing result can also be applied to the method based on the recognition processing result of the measured image in this embodiment.

<Embodiment 3>
Next, Embodiment 3 will be described. In the third embodiment, the image recognition unit 30 executes scene recognition processing to recognize what kind of scene (scene) it is, and the parameter determination unit 40 executes scene recognition processing according to the scene recognized in the scene recognition processing. This embodiment differs from the second embodiment in that predetermined imaging parameters are determined. Hereinafter, only the parts that are different from the second embodiment will be explained, and the other configurations and the like will be omitted because they are the same as the second embodiment.

In the imaging device of the second embodiment, the image recognition unit 30 in FIG. 1 recognizes a specific scene set in advance from the predicted image generated by the image generation unit 20. In the case of this embodiment, the specific scene set in advance is, for example, "a pedestrian running out", but there are also other scenes such as "a car running out", "a bicycle running out", or "a pedestrian falling down", etc. It may be any scene that has been learned. The image recognition unit 30 performs image recognition processing to recognize these specific scenes.

FIG. 12 is a flowchart showing the operation of the imaging device in this embodiment.
Since the processing from step S180 to step S183 and the processing from step S185 to step S187 in FIG. 12 are the same as those in the second embodiment, detailed description thereof will be omitted.

In the case of the third embodiment, when the predicted image generation process is started in step S183 and the predicted image is generated, the CPU 140 advances the process to step S189, which is performed by the image recognition unit 30.
Proceeding to the scene recognition process in step S189, the image recognition unit 30 performs a “pedestrian jumping out” scene recognition process on the predicted image generated by the image generation unit 20. As a result of the image recognition process, the image recognition unit 30 sets a true/false flag for the image recognition of the "pedestrian jumping out" scene and a rectangular area estimated for the recognized scene on the image. The coordinate values (upper left pixel position and lower right pixel position) are output. After the processing in step S189, the process advances to step S185.

Proceeding to the imaging parameter determination process in step S185, the parameter determination unit 40 uses the information (true/false flag) of the recognition result of the specific scene calculated in the scene recognition process in step S189. Determine imaging parameter values. The imaging parameter value in this embodiment is the value of the frame rate of the image acquired by the image sensor unit 10. Details of the method for determining the frame rate parameter value will be described later.

Next, the scene recognition process executed by the image recognition section 30 described above will be described in detail.
The scene recognition process in this embodiment is performed by a hierarchical convolutional neural network that is widely applied as a recognition processing technique, and is executed by the same calculation process as the hierarchical convolutional neural network shown in FIG. 7 described in the second embodiment.

The image recognition unit 30 temporarily stores the input predicted image data in a frame buffer, and then inputs the data to a hierarchical convolutional neural network module to execute processing for recognizing a "pedestrian jumping out" scene. Processing in the hierarchical convolutional NN module is expressed by equation (2) similarly to the processing in the hierarchical convolutional NN in the second embodiment. In the case of Embodiment 3, it is possible to recognize a "pedestrian jumping out" scene in an image by learning in advance using image data that has training data regarding the "pedestrian jumping out" scene. .

Although the image recognition unit 30 in this embodiment is configured using a hierarchical convolutional neural network module, it is not limited to this example, and various other methods may be used to recognize a specific scene. It's okay to be hit.

As described above, in the scene recognition process of step S189, the image recognition unit 30 executes a process of recognizing a "pedestrian jumping out" scene on the input predicted image, and determines that the scene exists in the predicted image. Estimate whether. As shown in FIG. 13, the image recognition unit 30 includes a true/false flag indicating whether a "pedestrian jumping out" scene exists in the predicted image 1300, and a rectangular area 1310 estimated from the recognized scene on the image. The coordinate values (upper left pixel position 1311 and lower right pixel position 1312) are output.

Next, the imaging parameter determination process executed by the parameter determination unit 40 will be described.
The parameter determination unit 40 uses a true/false flag estimated regarding the presence or absence of a "pedestrian jumping out" scene in the predicted image, and information on coordinate values on the image of a rectangular area estimated for the recognized scene. Based on this, the frame rate of the image acquired by the image sensor unit 10 is determined.

The parameter determination unit 40 holds the frame rate of the image determined in advance in accordance with the presence or absence of the "pedestrian jumping out" scene (true/false flag) as an LUT, and determines the image frame rate based on the presence or absence of the scene. The frame rate of the image sensor of the sensor unit 10 is determined. For example, when a "pedestrian jumping out" scene is recognized in the predicted image (a true flag is input), the parameter determining unit 40 changes the frame rate (60 fps) during normal operation of the image sensor unit 10 to 240 fps. decide what to do. Then, the parameter determining unit 40 determines the frame rate parameter after the change.

The parameter setting unit 50 sets the imaging parameters (frame rate parameters) determined as described above in the image sensor unit 10 in the imaging parameter setting process of step S186.
Note that in this embodiment, the imaging parameter determined in the imaging parameter determination process is the frame rate of the imaging sensor of the image sensor unit 10, but the present invention is not limited to this. For example, based on the information on the coordinate values on the image of the rectangular area estimated for the "pedestrian jumping out" scene, pixel circuits corresponding to each area of the image sensor of the image sensor unit 10, as in the second embodiment. The exposure time, dynamic range and gain value for each may be determined.
The imaging parameter setting process executed by the parameter setting unit in step S186 in FIG. 12 is the same as that in the second embodiment, and therefore a description thereof will be omitted.

In the case of scene recognition processing, the presence or absence of a scene estimated from past images and the scene recognition area may be different from the presence or absence of a scene and the scene recognition area in future images, and as a result, the imaging parameters may be different. Adjustments may no longer fit the actual scene. In contrast, in the case of this embodiment, the presence or absence of a specific scene and the area of the scene are determined based on the predicted image, so even if the scene changes from image frame to image frame, the presence or absence of a more appropriate scene and the area of the scene are determined. In contrast, it becomes possible to set imaging parameters. Further, the parameter determination unit 40 may perform parameter determination processing such as controlling the value of a specific imaging parameter among the imaging parameters depending on the recognized scene. For example, when a "pedestrian jumping out" scene is recognized, in addition to determining the frame rate parameter to increase the frame rate as described above, it is also possible to determine the imaging area parameter to read out the area of the pedestrian. You can also do it. In addition, a resolution parameter may be determined to control the resolution of the image sensor to be increased when a "pedestrian jumping out" scene is recognized.

Note that in the third embodiment, as described above, the method for generating a predicted image, the recognition processing method for the predicted image, and the method for determining and setting imaging parameters based on the recognition processing result are not limited. In particular, as in the second embodiment, various methods can be applied to determine and set imaging parameters based on recognition processing results, and as methods based on recognition processing results for predicted images.

<Embodiment 4>
Next, Embodiment 4 will be described.
FIG. 14 is a block diagram showing a configuration example of an imaging device according to the fourth embodiment. As can be seen from the comparison with the configuration example of FIG. 1, the imaging device of the fourth embodiment shown in FIG. 14 has a vector calculation unit 200 added to the configuration of FIG.

Here, the vector calculation unit 200 has a motion vector detection function that detects (calculates) a motion vector from an image captured by the image sensor unit 10. The image generation unit 210 of the fourth embodiment has a function of generating a predicted image based on input image data and a motion vector detected (calculated) from the image data by the vector calculation unit 200. Detailed explanations of each unit other than the vector calculation unit 200 and the image generation unit 210 will be omitted as they have the same functions as those in the first embodiment.

Next, the operation of the imaging device according to the fourth embodiment having the configuration shown in FIG. 14 will be described. FIG. 15 is a flowchart showing the operation of the imaging device according to this embodiment.
Steps S180 and S181 in FIG. 15, and each process from step S183 to step S187 are the same as those in the first embodiment, so detailed explanations will be omitted.
In the case of the fourth embodiment, after the process of step S181, the CPU 140 advances the process to step S201 performed by the vector calculation unit 200.

Proceeding to step S201, the vector calculation unit 200 calculates a motion vector based on two temporally consecutive frames of image data acquired by the image sensor unit 10 as a motion vector calculation process. Execute. Then, the vector calculation unit 200 outputs a motion vector (including information on the direction of movement and the magnitude of movement) at each pixel position of the image as a result of the motion vector calculation process.

Next, proceeding to step S202, the image generation unit 210 first stores the image data acquired by the image sensor unit 10 and the motion vector data calculated by the vector calculation unit 200 in an internal frame buffer in units of frames. do. Then, the image generation unit 21 generates a predicted image based on the image data and motion vector data.

The image generation process in step S202 executed by the image generation unit 210 will be described below. The image generation unit 210 is different from the first to third embodiments described above in that it generates a predicted image based on the image data input from the image sensor unit 10 and the motion vector data calculated by the vector calculation unit 200. are different.

The image generation process in the image generation unit 210 of the fourth embodiment is easily realized by expanding the input terminal of the NN module that performs the convolution LSTM described in the first to third embodiments. The calculations and learning processing performed by the image generation unit 210 are the same as in the first to third embodiments. Note that in this embodiment, it is assumed that the method disclosed in the above-mentioned Reference 1 is used as a method for realizing image generation processing, but the method is not limited to this, as in Embodiments 1 to 3. The same is true.

In the case of the fourth embodiment, a predicted image is generated from past images and motion vectors, and imaging parameters for future images are controlled based on recognition results for the predicted image. In the case of the fourth embodiment as well, it is possible to prevent mismatches in imaging parameters that may occur when imaging parameters for future images are controlled based on recognition results generated from past images.

In this embodiment, an example in which a vector calculation unit (motion vector calculation processing) is further added to the configuration of Embodiment 1 has been described, but the vector calculation unit (motion vector calculation processing) is also added to the configuration of Embodiment 2 and 3. A calculation unit (motion vector calculation processing) may be added. Detailed explanations of these examples will be omitted as they can be easily expanded from this embodiment.

Furthermore, in this embodiment, an example has been described in which motion vector data is calculated by the vector calculation unit 200 and provided as input data to the image generation unit 210, but the image generation unit 210 predicts motion vector data in a future image ( It is also possible to calculate Reference 5 discloses a method for calculating predicted motion vector data for a future image, and it is also possible to implement the image generation unit in this embodiment using this method.

Reference 5: Xiaodan Liang, Lisa Lee, Wei Dai, Eric P. Xing, "Dual Motion GAN for Future-Flow Embedded Video Prediction", ICCV 2017

Further, as a modification of the present embodiment, it is also possible to input information on the predicted motion vector together with the predicted image to the image recognition unit 30. This can be easily achieved by further expanding the input terminal of the image recognition unit 30 described in Embodiments 1 to 3 to input predicted motion vector information.

According to this embodiment, by generating a predicted image using motion vector data in addition to image data, the predicted image accuracy of images with movement is improved, so that imaging parameters that are more suitable for future images are improved. Settings are now possible.

<Embodiment 5>
Next, Embodiment 5 will be described. FIG. 16 is a block diagram showing a configuration example of an imaging device according to the fifth embodiment.
As can be seen by comparison with the configuration example of FIG. 1, the imaging device of the fifth embodiment shown in FIG. 16 is configured by providing a plurality of image recognition units 30 and RAMs 80 in the first to third embodiments. The image recognition unit 30A in FIG. 16 is in charge of the image recognition process (area division process) described in the first embodiment, and the image recognition unit 30B is in charge of the image recognition process (object recognition process) in the second embodiment. 30C is in charge of image recognition processing (scene recognition processing) in the third embodiment.

Further, FIG. 17 shows a flowchart showing the operation of the imaging apparatus according to the fifth embodiment. In the case of the flowchart shown in FIG. 17, all of the region division processing in step S184, the object recognition processing in step S188, and the scene recognition processing in step S189, which were explained as image recognition processing in Embodiments 1 to 3, are included.

In FIG. 17, when a predicted image is generated in step S183, the CPU 140 performs image recognition processing (area division processing) in step S184, image recognition processing (object recognition processing) in step S188, and image recognition processing (scene recognition processing) in step S189. The process proceeds to (recognition processing).

In step S184, the image recognition unit 30A executes image recognition processing (region division processing), in step S183, the image recognition unit 30B executes image recognition processing (object recognition processing), and in step S189, the image recognition unit 30C performs image recognition. Execute processing (scene recognition processing). After steps S184, S188, and S189, the process proceeds to step S185 performed by the parameter determining section 40. The image recognition processes in step S184, step S188, and step S189 are the same as those described in each of the above-described embodiments, so the description thereof will be omitted. Note that in the fifth embodiment, as in the second embodiment, in addition to the object recognition process in step S188, it is possible to further perform object posture recognition processing.

Further, the processing performed in the imaging parameter determination processing in step S185 and the imaging parameter setting processing in step S186 in FIG. A combination of these processes may also be used. For example, regarding the dynamic range of a background region (for example, a sky region), values of imaging parameters may be determined by the processing described in the first embodiment. Further, for example, regarding the dynamic range of the human body region, the values of the imaging parameters may be determined by the processing described in the second embodiment. Furthermore, regarding the frame rate, the process of determining the value of the imaging parameter using the method described in the third embodiment may be used. Furthermore, in the imaging device of the fifth embodiment, it is also possible to add a vector calculation unit and perform processing based on motion vector data as in the fourth embodiment.

In this manner, in the imaging device according to the fifth embodiment, the image generation unit 20 first generates a predicted image, and the image recognition units 30A, 30B, and 30C each perform image recognition processing on the predicted image. Then, the parameter determination unit 40 determines imaging parameters according to the execution results of those image recognition processes. Furthermore, in the case of the fifth embodiment, multiple types of processing are executed as image recognition processing. Furthermore, in the case of the fifth embodiment, once a predicted image is generated, the image recognition processing applied to the generated predicted image is not limited as in the above-described first to fourth embodiments. Therefore, in the case of Embodiment 5, it is also possible to execute processes other than the image recognition processes in Embodiments 1 to 4 and even in this embodiment. For example, as a recognition process different from the image recognition process described above, it is also possible to perform a moving behavior detection process for detecting abnormal behavior of a person, a car, etc. Also in the fifth embodiment, the imaging parameters described in the first to fourth embodiments or other imaging parameters can be controlled in accordance with the image recognition processing to be executed. In this case as well, the imaging parameters described in Embodiments 1 to 4 are merely examples, and the types of imaging parameters are not particularly limited in the present invention.

The configuration of the image processing device of each embodiment described above or the processing of each flowchart may be realized by a hardware configuration, or may be realized by a software configuration by, for example, a CPU executing a program according to the present embodiment. good. Further, a part may be realized by a hardware configuration and the rest by a software configuration. The program for the software configuration may not only be prepared in advance, but may also be obtained from a recording medium such as an external memory (not shown) or via a network (not shown).

A program for realizing one or more functions in the control processing according to the present invention can be supplied to a system or device via a network or a storage medium, and can be read and executed by one or more processors of a computer in the system or device. It is possible to achieve this by doing so.
The embodiments described above are merely examples of implementation of the present invention, and the technical scope of the present invention should not be construed as limited by these embodiments. That is, the present invention can be implemented in various forms without departing from its technical idea or main features.

10: Image sensor section, 20: Image generation section, 30: Image recognition section, 40: Parameter determination section, 50: Parameter setting section, 80, 160: RAM, 140: CPU, 150: ROM

Claims (11)

an imaging means that is equipped with an imaging sensor that can set imaging parameters for each pixel or region of the image, and that captures an image of a moving object ;
an image generation unit that generates a predicted image that is a prediction of an image that will be captured after the object moves by the imaging unit based on the acquired image;
image recognition means for performing image recognition processing on the predicted image generated by the image generation means ;
Parameter determining means for determining the imaging parameters of the image sensor for each pixel or region based on the position of the object in the predicted image obtained by the image recognition process;
parameter setting means for setting the determined imaging parameters for each pixel or region for the imaging sensor ;
has
The image capturing device is characterized in that the image capturing means acquires an image according to parameters set by the parameter setting means . The imaging device according to claim 1 , wherein the image recognition means executes object recognition processing to recognize a predetermined object in the predicted image. The image recognition means executes scene recognition processing to recognize a specific scene in which the object makes a specific movement based on the predicted image ,
The parameter determining means determines the imaging parameters of the imaging sensor for each pixel or region based on the position of the object in the predicted image in the specific scene recognized by the scene recognition process. The imaging device according to claim 1 or 2. The imaging apparatus according to any one of claims 1 to 3 , wherein the image recognition means executes a plurality of different image recognition processes. The imaging device according to any one of claims 1 to 4 , wherein the image generation unit generates the predicted image from a plurality of images acquired by the imaging unit. The imaging device according to any one of claims 1 to 5 , wherein the image generation means generates a plurality of predicted images. The imaging device according to any one of claims 1 to 6 , wherein the image generation means generates the predicted image using a neural network. 8. The imaging apparatus according to claim 1 , wherein the image recognition means executes the image recognition process using a neural network. The imaging parameter determined by the parameter determining means is one or more of an exposure parameter, a focus parameter, a dynamic range parameter, a gain parameter, a frame rate parameter, an imaging area parameter, and a resolution parameter. The imaging device according to any one of claims 1 to 8 . an imaging step of acquiring an image of a moving object using an imaging sensor that can set imaging parameters for each pixel or region of the image ;
an image generation step of generating a predicted image that is a prediction of an image to be captured by the image sensor after the object moves based on the acquired image;
an image recognition step of performing image recognition processing on the predicted image generated in the image generation step ;
a parameter determining step of determining the imaging parameters of the image sensor for each pixel or region based on the position of the object in the predicted image obtained by the image recognition process;
a parameter setting step of setting the determined imaging parameters for each pixel or region for the imaging sensor ;
has
A method for controlling an imaging apparatus, characterized in that, in the imaging step, an image is acquired according to the parameters set in the parameter setting step . A program for causing a computer to function as each means included in the imaging device according to any one of claims 1 to 9 .

Images