JP2021061546A - Imaging apparatus, control method of the same, and program - Google Patents

Abstract

To make it possible to determine better adapted imaging parameters for an image to be captured based on the imaging parameters.SOLUTION: An imaging apparatus includes image acquisition means (10) for acquiring an image through image capturing, image generation means (20) for generating a prediction image based on the acquired image, image recognition means (30) for performing image recognition on the prediction image, parameter determination means (40) for determining imaging parameters based on the results of image recognition, and parameter setting means (50) for setting the determined imaging parameters to the image acquisition means.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for controlling imaging parameters used during imaging.

As a technique for controlling an imaging parameter that makes it possible to acquire a desired image in an imaging device, there is a wide range of techniques for executing recognition processing on an image and controlling the imaging parameter based on the obtained recognition result. Proposed. Patent Document 1 describes the vertical direction, horizontal direction, rotation direction, etc. of the camera so that the face of the person can be easily confirmed based on the face recognition result of the person using the image captured by the camera of the monitoring device or the like. A technique for determining at least one imaging parameter of an imaging direction and an angle of view is disclosed.

Japanese Unexamined Patent Publication No. 2014-64083

In the case of the imaging parameter control technology based on the image recognition result described above, the image to which the recognition process is applied is an image that has already been acquired. Therefore, when the imaging parameters are controlled based on the image recognition result, the position on the image of the recognition target and the brightness data at the pixel level have already changed in the newly acquired image after reflecting the imaging parameters. There is a possibility that it is. In this case, the value of the imaging parameter controlled to acquire a desired image such as a face image may become an inappropriate value for an image newly imaged based on the imaging parameter. It is possible. That is, the imaging parameters may not be compatible with the newly captured image.

Therefore, an object of the present invention is to make it possible to determine a more suitable imaging parameter for an image captured based on the imaging parameter.

The image pickup apparatus of the present invention includes an image acquisition means for acquiring an image by imaging, an image generation means for generating a predicted image based on the acquired image, and an image recognition means for executing image recognition on the predicted image. It is characterized by having a parameter determining means for determining an imaging parameter based on the result of the image recognition, and a parameter setting means for setting the determined imaging parameter with respect to the image acquiring means.

According to the present invention, it is possible to determine a more suitable imaging parameter for an image captured based on the imaging parameter.

It is a figure which shows the structural example of the image pickup apparatus which concerns on Embodiment 1. FIG. It is a processing flowchart which concerns on Embodiment 1. It is a figure which shows the configuration example which realizes the prediction image generation processing which concerns on Embodiment 1. It is a figure which shows the configuration example which realizes the image recognition processing which concerns on Embodiment 1. It is explanatory drawing of the image recognition processing which concerns on Embodiment 1. FIG. It is a processing flowchart which concerns on Embodiment 2. It is a figure which shows the configuration example which realizes the image recognition processing which concerns on Embodiment 2. It is explanatory drawing of the image recognition processing (object recognition processing) which concerns on Embodiment 2. It is explanatory drawing of the image recognition processing (object posture recognition processing) which concerns on Embodiment 2. It is explanatory drawing of the calculation method of the human body head center position which concerns on Embodiment 2. It is explanatory drawing of the imaging region which concerns on Embodiment 2. It is a processing flowchart which concerns on Embodiment 3. It is explanatory drawing of the image recognition processing which concerns on Embodiment 3. It is a figure which shows the structural example of the image pickup apparatus which concerns on Embodiment 4. FIG. It is a processing flowchart which concerns on Embodiment 4. It is a figure which shows the structural example of the image pickup apparatus which concerns on Embodiment 5. It is a processing flowchart which concerns on Embodiment 5.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.
<Embodiment 1>
First, the first embodiment according to the present invention will be described. FIG. 1 is a block diagram showing a configuration example of an imaging device according to the present embodiment.

As shown in FIG. 1, the image pickup apparatus of this embodiment includes an image sensor unit 10, an image generation unit 20, an image recognition unit 30, a parameter determination unit 40, a parameter setting unit 50, a RAM 80, an image bus 120, and a CPU bus 170. Has. Further, the image pickup apparatus includes a bridge 130, a DMAC (Direct Memory Access Controller) 70, a CPU (Central Processing Unit) 140, a ROM 150, and a RAM 160. The image sensor unit 10, the image generation unit 20, the image recognition unit 30, the parameter determination unit 40, the parameter setting unit 50, and the DMAC 70 are connected to each other via the image bus 120. The CPU 140, ROM 150, and RAM 160 are connected to each other via the CPU bus 170. The bridge 130 enables data transfer between the image bus 120 and the CPU bus 170.

The image sensor unit 10 includes an optical system, an imaging sensor, a driver circuit that controls the imaging sensor, an A / D conversion circuit that converts an imaging signal acquired by the imaging sensor into a digital signal, and an imaging that is converted into a digital signal. It is equipped with a development circuit that develops the signal as an image. The imaging sensor is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor.

Here, the image sensor unit 10 has a function of controlling an imaging parameter that determines the operating characteristics of the imaging sensor. For example, a driver circuit in the image sensor unit 10 is responsible for the function of controlling the imaging parameters. The imaging parameters include exposure parameters, focus parameters, dynac range parameters, gain parameters, frame rate parameters, imaging area parameters, resolution parameters, and the like. The exposure parameter is a parameter that controls the exposure time of the image sensor. The focus parameter is a parameter that controls the operation of the focus pixel or the like on the image sensor. The gain parameter is a parameter that controls the gain of the signal captured by the image pickup sensor. The frame rate parameter is a parameter that controls the frame rate at the time of imaging in the imaging sensor. The image pickup area parameter is a parameter that controls the area to be imaged on the image pickup sensor. The resolution parameter is a parameter that controls the resolution of the image pickup sensor. Specific examples of the control function based on the imaging parameters will be described later.

The image generation unit 20 has a function of generating a predicted image predicted as a future image from at least one image data acquired by the image sensor unit 10. The details of the predicted image generation function in the image generation unit 20 will be described later.
The image recognition unit 30 has a function of executing a predetermined image recognition process on the predicted image generated by the image generation unit 20. In the first embodiment, a case where the image recognition unit 30 executes the area division process as the image recognition process will be described. The details of the image recognition processing function (area division processing function) in the image recognition unit 30 will be described later.

The parameter determination unit 40 determines the type and value of the image pickup parameter for controlling the image sensor of the image sensor unit 10 based on the recognition result by the image recognition unit 30. The details of the imaging parameter determination function in the parameter determination unit 40 will be described later.
The parameter setting unit 50 sets the value of the imaging parameter determined by the parameter determination unit 40 with respect to the image sensor unit 10. The details of the imaging parameter setting function in the parameter setting unit 50 will be described later.

The DMAC 70 controls data transfer between the image sensor unit 10, the image generation unit 20, the image recognition unit 30, the parameter determination unit 40, the parameter setting unit 50, and the CPU bus 170.
The ROM (Read Only Memory) 150 stores a program executed by the CPU 140, an instruction defining the operation of the CPU 140, parameter data, a weighting coefficient, and the like.
The CPU 140 controls the overall operation of the image pickup apparatus and performs various calculations while reading the programs, instructions, parameter data, and the like from the ROM 150. The CPU 140 can also access the RAM 80 on the image bus 120 via the bridge 130.
The RAM 160 is used as a work area of the CPU 140 when the CPU 140 controls an imaging device or the like.

Next, the operation of the image pickup apparatus having the configuration shown in FIG. 1 will be described.
FIG. 2 is a flowchart showing the operation of the image pickup apparatus according to the present embodiment. The control of the entire processing flow shown in FIG. 2 is executed by the CPU 140 based on a preset program.

First, in step S180, the CPU 140 executes various initialization processes in the image pickup apparatus prior to the start of the image pickup process. For example, the CPU 140 transfers the weighting coefficients necessary for the operation of the image recognition unit 30 and the image generation unit 20 from the ROM 150 to the RAM 160, and sets various registers for defining the operations of the image recognition unit 30 and the image generation unit 20. Do. Specifically, the CPU 140 sets predetermined values in a plurality of registers existing in the image recognition unit 30 and the control unit in the image generation unit 20. Similarly, the CPU 140 writes a value required for operation to the image sensor unit 10. At this time, as the value of the image pickup parameter for controlling the image sensor of the image sensor unit 10, a predetermined initial value determined in advance or an initial value set by the user is set.

Next, in step S181, the image sensor unit 10 executes an imaging process to acquire captured image data.
Subsequently, in step S182, the image generation unit 20 stores the image data acquired by the image sensor unit 10 in the internal frame buffer in frame units. Here, in the case of the present embodiment, it is assumed that the image generation unit 20 starts the generation and output of the predicted image for one frame after the image data for five frames is input. Therefore, the image generation unit 20 acquires image data from the image sensor unit 10 in frame units, and determines in step S183 whether to start generating the predicted image. When the image generation unit 20 can acquire the image data for 5 frames, the image generation unit 20 starts the generation of the predicted image in step S183, generates the predicted image for 1 frame, and outputs the predicted image. On the other hand, when the image data for 5 frames has not been acquired, the image generation unit 20 returns to the image acquisition process in step S181 by the image sensor unit 10 without starting the generation of the predicted image. In the present embodiment, the image generation unit 20 performs predetermined processing (prediction image such as feature extraction processing, etc.) on the image data already stored in the frame buffer until the image data of the next frame is acquired. Various processes required for generation) shall be executed. The details of the predicted image generation process in the image generation unit 20 will be described later.

When the prediction image generation process is started in step S183 and the prediction image is generated, the CPU 140 proceeds to step S184 performed by the image recognition unit 30.
Proceeding to step S184, the image recognition unit 30 performs an image recognition process using the predicted image generated by the image generation unit 20. In the case of the present embodiment, the image recognition unit 30 executes the image recognition process and outputs the image recognition result in units of the divided regions in which the predicted image generated by the image generation unit 20 is divided into a plurality of regions. Then, the image recognition unit 30 outputs the coordinate value of the divided region (coordinate value indicating the position of the pixel on the image) on the image (predicted image) as a result of the image recognition process.

Next, as the image pickup parameter determination process in step S185, the parameter determination unit 40 determines the value of the above-mentioned image pickup parameter that determines the operating characteristics of the image sensor unit 10. In the case of the present embodiment, the parameter determination unit 40 has information on the coordinate values of the divided region on the image calculated as the recognition result by the image recognition process in step S184, and the predicted image generated by the image generation process in step S182. Based on the above, the value of the imaging parameter is determined. A specific example of the method for determining the imaging parameter value will be described later.

Next, as the image pickup parameter setting process in step S186, the parameter setting unit 50 sets the value of the image pickup parameter determined in step S185 to the image sensor unit 10.
When the setting of the imaging parameter in step S186 is completed, the CPU 140 confirms the issuance of the end notification (for example, an interrupt signal to the CPU 140 based on the end instruction by the user) in step S187. If the end notification is not issued, the CPU 140 returns the processing of the imaging device to the image acquisition processing of step S181, and continues the processing after step S181. On the other hand, when the CPU 140 confirms the issuance of the end notification, the CPU 140 ends the process of the flowchart of FIG.

In the above description, the image generation unit 20 starts outputting the predicted image for one frame after the image data for five frames is input, but the present invention is not limited to this example. For example, the image generation unit 20 may start outputting the predicted image for one frame after the image data for any n frames in the past is input. In this case, "n" for any n frames in the past is a value of 1 or more. For example, when "n" is 1, the image generation unit 20 will generate a predicted image based on the image data for one frame in the past. Further, in the present embodiment, the image generation unit 20 outputs the predicted image for one frame, but may generate and output the predicted image for a plurality of frames. The predicted images for a plurality of frames may be predicted images of a plurality of frames in chronological order, or may be a plurality of predicted images corresponding to the same frame time. The plurality of predicted images corresponding to the same frame time may be, for example, a plurality of patterns of predicted images in the order of probability predicted from the image acquired by imaging (for example, in order of high probability of prediction).

Next, the image generation process executed by the image generation unit 20 described above will be described in detail with reference to FIG. FIG. 3 is a diagram schematically showing the configuration of a neural network (hereinafter, referred to as NN) that realizes an image generation process for generating a predicted image.
As for the NN (neural network) used in the present embodiment, the one disclosed in detail in Reference 1 below can be applied, and the image generation unit 20 applies the NN to a plurality of frames. It has a function to generate a predicted image based on the image data. In the present embodiment, it is assumed that the method disclosed in Reference 1 is applied as a method for realizing the image generation processing, but the method is not limited to this method. For example, a method as disclosed in Reference 2 below may be used. Further, the method for realizing the image generation processing in the present embodiment may be a method using not only the methods disclosed in References 1 and 2 but also other methods.

Reference 1: Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting", arXiv 2015

Reference 2: William Lotter, Gabriel Kreiman, David Cox, "Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning", arXiv 2017

_{The image generation unit 20 temporarily stores the image data X t} input from the image sensor unit 10 in the internal frame buffer, and then inputs the image data X t to the convolution RSTM module as shown in FIG. The convolutional LSTM module is an NN that executes processing including time series information on time series data. The convolutional LSTM module is an extension of the widely used LSTM (long short-term memory) to time-series image data. Reference 1 above describes in detail the convolutional LSTM.

The processing in the convolutional LSTM is represented by the following equation (1). As described above, the convolutional LSTM is an NN capable of obtaining a desired output by learning in advance using time series image data. In the present embodiment, learning is performed in advance so as to generate the predicted image X _{5 of the next frame from the time series image data X 0 to} X ₄ for the past 5 frames based on the equation (1). ..

Here, the contents indicated by each symbol in the equation (1) are as follows. The Hadamard product is described in equation (1).
X: Input data C: Cell output data H: State of hidden layer i: Input gate f: Oblivion gate o: Output gate t: Time (frame)
W _xo : Input load matrix (output gate)
W _ho : State load matrix (output gate)
W _co : Cell output load matrix (output gate)
_bo : Bias (output gate)
W _xc : Input load matrix (cell output)
W _hc : State load matrix (cell output)
b _c : Bias (cell output)
W _xf : Input load matrix (forgetting gate)
W _hf : State load matrix (forgetting gate)
W _cf : Cell output load matrix (forgetting gate)
b _f : Bias (forgetting gate)
W _xi : Input load matrix (input gate)
W _hi : State load matrix (input gate)
W _ci : Cell output load matrix (input gate)
b _i: bias (input gate)
*: Convolution operation

The image generation unit 20 of the present embodiment is configured by using the convolutional LSTM as described above, but the present invention is not limited to this. Regarding the method for generating the predicted image, various methods such as Non-Patent Document 2 have been proposed in addition to the above-mentioned method, and other methods may be used.

In the NN configuration shown in FIG. 3, the convolutional LSTM outputs a predicted image for one frame based on the images for the past five frames. In the example of FIG. 3, as the image generation process, a predicted image for one frame may be output based on an image for an arbitrary n frames in the past, or a predicted image for a plurality of frames may be output. It may be output. The image generated by the convolutional LSTM is temporarily stored in the frame buffer and input to the image recognition process in step S184 in FIG.

Next, the image recognition process of step S184 executed by the image recognition unit 30 described above will be described in detail with reference to FIG. FIG. 4 is a diagram schematically showing a configuration example of an NN that realizes a predetermined image recognition process for an input predicted image.
The NN shown in FIG. 4 is composed of a hierarchical convolution NN (neural network) module 190 and a hierarchical deconvolution NN module 191, and each layer is composed of convolution arithmetic elements 401 to 408. The NN as shown in FIG. 4 is disclosed in detail in Reference 3 below. The image recognition unit 30 has a function of executing an image recognition process (a process of obtaining a divided region as a recognition result) for each object in the image with respect to the input predicted image data by the NN as shown in FIG.

Reference 3: Hyeonwoo Noh, Seunghoon Hong, Bohyung Han, "Learning Deconvolution Network for Semantic Segmentation", ICCV 2015

The image recognition unit 30 temporarily stores the input predicted image data in the internal frame buffer, inputs the input to the hierarchical convolution NN module 190 shown in FIG. 4, and further inputs the input to the hierarchical deconvolution NN module 191. The NN composed of the hierarchical convolution NN module 190 and the hierarchical deconvolution NN module 191 is widely used for image processing. In particular, an example in which the region division is applied as in the present embodiment is described in detail in Reference 3.

The process in the hierarchical convolutional NN module 190 is represented by the following equation (2).

Here, the contents indicated by each symbol in the equation (2) are as follows.
x: Input data y: Output data w: Convolution filter v: Convolution filter height h: Convolution filter width m: Front layer feature surface index n: Rear layer feature surface index i: Front layer calculation position y coordinates j: Front stage Layer calculation position x coordinates i': Previous layer calculation position y coordinates j': Rear layer calculation position x coordinates k: Convolution filter y coordinates l (t): Convolution filter x coordinates

The hierarchical deconvolutional NN module 191 is represented by the equation (3).

Here, the contents indicated by each symbol in the equation (3) are as follows.
x: Input data (data enlarged by inserting a zero value in the feature plane of the previous layer)
y: Output data w: Deconvolution filter v: Deconvolution filter height h: Deconvolution filter width m: Front layer feature surface index n: Rear layer feature surface index i: Front layer calculation position y coordinates j: Front layer Calculation position x coordinates i': Previous layer calculation position y coordinates j': Rear layer calculation position x coordinates k: Deconvolution filter y coordinates l (t): Deconvolution filter x coordinates

In the present embodiment, a desired NN composed of the hierarchical convolution NN module 190 and the hierarchical deconvolution NN module 191 is subjected to pre-learning using image data having teacher data related to region division. It is possible to obtain the output. A general learning method can be applied to the learning process, and a detailed description of the learning method will be omitted.

The image recognition unit 30 in the present embodiment is configured by using the NN composed of the hierarchical convolution NN module 190 and the hierarchical deconvolution NN module 191 as described above, but is limited to this example. It's not a thing. As for the method for realizing the image recognition processing accompanied by the region division, various methods including Reference 3 have been proposed in addition to this, and other methods may be used.

In the image recognition process of step S184 of FIG. 2 described above, the image recognition unit 30 executes the area division process on the input predicted image, and uses the divided area defined for each object in the predicted image as the recognition result. Perform the calculation process. Specifically, as shown in FIG. 5, the image recognition unit 30 outputs the coordinates of the pixels corresponding to the boundary line 510 of each region of the object included in the predicted image 500.

Next, the imaging parameter determination process of step S185 executed by the parameter determination unit 40 described above will be described.
First, the parameter determination unit 40 averages the brightness values in each region in the predicted image 500 shown in FIG. 5 based on the region information defined for each object in the predicted image and the data of the predicted image. Calculate the value.

Here, when, for example, an exposure parameter is determined as an imaging parameter, the parameter determination unit 40 sets an appropriate exposure time value determined in advance corresponding to the range of the average value of the brightness values in each region to a LUT (lookup table). ) Is held as. The parameter determination unit 40 uses this LUT to determine an exposure parameter that determines the exposure time for each pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 based on the calculated average value of the brightness values. ..

When, for example, a dynamic range parameter is determined as an image pickup parameter, the parameter determination unit 40 is a pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 based on the average value of the brightness values calculated as described above. Determine the dynamic range for each. In this case, the parameter determination unit 40 holds as a LUT an appropriate dynamic range value determined in advance corresponding to the range of the average value of the brightness values in each region. Then, the parameter determination unit 40 uses the LUT and determines the dynamic range for each pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 based on the calculated average value of the luminance values. To determine.

Further, when determining, for example, a gain parameter as an imaging parameter, the parameter determining unit 40 sets each region with respect to the output value of the imaging sensor of the image sensor unit 10 based on the average value of the brightness values calculated as described above. The gain value is determined for each corresponding pixel. In this case, the parameter determination unit 40 holds an appropriate gain value determined in advance corresponding to the range of the average value of the brightness values in each region as the LUT. Then, the parameter determination unit 40 uses the LUT to determine a gain parameter that determines the gain value for each pixel corresponding to each region of the image sensor of the image sensor unit 10 based on the calculated average value of the luminance values. To do.

After that, in the image pickup parameter determination process of step S185, the parameter determination unit 40 outputs the image pickup parameter determined as described above to the image pickup parameter setting process of step S186.
The focus parameter, frame rate parameter, imaging area parameter, and resolution parameter will be described in other embodiments described later.

Further, in the present embodiment, as described above, an example of determining the imaging parameter for each divided region has been described, but the present embodiment is not limited to this example. The imaging parameter determination process may be a process of determining an imaging parameter determined from a specific region as an imaging parameter to be used for the entire region. For example, in FIG. 5, the imaging parameter determined from the region of the automobile in the predicted image 500 may be used as the imaging parameter for the entire region. Further, the case where the imaging parameter is determined for each of the above-mentioned divided regions and the case where the imaging parameter used for the entire region may be determined may be switched. For example, the imaging parameters to be used for the entire region may be determined only in a predetermined frame period, and the imaging parameters may be determined for each divided region in other cases.

Next, the imaging parameter setting process of step S186 executed by the parameter setting unit 50 will be described.
The parameter setting unit 50 has an image pickup parameter setting function that sets the image pickup parameters input from the parameter determination unit 40 to the image sensor unit 10.
When an exposure parameter is input as an imaging parameter, the parameter setting unit 50 enters a register related to exposure time control for each pixel of the imaging sensor in the driver circuit based on the exposure time determined for each region by the exposure parameter. , Set the exposure time.

When a dynamic range parameter is input, the parameter setting unit 50 registers the dynamic range in the register related to the dynamic range control for each pixel of the image sensor in the driver circuit based on the dynamic range determined for each area. Set the value of.

When a gain parameter is input, the parameter setting unit 50 sets the gain value in the register related to the gain value control for each pixel of the image sensor in the driver circuit based on the gain value determined for each region. To do.

Then, in the image sensor unit 10, the driver circuit controls the operation of the image sensor based on the values (value indicating the exposure time, the value of the dynamic range, and the gain value) set in the register as described above.
The image sensor in the present embodiment has a configuration in which the exposure time, dynamic range, and gain value can be controlled for each pixel, and the exposure time, dynamic range, and gain value can be set. To do. That is, the image sensor in the present embodiment can control the exposure time for each pixel by adopting a configuration in which the reset timing of the photodiode can be set for each pixel. Further, the image sensor can control the dynamic range and the gain value for each pixel by forming an independent A / D conversion circuit for each pixel.

Further, in the imaging parameter determination process, for example, when the imaging parameter determined from a specific region is determined as the imaging parameter to be used for the entire region, the parameter setting unit 50 uses the input imaging parameter as the entire region of the imaging sensor. Set for. Specifically, each imaging parameter is set in the register related to the control of the entire area of the imaging sensor in the driver circuit.

As described above, the imaging device of the present embodiment generates a predicted image from the past captured images, and controls the imaging parameters based on the image recognition processing result for the predicted image. Thereby, according to the image pickup apparatus of the embodiment, it is possible to prevent nonconformity of the image pickup parameters that may occur when the image pickup parameters for the future image are controlled based on the image recognition processing result estimated from the past image. it can. That is, according to the present embodiment, it is possible to realize control of imaging parameters suitable for future images. For example, if the area of each object in the image changes for each image frame, the area of each object obtained from the past image may shift in the future image, and as a result, the adjustment area of the imaging parameter also becomes. It is possible that it will shift. On the other hand, in the imaging device of the present embodiment, since the imaging parameter of the region for each object is determined based on the image recognition processing result of the predicted image, the region for each object in the image changes for each image frame. However, it is possible to set more appropriate imaging parameters.

As described above, in the first embodiment, the imaging parameters of the imaging sensor are controlled based on the recognition result for the predicted image, but the method for generating the predicted image, the recognition processing method for the predicted image, and the imaging parameters based on the recognition processing result. There is no particular limitation on the determination and setting method of. In particular, regarding the determination and setting method of the imaging parameter based on the recognition processing result, many methods based on the recognition processing result are disclosed as in Reference 4 below, and these are the recognition processing results for the predicted image in the present embodiment. It is also applicable to the method based on.

Reference 4: Japanese Patent Application Laid-Open No. 2011-130031

<Embodiment 2>
Next, the second embodiment will be described.
In the first embodiment, an example of setting the imaging parameters for the divided region (or the region of the entire image) based on the result of the image recognition processing has been given. On the other hand, in the case of the second embodiment, the image recognition unit 30 shown in FIG. 1 executes the object recognition process as the image recognition process, and the parameter determination unit 40 is the recognition target determined by the object recognition process. It differs from the first embodiment in that a predetermined imaging parameter is determined for a region of an object. Hereinafter, only the parts different from the first embodiment will be described, and the other parts will be the same as those of the first embodiment, and thus the description thereof will be omitted.

In the image pickup apparatus of the present embodiment, the image recognition unit 30 of FIG. 1 has an image recognition processing function for recognizing a specific object set in advance from the predicted image generated by the image generation unit 20. Although the specific object set in advance is a human body in the present embodiment, it is possible to recognize any object learned in advance such as a car, a bicycle, or a traffic light.

FIG. 6 is a flowchart showing the operation of the image pickup apparatus according to the present embodiment.
In FIG. 6, since the processes of steps S180 to S183 are the same as the processes described in the above-described first embodiment, detailed description thereof will be omitted.

In the case of the second embodiment, when the prediction image generation process is started in step S183 and the prediction image is generated, the CPU 140 proceeds to step S188 performed by the image recognition unit 30.
Proceeding to the object recognition process in step S188, the image recognition unit 30 executes the human body recognition process on the predicted image generated by the image generation unit 20. Then, the image recognition unit 30 outputs the coordinate values (upper left pixel position and lower right pixel position) on the image of the rectangular region estimated for the recognized human body as a result of the image recognition process. The details of the process of recognizing the human body from the predicted image will be described later. After the process of step S188, the process proceeds to step S185.

Proceeding to the imaging parameter determination process of step S185, the parameter determination unit 40 determines the information of the coordinate values of the rectangular region of the human body obtained by the object recognition process of step S188 and the predicted image generated by the image generation process of step S182. Based on the above, the imaging parameter value is determined. Details of the method for determining the imaging parameter value will be described later.

Subsequently, the image recognition process of step S188 executed by the image recognition unit 30 of the second embodiment will be described in detail with reference to FIG. 7. FIG. 7 is a diagram schematically showing a configuration example of an NN that realizes a predetermined image recognition process for the input predicted image (X).

The image recognition process in the present embodiment is composed of a hierarchical convolutional NN widely applied as a recognition processing technique, and includes the hierarchical convolutional NN module 190 of FIG. 4 included in the NN described in the above-described first embodiment. Perform the same arithmetic processing. Each layer of the hierarchical convolutional NN module is composed of convolutional arithmetic elements 701 to 704.

The image recognition unit 30 temporarily stores the input predicted image data in the internal frame buffer, then inputs the input to the hierarchical convolutional NN module, and executes a process of recognizing the human body. Here, the processing in the hierarchical convolutional NN module is executed by sequentially scanning a predetermined area in the image in the same manner as the general object recognition processing.

The process in the hierarchical convolutional NN is represented by the equation (2) in the same manner as the process in the hierarchical convolutional NN in the first embodiment. In the case of the second embodiment, it is possible to recognize the human body in the image by learning in advance using the image data having the teacher data regarding the position and size of the human body on the image.

The image recognition unit 30 in the present embodiment is configured by using a hierarchical convolutional NN module, but the present invention is not limited to this, and various methods for recognizing a specific object have been proposed. And other methods may be used.

In the object recognition process of step S188 of FIG. 6, the image recognition unit 30 executes a process of recognizing a human body with respect to the input predicted image, and estimates the position and size of the human body in the predicted image. Specifically, as shown in FIG. 8, the image recognition unit 30 has coordinate values (upper left pixel position 811 and lower right) on the image of the rectangular region 810 estimated from the predicted image 800 with respect to the recognized human body. The pixel position 812) is output.

Further, the image recognition unit 30 can further execute the human body posture recognition process in addition to the human body recognition process as the image recognition process in step S188. The human body posture recognition process can be calculated by the above-mentioned hierarchical convolutional NN module. In this case, in the object recognition process of step S188, the image recognition unit 30 adds the coordinate values on the image of the rectangular region estimated as described above, and as shown in FIG. 9, the position of each part of the human body from the predicted image 900. Each coordinate value indicating 910 to 915 is calculated. In the example of FIG. 9, the coordinate values of the head center position 911, the torso center position 910, the left hand position 912, the right hand position 913, the left foot position 914, and the right foot position 915 of the predicted image 900 are output. .. The object to be recognized is not limited to the posture of the human body, and an object posture recognition process for recognizing the posture of an arbitrary object may be executed. For example, when recognizing the posture of an automobile, it is possible to recognize each part such as a headlamp position, a tire position, and a roof position.

Next, the imaging parameter determination process of step S185 executed by the parameter determination unit 40 in the second embodiment will be described.
The parameter determination unit 40 obtains information on the coordinate values (upper left pixel position 811 and lower right pixel position 812) on the image of the rectangular region 810 estimated for the human body in the predicted image 800 of FIG. 8 and the data of the predicted image 800. Based on this, the average value of the brightness values in each region is calculated.

When determining the exposure parameter, the parameter determination unit 40 holds as a LUT an appropriate exposure time determined in advance corresponding to the range of the average value of the luminance values in each region. Then, the parameter determination unit 40 uses the LUT and determines the exposure time for each pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 based on the calculated average value of the luminance values, thereby determining the exposure parameter. To determine.

When determining the dynamic range parameter, the parameter determination unit 40 determines the dynamic range for each pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 based on the average value of the luminance values calculated as described above. To determine. In this case, the parameter determination unit 40 holds an appropriate dynamic range determined in advance corresponding to the range of the average value of the luminance values in each region as the LUT. Then, the parameter determination unit 40 uses the LUT to determine the dynamic range for each pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 based on the calculated average value of the luminance values, thereby determining the dynamic range. Determine the parameters.

When determining the gain parameter, the parameter determination unit 40 determines the pixels corresponding to each region with respect to the output value of the image sensor of the image sensor unit 10 based on the average value of the brightness values calculated as described above. The gain value is determined for each. In this case, the parameter determination unit 40 holds an appropriate gain value determined in advance corresponding to the range of the average value of the brightness values in each region as the LUT. The parameter determination unit 40 determines the gain parameter by using the LUT and determining the gain value for each pixel corresponding to each region of the image sensor of the image sensor unit 10 based on the calculated average value of the brightness values. ..

Also in the second embodiment, the parameter determination unit 40 sends each imaging parameter determined as described above to the parameter setting unit 50 as the imaging parameter setting process in step S186.

Further, in the case of the second embodiment, the parameter determination unit 40 sets the imaging region parameter (imaging region parameter (upper left pixel position and lower right pixel position) as the imaging parameter based on the coordinate values (upper left pixel position and lower right pixel position) on the image of the rectangular region estimated for the human body. It is also possible to determine the area to be imaged by the image sensor). Then, when the imaging region parameter is set by the parameter setting unit 50, the image sensor unit 10 acquires only the image of the region defined by the imaging region parameter, as will be described later.

Further, in the second embodiment, the parameter determination unit 40 calculates the pixel position to be focused based on the coordinate values (upper left pixel position and lower right pixel position) on the image of the rectangular region estimated with respect to the human body. You can also do it. For example, the parameter determination unit 40 calculates the center position of the human body head, for example, 1/10 from the upper part of the rectangular region 1011 estimated with respect to the human body and the left and right center positions 1010 as shown in FIG. Based on, the focus parameter that determines the focus position is determined. Then, when the focus parameter is set by the parameter setting unit 50, the image sensor unit 10 can perform focusing at the focus position set by the focus parameter.

In the second embodiment, when the image recognition unit 30 further executes the human body posture recognition process in addition to the human body recognition process, each of the input human bodies is based on a preset part or a part designated by the user. You can also select one from the coordinate values of the part. In this case, the parameter determination unit 40 can also determine a preset portion or a portion designated by the user (for example, the center position of the human body head) as a focus parameter for determining the focus position.

Also in the case of the second embodiment, the parameter setting unit 50 sets the image pickup parameter determined by the parameter determination unit 40 to the image sensor unit 10 as the image pickup parameter setting process in step S186 of FIG.

For example, when setting the image pickup area parameter, the parameter setting unit 50 sets the register of the driver circuit of the image sensor unit 10 related to the image pickup area control of the image sensor based on the image pickup area parameter determined by the parameter determination unit 40. Set the imaging region (ROI).

When setting the focus parameter, the parameter setting unit 50 sets the focus parameter in the register related to the autofocus control of the image sensor of the driver circuit based on the focus parameter determined by the parameter determination unit 40.
The other imaging parameter setting processes executed by the parameter setting unit 50 are the same as those in the first embodiment, and thus the description thereof will be omitted.

Then, in the image sensor unit 10, the driver circuit controls the operation of the image sensor based on the value set in the register, as in the first embodiment.
The image sensor in the present embodiment has a configuration capable of controlling the exposure time, the dynamic range, and the gain value for each pixel, as described above. Further, the image sensor in the present embodiment has a function of imaging only the area based on the set image area. For example, as shown in FIG. 11, the image sensor in the second embodiment captures only the human body region (region 1110 surrounded by the black rectangle in FIG. 11) recognized from the predicted image based on the control of the driver circuit, and images the image. Get as data. Further, the image sensor of the present embodiment has a function of realizing autofocus processing including control of the optical system based on the set focus position.

As described above, the image pickup apparatus of the second embodiment controls the image pickup parameters for the future image based on the image recognition result for the predicted image generated from the past image. Thereby, as in the above-described first embodiment, it is possible to prevent the nonconformity of the imaging parameters that may occur when the imaging parameters for the future image are controlled based on the recognition result in the past image. In the case of performing image recognition of a person as in the second embodiment, if the position / size of an object such as the human body changes for each image frame, the area of the human body or the like estimated from the past image shifts in the future image. There is a possibility that the adjustment range of the imaging parameters may shift. In addition, when the size and posture of the human body, etc. in the image change, the position of the specific part based on the area estimated from the past image and the position of the specific part based on the posture of the object will also shift in the future image. there is a possibility. On the other hand, in the present embodiment, since the region and posture of the human body or the like in the future image can be estimated based on the predicted image, even if the position, size and posture of the human body or the like change for each image frame, the region and the posture can be estimated. Appropriate imaging parameters can be set for the posture.

In the present embodiment, the imaging parameters of the image sensor unit are controlled based on the recognition result for the predicted image, but as in the first embodiment, the method for generating the predicted image, the recognition processing method for the predicted image, and the recognition processing result are used. The method for determining and setting the imaging parameters is not limited. Regarding the method of determining and setting the imaging parameter based on the recognition processing result, various methods based on the recognition processing result can also be applied to the method based on the recognition processing result of the measurement image in the present embodiment.

<Embodiment 3>
Next, the third embodiment will be described. In the third embodiment, the image recognition unit 30 executes a scene recognition process for recognizing what kind of scene (scene) is performed as the recognition process, and the parameter determination unit 40 determines the scene according to the scene recognized in the scene recognition process. It differs from the second embodiment in that a predetermined imaging parameter is determined. Hereinafter, only the parts different from the second embodiment will be described, and the other configurations and the like are the same as those of the second embodiment, so the description thereof will be omitted.

In the image pickup apparatus of the second embodiment, the image recognition unit 30 of FIG. 1 recognizes a specific preset scene from the predicted image generated by the image generation unit 20. In the case of the present embodiment, the specific scene set in advance is, for example, "pedestrian jumping out", but in addition, "car jumping out", "bicycle jumping out", "pedestrian falling out", etc. It may be any scene learned in. The image recognition unit 30 performs an image recognition process for recognizing these specific scenes.

FIG. 12 is a flowchart showing the operation of the image pickup apparatus according to the present embodiment.
Since the processes from step S180 to step S183 and the processes from steps S185 to S187 in FIG. 12 are the same as those in the second embodiment, detailed description thereof will be omitted.

In the case of the third embodiment, when the prediction image generation process is started in step S183 and the prediction image is generated, the CPU 140 proceeds to step S189 performed by the image recognition unit 30.
Proceeding to the scene recognition process of step S189, the image recognition unit 30 executes the recognition process of the "pedestrian jumping out" scene on the predicted image generated by the image generation unit 20. As a result of the image recognition processing, the image recognition unit 30 sets a True / False flag for image recognition of the "pedestrian pop-out" scene and the image of the rectangular area estimated for the recognized scene. (Upper left pixel position and lower right pixel position) are output. After the process of step S189, the process proceeds to step S185.

Proceeding to the imaging parameter determination process of step S185, the parameter determination unit 40 is based on the recognition result information (true / false flag: True / False flag) of the specific scene calculated in the scene recognition process of step S189. Determine the imaging parameter value. The imaging parameter value in this embodiment is a value of the frame rate of the image acquired by the image sensor unit 10. Details of the method for determining the frame rate parameter value will be described later.

Next, the scene recognition process executed by the image recognition unit 30 described above will be described in detail.
The scene recognition process in the present embodiment is performed by a hierarchical convolutional NN widely applied as a recognition processing technique, and is executed by the same arithmetic processing as the hierarchical convolutional NN shown in FIG. 7 described in the second embodiment.

The image recognition unit 30 temporarily stores the input predicted image data in the frame buffer, then inputs the input to the hierarchical convolution NN module, and executes a process of recognizing the "pedestrian jumping out" scene. The process in the hierarchical convolutional NN module is represented by the equation (2) in the same manner as the process in the hierarchical convolutional NN in the second embodiment. In the case of the third embodiment, it is possible to recognize the "pedestrian jumping out" scene in the image by learning in advance using the image data having the teacher data regarding the "pedestrian jumping out" scene. ..

The image recognition unit 30 in the present embodiment is configured by using a hierarchical convolutional NN module, but the present invention is not limited to this example, and various other methods are used as a method for recognizing a specific scene. May be done.

As described above, in the scene recognition process of step S189, the image recognition unit 30 executes a process of recognizing the "pedestrian jumping out" scene for the input predicted image, and the scene exists in the predicted image. Estimate whether or not. As shown in FIG. 13, the image recognition unit 30 has a true / false flag indicating whether or not a “pedestrian pop-out” scene exists in the predicted image 1300, and an image of the rectangular region 1310 estimated in the recognized scene. (Upper left pixel position 1311 and lower right pixel position 1312) are output.

Next, the imaging parameter determination process executed by the parameter determination unit 40 will be described.
The parameter determination unit 40 includes information on the estimated true / false flag regarding the presence or absence of the "pedestrian jumping out" scene in the predicted image, and the information on the coordinate value on the image of the estimated rectangular area for the recognized scene. Based on the above, the frame rate of the image acquired by the image sensor unit 10 is determined.

The parameter determination unit 40 holds the frame rate of the image determined in advance corresponding to the presence / absence (true / false flag) of the "pedestrian pop-out" scene as a LUT, and based on the presence / absence of the scene, the image The frame rate of the image sensor of the sensor unit 10 is determined. For example, when a "pedestrian jumping out" scene is recognized in the predicted image (a true flag is input), the parameter determination unit 40 changes the frame rate (60 fps) of the image sensor unit 10 during normal operation to 240 fps. Decide to do. Then, the parameter determination unit 40 determines the frame rate parameter after the change.

The parameter setting unit 50 sets the image pickup parameter (frame rate parameter) determined as described above in the image sensor unit 10 in the image pickup parameter setting process of step S186.
In the present embodiment, the image pickup parameter determined by the image pickup parameter determination process is the frame rate of the image sensor of the image sensor unit 10, and the present invention is not limited to this. For example, based on the information of the coordinate values on the image of the rectangular region estimated for the "pedestrian jumping out" scene, the pixel circuit corresponding to each region of the image sensor of the image sensor unit 10 is similar to the second embodiment. Each exposure time, dynamic range and gain value may be determined.
Since the imaging parameter setting process executed by the parameter setting unit in step S186 of FIG. 12 is the same as that of the second embodiment, the description thereof will be omitted.

Even in the case of the scene recognition process, the presence / absence of the scene and the area where the scene is recognized may be different from the presence / absence of the scene and the recognition area of the scene in the future image, and as a result, the imaging parameter Adjustments may not fit the actual scene. On the other hand, in the case of the present embodiment, since the presence / absence of a specific scene and the area of the scene are determined based on the predicted image, even if the scene changes for each image frame, the presence / absence and area of a more appropriate scene can be obtained. On the other hand, it is possible to set the imaging parameters. Further, the parameter determination unit 40 may perform a parameter determination process for controlling the value of a specific imaging parameter among the imaging parameters according to the recognized scene. For example, when a "pedestrian pop-out" scene is recognized, in addition to the example of determining the frame rate parameter for increasing the frame rate as described above, the imaging region parameter for reading the pedestrian region is determined. You can do it. Furthermore, when a "pedestrian jumping out" scene is recognized, a resolution parameter that controls to increase the resolution of the image sensor may be determined.

Also in the third embodiment, similarly to the above, the method of generating the predicted image, the method of recognizing the predicted image, and the method of determining and setting the imaging parameter based on the recognition processing result are not limited. In particular, various methods can be applied to the method of determining and setting the imaging parameter based on the recognition processing result and the method based on the recognition processing result for the predicted image, as in the second embodiment.

<Embodiment 4>
Next, the fourth embodiment will be described.
FIG. 14 is a block diagram showing a configuration example of the image pickup apparatus according to the fourth embodiment. As can be seen in comparison with the configuration example of FIG. 1, the image pickup apparatus of the fourth embodiment shown in FIG. 14 has a vector calculation unit 200 added to the configuration of FIG.

Here, the vector calculation unit 200 has a motion vector detection function that detects (calculates) a motion vector from the image captured by the image sensor unit 10. Then, the image generation unit 210 of the fourth embodiment has a function of generating a predicted image based on the input image data and the motion vector detected (calculated) by the vector calculation unit 200 from the image data. The vector calculation unit 200 and the other units other than the image generation unit 210 are assumed to have the same functions as those in the first embodiment, and detailed description thereof will be omitted.

Next, the operation of the image pickup apparatus of the fourth embodiment having the configuration shown in FIG. 14 will be described. FIG. 15 is a flowchart showing the operation of the image pickup apparatus according to the present embodiment.
Since each process of step S180 and step S181 of FIG. 15 and steps S183 to S187 is the same as that of the first embodiment, detailed description thereof will be omitted.
In the case of the fourth embodiment, after the processing of step S181, the CPU 140 proceeds to the processing of step S201 performed by the vector calculation unit 200.

Proceeding to step S201, the vector calculation unit 200 calculates the motion vector as the motion vector calculation process based on the image data for two time-continuous frames acquired by the image sensor unit 10. Execute. Then, the vector calculation unit 200 outputs a motion vector (including information on the direction of motion and the magnitude of motion) at each pixel position of the image as a result of the motion vector calculation process.

Subsequently, in step S202, the image generation unit 210 first stores the image data acquired by the image sensor unit 10 and the motion vector data calculated by the vector calculation unit 200 in the internal frame buffer in frame units. To do. Then, the image generation unit 21 generates a predicted image based on the image data and the motion vector data.

Hereinafter, the image generation process of step S202 executed by the image generation unit 210 will be described. The point that the image generation unit 210 generates a predicted image based on the image data input from the image sensor unit 10 and the motion vector data calculated by the vector calculation unit 200 is the above-described first to third embodiments. Is different.

The image generation process in the image generation unit 210 of the fourth embodiment is easily realized by expanding the input terminal of the NN module that performs the convolutional LSTM described in the first to third embodiments. The calculation and learning processes executed by the image generation unit 210 are the same as those in the first to third embodiments. In the present embodiment, it is assumed that the method disclosed in Reference 1 described above is used as the method for realizing the image generation processing, but the method is not limited to this, as in the first to third embodiments. The same is true.

In the case of the fourth embodiment, the predicted image is generated from the past image and the motion vector, and the imaging parameters for the future image are controlled based on the recognition result for the predicted image. Also in the case of the fourth embodiment, it is possible to prevent the nonconformity of the imaging parameters that may occur when the imaging parameters for the future image are controlled based on the recognition result generated from the past image.

In the present embodiment, an example in which a vector calculation unit (motion vector calculation process) is further added to the configuration of the first embodiment has been described, but the same vector is applied to the second and third embodiments. A calculation unit (motion vector calculation process) may be added. These examples will be omitted in detail as they can be easily extended from the present embodiment.

Further, in the present embodiment, the example in which the motion vector data is calculated by the vector calculation unit 200 and given as the input data of the image generation unit 210 has been described, but the image generation unit 210 predicts the motion vector data in the future image ( It is also possible to calculate). Reference 5 discloses a method for calculating predicted motion vector data in a future image, and it is also possible to realize an image generation unit in the present embodiment by using this method.

Reference 5: Xiaodan Liang, Lisa Lee, Wei Dai, Eric P. Xing, "Dual Motion GAN for Future-Flow Embedded Video Prediction", ICCV 2017

Further, as a modification of the present embodiment, it is also possible to input the predicted motion vector information together with the predicted image to the image recognition unit 30. This can be easily realized by further expanding the input terminal of the image recognition unit 30 described in the first to third embodiments for inputting the predicted motion vector information.

According to the present embodiment, by generating a predicted image using motion vector data in addition to the image data, the predicted image accuracy of the image accompanied by motion is improved, so that the imaging parameters more suitable for future images can be obtained. It can be set.

<Embodiment 5>
Next, the fifth embodiment will be described. FIG. 16 is a block diagram showing a configuration example of the image pickup apparatus according to the fifth embodiment.
As can be seen in comparison with the configuration example of FIG. 1, the image pickup apparatus of the fifth embodiment shown in FIG. 16 is configured by providing a plurality of image recognition units 30 and RAM 80 of the first to third embodiments. The image recognition unit 30A of FIG. 16 is in charge of the image recognition process (area division process) described in the first embodiment, and the image recognition unit 30B is in charge of the image recognition process (object recognition process) of the second embodiment. 30C is in charge of the image recognition process (scene recognition process) of the third embodiment.

Further, FIG. 17 shows a flowchart showing the operation of the image pickup apparatus according to the fifth embodiment. In the case of the flowchart shown in FIG. 17, in the first to third embodiments, the area division process of step S184, the object recognition process of step S188, and the scene recognition process of step S189 described as the image recognition process are all included.

In FIG. 17, when the predicted image is generated in step S183, the CPU 140 performs the image recognition process (region division process) in step S184, the image recognition process (object recognition process) in step S188, and the image recognition process (scene) in step S189. Proceed to the recognition process).

In step S184, the image recognition unit 30A executes image recognition processing (area division processing), in step S183, the image recognition unit 30B executes image recognition processing (object recognition processing), and in step S189, the image recognition unit 30C performs image recognition. Execute processing (scene recognition processing). Then, after the steps S184, S188, and S189, the process proceeds to step S185 performed by the parameter determination unit 40. Since each image recognition process of step S184, step S188, and step S189 is the same process as described in each of the above-described embodiments, the description thereof will be omitted. In the fifth embodiment, it is possible to further execute the object posture recognition process in addition to the object recognition process in step S188, which is the same as the second embodiment.

Further, the processing performed in the imaging parameter determination process in step S185 and the imaging parameter setting process in step S186 in FIG. 17 may be a process of appropriately executing a part of the processes described in the first to third embodiments, or a plurality of processes. The process may be a combination of the above processes. For example, with respect to the dynamic range of the background region (for example, the empty region), the value of the imaging parameter may be determined by the process described in the first embodiment. Further, for example, with respect to the dynamic range of the human body region, the value of the imaging parameter may be determined by the process described in the second embodiment. Further, the frame rate may be a process of determining the value of the imaging parameter by the method described in the third embodiment. Further, also in the image pickup apparatus of the fifth embodiment, it is naturally possible to add a vector calculation unit and execute the process based on the motion vector data as in the fourth embodiment.

As described above, in the image pickup apparatus according to the fifth embodiment, the image generation unit 20 first generates a predicted image, and the image recognition units 30A, 30B, and 30C execute image recognition processing on the predicted image, respectively. Then, the parameter determination unit 40 determines the imaging parameters according to the execution result of those image recognition processes. Further, in the case of the fifth embodiment, a plurality of types of processes are executed as the image recognition process. Further, in the case of the fifth embodiment, once the predicted image is generated, the image recognition process applied to the generated predicted image is not limited as in the above-described first to fourth embodiments. Therefore, in the case of the fifth embodiment, it is possible to execute the processes other than the image recognition processes of the first to fourth embodiments and the present embodiment. For example, as a recognition process different from the image recognition process described above, it is possible to execute a movement action detection process or the like for detecting an abnormal behavior of a person, a car, or the like. Further, also in the fifth embodiment, the imaging parameters described in the first to fourth embodiments or other imaging parameters can be controlled in response to the image recognition process to be executed. Also in this case, the imaging parameters described in the first to fourth embodiments are examples, and the types of imaging parameters in the present invention are not particularly limited.

The configuration of the image processing device of each of the above-described embodiments or the processing of each flowchart may be realized by a hardware configuration, or may be realized by a software configuration by, for example, a CPU executing a program according to the present embodiment. Good. Further, a part may be realized by a hardware configuration and the rest may be realized by a software configuration. The program for software configuration may be acquired not only when it is prepared in advance, but also from a recording medium such as an external memory (not shown) or via a network (not shown).

A program that realizes one or more functions in the control process according to the present invention can be supplied to a system or device via a network or storage medium, and is read and executed by one or more processors of the computer of the system or device. It is feasible by being done.
Each of the above-described embodiments is merely an example of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

10: Image sensor unit, 20: Image generation unit, 30: Image recognition unit, 40: Parameter determination unit, 50: Parameter setting unit, 80, 160: RAM, 140: CPU, 150: ROM

Claims (17)

Image acquisition means for acquiring images by imaging,
An image generation means for generating a predicted image based on the acquired image, and
An image recognition means that executes image recognition processing on the predicted image, and
A parameter determining means for determining imaging parameters based on the result of the image recognition process, and
A parameter setting means for setting the determined imaging parameters for the image acquisition means, and
An imaging device characterized by having. The imaging device according to claim 1, wherein the image recognition means outputs an image recognition result corresponding to a region in which the predicted image is divided. The imaging device according to claim 1 or 2, wherein the image recognition means executes an object recognition process for recognizing a predetermined object in the predicted image. The imaging device according to any one of claims 1 to 3, wherein the image recognition means executes an object posture recognition process for recognizing the posture of a predetermined object in the predicted image. The image pickup apparatus according to any one of claims 1 to 4, wherein the image recognition means executes a scene recognition process for recognizing a scene based on the predicted image. The imaging device according to claim 5, wherein the parameter determining means determines to control a specific imaging parameter according to the scene recognized by the scene recognition process. The image pickup apparatus according to any one of claims 1 to 6, wherein the image recognition means executes a plurality of different image recognition processes. The imaging apparatus according to any one of claims 1 to 7, wherein the image generating means generates the predicted image from at least one acquired image. The imaging device according to any one of claims 1 to 8, wherein the image generating means generates a plurality of predicted images. The image pickup apparatus according to any one of claims 1 to 9, wherein the image generation means generates the predicted image by using a neural network. The image pickup apparatus according to any one of claims 1 to 10, wherein the image recognition means executes the image recognition process by using a neural network. The imaging parameter determined by the parameter determining means is one or a plurality of imaging parameters among the exposure parameter, the focus parameter, the dynamic range parameter, the gain parameter, the frame rate parameter, the imaging area parameter, and the resolution parameter. The imaging device according to any one of claims 1 to 11. The imaging device according to any one of claims 1 to 12, wherein the parameter setting means sets the imaging parameter for each specific region in the image. The imaging device according to any one of claims 1 to 13, wherein the parameter setting means sets the imaging parameter for each pixel of an image. It has a vector detecting means for detecting a motion vector from the acquired image, and has
The imaging device according to any one of claims 1 to 14, wherein the image generating means generates the predicted image based on the acquired image and the motion vector. A control process that controls an image acquisition means that acquires an image by imaging,
An image generation step of generating a predicted image based on the acquired image, and
An image recognition step of executing image recognition on the predicted image, and
A parameter determination step of determining imaging parameters based on the result of image recognition, and
A parameter setting step of setting the determined imaging parameters for the image acquisition means, and
A method for controlling an imaging device, which comprises. A program for causing a computer to function as each means included in the imaging device according to any one of claims 1 to 15.

Images