JP2021044702A - Imaging apparatus and driving control method therefor, and program - Google Patents

Abstract

To provide an imaging apparatus and a driving control method therefor, and a program that can achieve focusing on a detected moving subject smoothly after detecting the moving subject as an event.SOLUTION: In a detection mode, a detection image is generated by adding pixels in each division region of a pixel region of an imaging element 203, and detection images are compared between frames to detect a difference as a subject of interest in each division region. In an imaging mode, a system 2 which generates a picked-up image from the imaging element 203 links a distance l to a fixed subject photographed in each division region. When the subject of interest is detected in the detection mode, a distance to the subject of interest is estimated based upon the distance l linked to the division region in which the subject of interest is photographed, and after a focus position of a focus lens is moved so that the focal length reaches the estimated distance, the imaging mode is entered.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image pickup apparatus and its drive control method, and a program, and more particularly to an image pickup apparatus for detecting a moving subject, its drive control method, and a program.

In recent years, in fields such as traffic and crime prevention, an imaging device that detects a moving subject as an event by image processing has been widely used. For example, in order to reduce power consumption until an event occurs, the pixel area of the image sensor is divided into a plurality of divided areas, and the pixels in each divided area are added to the pixels to generate an analog signal for event detection. An image sensor that operates in a detection mode read from is proposed. After detecting an event based on the analog signal for event detection in the detection mode, the image pickup apparatus shifts to the image pickup mode in which the analog signal for moving image recording is read from the image pickup device (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2018-22935

However, the AF process in Patent Document 1 is performed based on the analog signal for moving image recording read from the image sensor in the image pickup mode after shifting from the detection mode to the image pickup mode. Therefore, even if a moving subject is detected as an event in the detection mode, the moving subject will be out of focus in the moving image recorded in the imaging mode thereafter.

Therefore, an object of the present invention is to provide an image pickup apparatus capable of quickly focusing on the detected moving subject after detecting the moving subject as an event, a drive control method thereof, and a program.

The image pickup apparatus according to claim 1 of the present invention includes a dividing means for dividing a pixel region of an image sensor into a plurality of divided regions, and in a detection mode, pixel addition of pixels in the divided region is performed for each of the plurality of divided regions. The first generation means for generating the detection image and the detection image created by the first generation means in the detection mode are compared between frames, and the difference is set as the subject of interest for each of the plurality of divided regions. A detection means for detecting, a second generation means for generating an image to be captured from the image sensor in the image pickup mode, an AF means for moving the focus position of the focus lens, and a fixed subject reflected in each of the plurality of divided regions. When the attention subject is detected by the associating means for associating the distance to the attention subject and the detection means, the distance to the attention subject is estimated based on the distance associated with the divided area in which the attention subject is reflected. It is characterized in that the AF means moves the focus position of the focus lens so that the focal length matches the estimated distance, and then shifts to the image pickup mode.

According to the present invention, it is possible to quickly focus on the detected moving subject after detecting the moving subject as an event.

It is a flowchart which shows the outline of the event detection processing which concerns on Example 1 of this invention. It is a flowchart of the detection mode operation process of step S103 of FIG. 1A. It is a flowchart of the image pickup mode operation process of step S109 of FIG. 1A in Example 1. FIG. It is a flowchart of attention subject position correction processing of step S107 of FIG. 1A in Example 1. FIG. It is a flowchart of the surrounding environment information creation / update process of step S109-4 of FIG. 1C in Example 1. FIG. It is a figure which shows the outline of the hardware structure of the system as the image pickup apparatus which concerns on Example 1 of this invention. It is a figure which shows the schematic structure of the image pickup device in FIG. It is a figure which shows the schematic structure of the pixel part in the image pickup device of FIG. It is an equivalent circuit diagram of the pixel in the pixel part shown in FIG. It is a figure which shows the circuit structure of the digital addition part in the image pickup device of FIG. It is a block diagram which shows the structure of the event detection part in the image pickup device of FIG. It is a figure which shows the setting example of the detection frame which concerns on Example 1. FIG. It is a figure which shows the example of the fixed subject photographed by the image pickup module part in Example 1. FIG. It is a figure which shows the positional relationship between the plurality of division regions shown in FIG. 8 and the imaging region of a fixed subject shown in FIG. 9A. It is a figure for demonstrating the method of setting the imaging area of a fixed subject having a size of a predetermined value or more as a detection frame. It is a figure which shows the example which the attention subject appeared on the fixed subject shown in FIG. 9A within the angle of view of the image pickup module part in Example 1. FIG. It is a figure which shows the positional relationship between the plurality of division regions shown in FIG. 8 and the imaging region of the subject of interest of FIG. 10A. It is a flowchart of attention subject position re-estimation processing of step S107-2 of FIG. 1D. 2 is a flowchart of the attention subject position correction process in step S107 of FIG. 1A in the second embodiment. It is a flowchart of the surrounding environment information creation / update process of step S109-4 of FIG. 1C in Example 2. FIG. It is a figure which shows the digital addition number of each division area in Example 2. FIG. It is a figure which shows the division method of the pixel area of a pixel part in Example 2. FIG. It is a figure which shows the positional relationship between the plurality of division regions shown in FIG. 13B and the imaging region of a fixed subject shown in FIG. 9A. It is a figure which shows the re-estimation method of the distance lt to the attention subject with respect to the adjacent detection frame in step S107-5 of FIG. 12A. It is a figure for demonstrating the method of estimating the position of the attention subject when the attention subject appears within the angle of view of the image pickup module part in Example 2. FIG. It is a figure for demonstrating the method of updating the area setting of a detection frame, which is executed in step S109-42 of FIG. 12E.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the present invention according to the claims, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. ..

[Example 1]
In this embodiment, an example will be described in which an imaging device having an event detection function such as movement of a subject is associated with prioritized focus positions for each divided region of the imaging region and AF is started at the same time as detection. In this embodiment, the prioritization of the focus positions associated with each of the divided regions is set based on the frequency of occurrence of the subject. Hereinafter, this embodiment will be described with reference to FIGS. 1A to 1E, 2 to 8, 9A to 9C, 10A, 10B, and FIG.

Here, a subject for which it is desired to detect whether or not it is within the angle of view of an image pickup device such as a person is particularly set as a "attention subject", and is distinguished from a fixed subject such as a background such as a wall, floor, or ceiling, or furniture, which is also a subject. Describe. In particular, when "attention" is not given, it refers not only to the subject for which it is desired to detect whether or not it is within the angle of view of the image pickup device such as a person, but also to all objects that are reflected within the angle of view of the image pickup device.

FIG. 2 is a diagram showing an outline of the hardware configuration of the system 2 as the image pickup apparatus according to the present embodiment.

In FIG. 2, the system 2 includes an imaging module unit 20 and an external signal processing unit 21.

The image pickup module section 20 includes a lens 201, a lens drive section 202, and an image pickup element 203.

The external signal processing unit 21 includes an imaging signal processing circuit 210, a memory unit I211, an overall control calculation unit 212, a recording medium control I / F unit 213, a recording medium 214, an external I / F unit 215, and a memory unit II216.

The image pickup module unit 20 and the external signal processing unit 21 are connected by wire or physically separated from each other, and data is transmitted and received by wire or wirelessly.

Details of the image sensor 203 will be described later. The subject image passed through the lens 201 is formed on the image sensor 203. The subject image formed on the plurality of pixels in the image sensor 203 is converted into digital data in the image sensor 203 and sent to the image pickup signal processing circuit 210 as RAW data. The image pickup signal processing circuit 210 performs various image signal processing such as low-pass filter processing, shading processing, and WB processing for reducing noise, as well as various corrections such as scratch correction, dark shading correction, and blackening processing, and compression. Create image data.

The lens 201 is controlled by the lens driving unit 202, and the subject can be focused.

The overall control calculation unit 212 controls the entire system 2 and performs various calculations.

The memory unit I211 temporarily stores the image data.

The recording medium control I / F unit 213 records or reads image data on the recording medium.

The recording medium 214 is a removable storage medium such as a semiconductor memory, and records or reads out image data.

The external I / F unit 215 is an interface for communicating with an external computer or the like.

The memory unit II 216 stores the calculation results of the overall control calculation unit 212, various parameters such as shooting conditions, and various programs.

FIG. 3 is a diagram showing a schematic configuration of the image pickup device 203 in FIG. 2.

The image sensor 203 includes a pixel unit 301, a vertical scanning circuit 302, a control circuit 303, an AD conversion unit 304, a horizontal scanning circuit 305, an exposure control unit 306, an event detection unit 307, and a mode control unit 308. Further, the image sensor 203 includes a lens control unit 309, a switch 310, a digital addition unit 311 and a memory unit 312.

A plurality of pixels are arranged in a two-dimensional grid pattern in the pixel unit 301. Hereinafter, a set of pixels arranged in a predetermined direction (horizontal direction, etc.) is referred to as a "row", and a set of pixels arranged in a direction perpendicular to the row is referred to as a "column".

The control circuit 303 controls the vertical scanning circuit 302, the AD conversion unit 304, and the horizontal scanning circuit 305. A vertical synchronization signal (hereinafter referred to as VD) having a predetermined frequency (30 Hz or the like) is input to the control circuit 303 from the overall control calculation unit 212.

The vertical scanning circuit 302 outputs a signal necessary for row selection for reading the selected row and reading of the electric charge in each row to the circuit of each pixel in the pixel unit 301. Then, the circuit of each pixel is driven to generate an analog signal at a level corresponding to the exposure amount. An exposure control signal is input to the vertical scanning circuit 302 from the exposure control unit 306, and a mode signal MODE is input from the mode control unit 308 via the control circuit 303. The exposure control signal is a signal for controlling the exposure time.

The image sensor 203 can be driven in any of a detection mode, an image pickup mode, and a cutout mode as a drive mode.

The mode signal MODE is a digital addition information DA indicating a mode flag Mf for switching the drive mode of the image sensor 203 and how the addition data after AD conversion is added by the digital addition unit 311. Has.

The detection mode is a mode in which data (detection image) read by adding pixels of pixels from the pixel unit 301 is used for event detection. The image pickup mode is a mode in which data (image pickup image) read from the pixel unit 301 without pixel addition is output from the image pickup element 203, and the data output from the image pickup element 203 in this mode is recorded as an image or the like. Is done. Further, the cutout mode is a mode in which the pixels of the pixel unit 301 are partially read out without performing pixel addition, that is, the data obtained by cutting out a part of the captured image is used for estimating the focusing position of the subject.

The output signal from the vertical scanning circuit 302 is changed between the detection mode and the imaging mode and the cutout mode. In this embodiment, pixel addition is not performed in the imaging mode and the cutout mode, but for example, a small addition such as addition of two adjacent pixels of the same color may be performed to read from the pixel unit 301.

The AD conversion unit 304 performs AD conversion on an analog signal (addition signal or pixel signal) from the pixel unit 301. Then, the AD conversion unit 304 supplies the generated data (additional data or pixel data) to the switch 310.

The switch 310 switches the output destination of the data from the AD conversion unit 304 according to the mode flag Mf of the mode signal MODE. With this switch 310, it is possible to switch to one of the three terminals S1, S2, and S3, and when the mode flag Mf is set to the detection mode, the switch 310 switches to the terminal S1 and the image data from the AD conversion unit 304. Is supplied to the digital addition unit 311. On the other hand, when the mode flag Mf is set to the image pickup mode, the switch 310 switches to the terminal S3 and outputs the pixel data from the AD conversion unit 304 to the image pickup signal processing circuit 210 in the external signal processing section 21 as an output of the image pickup element 203. Supply. When the mode flag Mf is set to the cutout mode, the switch 310 switches to the terminal S2 and supplies the pixel data from the AD conversion unit 304 to the event detection unit 307.

The event detection unit 307 detects the presence or absence of a predetermined event based on the difference result of the addition data between frames, and generates a detection result DET. For example, when a subject that was not originally in the angle of view of the image pickup module 20 enters the angle of view, the movement of the subject is detected as an event. Further, the event detection unit 307 integrates the addition data corresponding to the entire area of the pixel unit 301 and supplies the addition data as the luminance integration data E to the exposure control unit 306. The luminance integration data E indicates the photometric quantity of the entire image. Details will be described later.

The exposure control unit 306 controls the exposure amount of the pixel unit 301 based on the luminance integration data E. The exposure control unit 306 obtains an appropriate exposure time for the next control based on the brightness integration data obtained by the integration unit in the event detection unit 307, which will be described later, and transmits the information to the control circuit 303.

In this embodiment, the exposure control unit 306 controls the exposure time, but instead of the exposure time, a diaphragm may be provided in the lens 201 to control the diaphragm.

Surrounding environment information is recorded in the memory unit 312. The surrounding environment information is composed of a distance map М, pixel area information P, attention subject information AS, and digital addition information DA. The surrounding environment information is created based on the image data sent from the image pickup device 203 by the image pickup signal processing circuit 210 of the external signal processing section 21, and is stored in the memory section II216. This surrounding environment information is read by the overall control calculation unit 212, and is transferred / stored in the memory unit 312.

Further, the event detection unit 307 detects an event using the surrounding environment information stored in the memory unit 312, and when an event is detected, reflects the result in the surrounding environment information and stores it in the memory unit 312. The details of the surrounding environment information and the method of creating the surrounding environment information will be described later.

The mode control unit 308 sets one of the detection mode, the image pickup mode, and the cutout mode. When the user performs a predetermined operation on an operation unit (not shown) of the system 2 and the image pickup module unit 20 is activated, the mode control unit 308 sets the mode flag Mf of the mode signal MODE to the detection mode. Further, from the surrounding environment information recorded in the memory unit 312, the digital addition information DA is obtained via the event detection unit 307 and written to the mode signal MODE.

This mode signal MODE is supplied to the vertical scanning circuit 302 and the switch 310. Further, the mode control unit 308 supplies the pixel area information P to the digital addition unit 311. The details of the pixel area information P will be described later.

Then, when a predetermined event occurs in the detection mode, the mode control unit 308 sets the mode flag Mf of the mode signal MODE to the imaging mode. In the imaging mode, if the event is not detected for a certain period of time, the mode control unit 308 sets the mode flag Mf of the mode signal MODE to the detection mode.

Analog data is input to the digital addition unit 311 when the switch 310 is in the detection mode. The input analog data is digitally added based on the pixel area information P from the mode control unit 308 and output to the event detection unit 307. The operation of the event detection unit 307 will be described later.

When the event is detected by the event detection unit 307 and the subject position is estimated, the lens control unit 309 sends a lens drive signal to the lens drive unit 202 based on the estimation result.

FIG. 4 is a diagram showing a schematic configuration of a pixel portion 301 in the image pickup device 203 of FIG.

In FIG. 4, the pixel unit 301 has a plurality of pixel blocks 330 and a plurality of pixel blocks 340 having 4 × 2 pixels 330 as one block.

The pixel unit 301 adds pixels in the pixel block 340, and when pixels are similarly added in a plurality of pixel blocks (not shown in FIG. 4), these are digitally added in the subsequent stage, and finally a pixel area of a predetermined size is obtained. Generates data by adding the pixels of. This data is used for event detection. The pixel area is hereinafter referred to as a detection frame.

In the pixel unit 301, a plurality of pixels 330 are arranged in a matrix. Further, a plurality of pixels 330 are gathered to form a pixel block 340.

In addition, in order to simplify the explanation, in this example, the pixel block 340 is arranged as a matrix 4 × 2, but the number of matrices and the arrangement are not limited to this.

The pixel block 340 has two reset control signals (for example, RST1-1 and RST1-2) and two row selection control signals (for example, SEL1-1 and SEL1-2) as horizontal signal lines in the row direction with respect to the block unit. It will be wired. Further, two transfer control signals (for example, TX1-1 and TX1-2), an addition signal ADD (for example, ADD1), and a signal selection control signal ADD_SEL (ADD_SEL1) after addition are wired.

A reset control signal, a row selection control signal, and a transfer control signal are wired in each pixel 330. Further, an addition signal ADD and a signal selection control signal ADD_SEL after addition are wired for each pixel block.

A vertical signal line 410 is wired in the column direction. The output from the pixel is input to the AD conversion unit 304 of the connection destination via the vertical signal line 410.

By performing the selective driving of each of the horizontal signal lines described above, output is sequentially performed from the image sensor line by line via each vertical signal line 410.

In the image pickup mode, the image sensor 203 sequentially outputs from each pixel, whereas in the detection mode, the addition switch for each FD is operated in the block described later, and the addition result is output from a specific pixel.

FIG. 5 is an equivalent circuit diagram of the pixel 330 in the pixel unit 301 shown in FIG. Hereinafter, as shown in FIG. 5, a plurality of pixels 330 are individually referred to as pixels 330-1 to 330-8.

FIG. 5 is a diagram illustrating a configuration example of an analog adder circuit in the detection mode of the image sensor 203, in which four pixels 330-1 to 330-8 arranged in the horizontal direction and two pixels in the vertical direction are arranged as one. It constitutes a pixel block 340.

Focusing on pixel 330-1 on the upper left, the electric charge generated and accumulated by the photodiode 406-1 is floated diffused (hereinafter referred to as FD) via the transfer switch 405-1 by controlling the transfer control signal TX1. Transfer to 407-1. The source follower amplifier 408-1 is configured together with a constant current source 411-1 connected to the vertical output line 410-1, amplifies a voltage signal based on the electric charge accumulated in the FD407-1, and outputs it as a pixel signal. The row selection control signal SEL1-1 controls the row selection switch 409-1 to connect the output of the source follower amplifier 408-1 to the vertical output line 410-1.

When resetting the unnecessary charge stored in the FD407-1, the reset switch 404-1 is controlled by the reset control signal RST1-1. Further, when resetting the photodiode 406-1, the transfer control signal TX1-1 is controlled together with the reset switch 404-1 to control the transfer switch 405-1 to execute the reset. The transfer control signal TX1-1, the reset control signal RST1-1, and the line selection control signal SEL1-1 are connected to the vertical scanning circuit 302, and each line has its own control signal.

When the image sensor 203 having such a configuration is used in the shooting mode, the signal transferred for each pixel 330 is read out by each pixel.

On the other hand, when in the detection mode, the horizontal addition switch 413 (413-1, 413-2, 413-3) and the vertical addition switch 414 (414-1, 414-2, 414) are set via the addition signal ADD1. -3, 414-4) is turned on. As a result, the FDs (407-1, 407-2, 407-3, 407-4, 407-5, 407-6, 407-7, 407-8) are connected in a circuit, and the signal obtained from the pixel 330 is added. Perform averaging. After that, in order to output the data added and averaged by the FD, the addition selection control signal ADD_SEL is controlled and the added result is read out. In this way, the result of addition by the FD of 4 × 2 pixels is used as the output of the detection mode.

In the above description, the configuration of 4 × 2 pixels has been described, but the unit of pixel addition is not limited to the configuration of 4 × 2 pixels as shown in FIG. Further, the pixel addition method is not limited to the addition by FD407-1 (FD addition and described later), the vertical addition is performed by connecting a plurality of lines with a vertical output line (output line addition and described later), and the horizontal addition is an AD conversion unit. It may be implemented by using the addition circuit before 304 (horizontal addition and described later).

FIG. 6 is a diagram showing a circuit configuration of the digital addition unit 311 in the image pickup device 203 of FIG.

When the detection mode is selected by the switch 310, the addition data after AD conversion is input to the integration circuit selection unit 2300.

The integration circuit selection unit 2300 selects the addition data of the pixel region constituting the detection frame based on the digital addition information DA of the mode signal MODE, inputs the addition data to the integration circuit 2301, and integrates the data.

As described above, the digital addition unit 311 can freely perform addition without being restricted by the circuit configuration of the pixel unit 301 of the image sensor 203.

In addition, the integration circuit 2301 is prepared with the number required for integration (for example, when the addition number is two columns in J column, J / 2 is prepared).

Then, when all the analog addition data in the pixel region constituting the detection frame enters the integration circuit 2301, it is stored in each addition data storage unit 2303 through the division unit 2302. When the recording of a certain block line in the addition data storage unit 2303 is completed, the data of the integration circuit 2301 is reset by the RST signal. After that, the integration of the analog addition data in the next row direction is started. Each integrated data is stored in the storage unit 2305 for one frame, respectively.

As for the addition data stored in each addition data storage unit 2303, the data selected by the addition data selection unit 2304 is sequentially transferred to the event detection unit 307.

FIG. 7 is a block diagram showing the configuration of the event detection unit 307 in the image pickup device 203 of FIG.

The event detected by the event detection unit 307 is motion detection of a subject not limited to the subject of interest, and cannot be detected only by the movement of the subject of interest.

The event detection unit 307 includes an addition data holding unit A701, an addition data holding unit B702, a difference calculation unit 703, an integration unit 704, and a comparison unit 705.

The addition data holding unit A701 is a holding unit that first holds the analog or digitally added data.

The addition data holding unit B702 moves the addition data held by the addition data holding unit A701 from the addition data holding unit A701 as information of the previous frame before the addition data of the next frame comes to the addition data holding unit A701. To do.

The integration unit 704 integrates the addition data corresponding to the entire area of the pixel unit 301. The integration unit 704 supplies the integrated data to the exposure control unit 306 as luminance integration data E.

The difference calculation unit 703 calculates the difference between the current addition data from the addition data holding unit A701 and the past luminance integration data from the addition data holding unit B702. The difference calculation unit 703 supplies the calculated difference to the comparison unit 705.

The comparison unit 705 compares the difference supplied from the difference calculation unit 703 with a predetermined threshold value. As a method of detecting the difference, the difference of the added data of each detection frame may be integrated, or the maximum value of the difference between the added data of each detection frame may be used as a representative difference value and compared with the threshold value. The comparison unit 705 supplies the comparison result as a detection result DET to the mode control unit 308. For example, when the difference exceeds the threshold value, it is detected that the subject has a movement (that is, there is an event), and when it does not, it is detected that the subject has no movement. Here, assuming that num is a value indicating the detection frame number, when the detection result DET = (1, num), there is an event in the detection frame where the detection frame number is num, and when the detection result DET = (0, −), the pixel unit 301 There will be no events in the entire area.

In this embodiment, the data of the current frame and the data of the past frame of each detection frame are retained, but the configuration is not limited to this. For example, the integrated data may be compared with the data of the current frame and the data of the past frame to obtain a difference value. Further, in the case of the above-mentioned integration, the weight to be multiplied by the addition data may be held for each addition data.

Further, although the event detection unit 307 detects the movement of the subject as an event, the event to be detected is not limited to the movement of the subject as long as it can be detected by using the added data. The event detection unit 307, for example, may perform image recognition to detect the presence or absence of a specific object such as a person.

An example of setting the detection frame in this embodiment will be described with reference to FIG.

FIG. 8A shows a method of generating additional data output from the AD conversion unit 304 of FIG. 3 in this embodiment. For example, when there are 1920 pixels in the horizontal direction and 1600 pixels in the vertical direction in the pixel area in the pixel unit 301, the pixel areas of 8 horizontal pixels and 160 vertical pixels are added in the pixel unit 301. Then, it is output from the AD conversion unit 304 as the addition data of 2400 at 240 × 10, and further input to the digital addition unit 311 via the changeover switch 310.

The addition data of FIG. 8A is input to the digital addition unit 311, and the addition data is further digitally added by 2 pixels in the vertical direction and 40 pixels in the horizontal direction by the digital addition unit 311 to add 6 × 5. It is output as data. That is, as shown in FIG. 8B, the area in which the pixel area of the pixel unit 301 is equally divided into 6 in the horizontal direction and 5 in the vertical direction (hereinafter referred to as the divided area) is set as the detection frame. Then, the pixels of 320 pixels × 320 pixels in each detection frame are added to obtain one output value. When the difference between frames of this output value is equal to or greater than the threshold value, it is detected as an event.

FIG. 9A is a diagram showing an example of a fixed subject photographed by the imaging module unit 20 in this embodiment.

As shown in FIG. 9A, the subject photographed by the image pickup module unit 20 includes a left wall 901, a ceiling 902, a right wall 903, a floor 904, a back wall 905, a door 906, and a window 907. The peripheral part of the angle of view is a near view, and the back wall 905 is a distant view.

9B is a diagram showing the positional relationship between the plurality of divided regions shown in FIG. 8B and the imaging region of the fixed subject shown in FIG. 9A. The dotted line shown in FIG. 9B indicates the boundary line of each divided region. Further, the numbers 1 to 30 described in the upper left of each divided area indicate the detection frame number num of each divided area.

Next, the surrounding environment information will be described. The surrounding environment information is composed of a distance map М, pixel area information P, attention subject information AS, and digital addition information DA.

The distance map М is distance data indicating the distance l from the image pickup module unit 20 of the subject imaged in the entire pixel region of the pixel unit 301, and is represented as follows.
М (i, j)
i: Horizontal coordinates in the pixel area of the pixel unit 301 i = 0 to 1919
j: Vertical coordinates in the pixel area of the pixel unit 301 j = 0 to 1599
The pixel area information P is information indicating the position of the pixel area in which the detection frame exists.

In this embodiment, not only each divided region of the pixel unit 301 is set as a detection frame as described above, but also a fixed subject having a size equal to or larger than a predetermined value is set as a detection frame.

The pixel area information P includes horizontal coordinates x at the upper left of the pixel area where the detection frame exists, vertical coordinates y (expressed as coordinates with the upper left of the angle of view as (0,0)), horizontal size h of the detection frame, and detection frame. It is composed of the vertical size v and the identification symbol id of, and is represented as follows.
Pnum (x, y, h, v, id) num: Natural number

For the detection frame number num, a natural number smaller than 100 is used for the position of each divided region of the pixel portion 301 set as the detection frame, and a natural number of 100 or more is used for the position of the fixed subject set as the detection frame. ..

The identification symbol id is used when the size of the fixed subject set as the detection frame is larger than a predetermined value, but one fixed subject is specified when the distance l from the image pickup module 20 is within a predetermined range. It is a symbol used. That is, when a plurality of pixel area information Ps have the same identification symbol id, one fixed subject set as a detection frame by the plurality of pixel area information Ps is described.

For example, the divided region at the upper left of the angle of view shown in FIG. 9B is the detection frame of the detection frame number 1, and the pixel area information P thereof is represented as follows.
P1 (0,0,320,320,)

Further, since the imaging region of a fixed subject such as a desk shown in FIG. 9C has a size equal to or larger than a predetermined value, the imaging region is set as the detection frame of the detection frame number 104, and the pixel region information P is as follows. It is expressed as.
P104 (1000, 1000, 300, 300,)

On the other hand, the door 906 shown in FIG. 9C is a fixed subject whose imaging region has a size equal to or larger than a predetermined value, similar to the desk described above. However, since the distance l from the image pickup module unit 20 of the door 906 extends over a predetermined range (a wide range of perspective) and cannot be represented by a single distance map M, the image pickup region is detected as one. Cannot be set as a frame. In such a case, a plurality of feature points for which the distance l is the same and the position of the door 906 is specified are set as the detection frame.

For example, since the feature points where the distance l is the same and the position of the door 906 is specified are the left and right ends thereof, the left end of the imaging region of the door 906 is set as the detection frame of the detection frame number 100, and the right end thereof is set. It is set as the detection frame of the detection frame number 101. Here, in this embodiment, the horizontal coordinate x and the vertical coordinate y at the upper left end of the imaging region of the door 906 are 160 pixels and 680 pixels, respectively, and the height of the left end is 640 pixels. Further, the upper right end thereof has 320 pixels and 660 pixels in the horizontal coordinate x and the vertical coordinate y, respectively, and the height of the right end thereof is 460 pixels. In this case, the pixel area information P of each of the detection frame numbers 100 and 101 is represented as follows.
P100 (160,680,0,640,0) (Left end of door 906)
P101 (320,660,0,460,0) (Right end of door 906)

Similarly, in the window 907 shown in FIG. 9C, a plurality of feature points in the imaging region are set as detection frames, respectively. Specifically, the left end of the imaging region of the window 907 is set as the detection frame of the detection frame number 102, and the right end thereof is set as the detection frame of the output frame number 103. Each pixel area information P is represented as follows.
P102 (1360, 560, 0, 320, 1) (left end of window 907)
P103 (1600,520,0,440,1) (right end of window 907)

The attention subject information AS is a attention subject calculated based on the number of times that a subject of interest such as a person is detected in the past in the detection frame represented by the pixel area information P in which neither the horizontal size h nor the vertical size v is 0. This is information indicating the ease of appearance of.

Specifically, the attention subject information AS includes a detection frame number num, a change rate c of a distance l of the attention subject from the imaging module unit 20, an appearance rate a of the attention subject, a disappearance rate d of the attention subject, a distance l, and a weight. It is composed of w and is represented as follows.
AS (num, c, a, d, l, w)

The rate of change c of the distance l from the image pickup module unit 20 of the subject of interest is the standard deviation of the value of the distance l from the past to the present.

The appearance rate a of the attention subject is the number of occurrences of an event (appearance event) in which the attention subject (person, etc.) has appeared in the detection frame specified by the detection number num from the past to the present. It is a value divided by the number of occurrences of the appearance event of the subject (person, etc.).

Further, the disappearance rate d of the attention subject is the number of occurrences of an event (disappearance event) in which the attention subject (person, etc.) disappears in the detection frame specified by the detection number num from the past to the present in all angles of view. It is a value divided by the number of occurrences of disappearance events of the subject of interest (people, etc.).

The weight w indicates the sum of the rate of change c, the rate of appearance a, and the rate of disappearance d in the detection frame specified by the detection number num. In this embodiment, the weight w is the sum of the rate of change c, the rate of appearance a, and the rate of disappearance d, but it may be a value indicating at least one of the rate of change c, the rate of appearance a, and the rate of disappearance d. ..

How should the digital addition information DA add the addition data output from the AD conversion unit 304 to the digital addition unit 311 in order to generate data obtained by adding the pixels of the detection frame represented by the pixel area information P? This is information indicating.

The event detection process in this embodiment will be described with reference to FIGS. 1A to 1E.

Hereinafter, an event detection processing operation when a subject of interest such as a person appears on the fixed subject shown in FIG. 9A within the angle of view of the imaging module unit 20 as shown in FIG. 10A will be described as an example.

FIG. 1A is a flowchart showing an outline of the event detection process according to the present embodiment. This process is executed by the overall control calculation unit 212 reading a program stored in the memory unit II 216 and controlling the image sensor 203 based on the program.

In step S101, when the user performs a predetermined operation on an operation unit (not shown) of the system 2 and the image pickup module unit 20 is activated, the image sensor 203 is initially set. Specifically, the mode control unit 308 sets the mode flag Mf of the mode signal MODE to the detection mode, and sends the mode signal MODE to the control circuit 303, the switch 310, and the digital addition unit 311. The switch 310 sets the output destination of the data from the AD conversion unit 304 to the terminal S1 based on the mode flag Mf of the mode signal МODE being set to the detection mode.

In step S102, the image pickup signal processing circuit 210 of the external signal processing section 21 creates peripheral environment information based on the image data generated from the RAW data from the image pickup element 203, and stores it in the memory section II216. After that, the overall control calculation unit 212 reads out the surrounding environment information stored in the memory unit II 216, that is, the distance map М, the pixel area information P, the attention subject information AS, and the digital addition information DA, and the memory unit 312 in the image sensor 203. To get the latest information on the surrounding environment. When the step S102 is executed again after executing the step S110 described later, the peripheral environment information already transferred from the memory unit II 216 is stored in the memory unit 312. In such a case, only the difference data from the previously transferred surrounding environment information is transferred from the memory unit II 216 to the memory unit 312.

In step S103, the detection mode operation process for driving the image sensor 203 in the detection mode is executed. Details of this detection mode operation process are shown in FIG. 1B.

In FIG. 1B, in step S103-1, the control circuit 303 drives and controls the vertical scanning circuit 302 in the detection mode. Then, based on the drive pulse from the vertical scanning circuit 302, the pixel unit 301 performs vertical addition and FD addition of the signal obtained from the pixel 330. Further, by driving and controlling the AD conversion unit 304 and the horizontal scanning circuit 305 in the detection mode by the control circuit 303, the above-mentioned vertical addition and FD addition signals are AD-converted and then horizontally added by the pixel unit 301. , Generate an additional image. The generated added image is then supplied to the digital adding unit 311. Next, the control circuit 303 drives and controls the digital addition unit 311 in the detection mode, so that the added image supplied by the digital addition unit 311 is further digitally added and supplied to the event detection unit 307.

In step S103-2, the event detection unit 307 detects the presence or absence of a predetermined event between frames based on the added image from the AD conversion unit 304. After that, the detection result DET and the luminance integration data E are obtained as outputs from the event detection unit 307, and this process is terminated.

Returning to FIG. 1A, in step S104, when the detection result DET is (1, num), the event detection unit 307 determines that an event, that is, the movement of the subject of interest has been detected, and proceeds to step S105. On the other hand, if the detection result DET is (0, −), the process returns to step S103. Hereinafter, as shown in FIG. 10B, when an event is detected in the divided area of the detection frame number 20 (the subject of interest is reflected in the divided area of the detection frame number 20), that is, the detection result DET is (1,20). The case where is described will be described.

In step S105, the attention subject information AS (20, c20, a20, d20, l20, w20) of the detection frame in which the event is detected is read from the memory unit 312. After that, the distance l20 from the image pickup module unit 20 of the subject located in the divided region of the detection frame number 20 which is the detection frame in which the event is detected is firstly estimated to be the distance lt to the subject of interest.

In step S106, a control signal is sent from the lens control unit 309 to the lens drive unit 202 so that the focal length matches the primary estimated distance lt (that is, the distance l20), and the focus lens of the lens 201 is driven.

In step S107, the attention subject position correction process for correcting the position of the attention subject (distance lt to the attention subject) is executed. Details of this attention subject position correction process are shown in FIG. 1D.

In FIG. 1D, in step S107-1, the event detection unit 307 sends the detection result DET to the mode control unit 308, and the pixel area information P20 (x, y, h, v,) is sent to the mode control unit 308. In the mode control unit 308, when the detection result DET (1,20) and the pixel area information P20 (x, y, h, v,) are sent, the mode flag Mf of the mode signal МОDE is set to the cutout mode. To do. After that, the mode signal МОDE is sent to the control circuit 303, the switch 310, and the digital adder 311. The switch 310 sets the output destination of the data from the AD conversion unit 304 to the terminal S2 based on the mode flag Mf of the mode signal МODE being set to the cutout mode.

Next, the control circuit 303 drives and controls the vertical scanning circuit 302 and the AD conversion unit 304 in the cutout mode. From the signal obtained from the pixel 330 as a result, for each detection frame of the detection frame number 20, which is a detection frame designated based on the pixel area information P20 (x, y, h, v,), and the detection frames around the detection frame, Image data without pixel addition is generated. The generated image data is supplied from the AD conversion unit 304 to the event detection unit 307 via the switch 310.

In step S107-2, the attention subject position re-estimation process for re-estimating the attention subject position from the image data generated in step S107-1 is executed. The details of this attention subject position re-estimation process are shown in FIG.

In FIG. 11, in step S107-21, the image of the subject of interest is extracted from the detection frame of the detection frame number 20 read from the image data generated in step S107-1 and the detection frame around the detection frame.

In step S107-22, it is determined whether or not the foot position of the subject of interest can be confirmed based on the extracted image, and if it can be confirmed, the process proceeds to step S107-26, and if it cannot be confirmed, the process proceeds to step S107-23.

In step S107-23, the respective weights w are read from the detection frame number 20 read from the memory unit 312 and the attention subject information AS of the detection frames around the detection frame number 20 and the attention subject information AS around the detection frame number 20. After that, the detection frame in which the value of the read weight w is equal to or greater than the threshold value wth is determined to be the priority appearance / disappearance frame of the subject of interest, and the process proceeds to step S107-25. On the other hand, if the values of the read weights w are all less than the threshold value wth, the process proceeds to step S107-24.

In step S107-24, it is estimated that the subject of interest is at the position of the detection frame of the detection frame number 20. That is, it is estimated that the subject of interest is reflected in the detection frame of the detection frame number 20. Then, the distance l20 included in the attention subject information AS (20, c20, a20, d20, l20, w20) of the detection frame of the detection frame number 20 is re-estimated as the distance lt to the attention subject, and this process is terminated.

In step S107-25, for example, when the weight w of the detection frame of the detection frame number 19 is equal to or greater than the threshold value wth and it is determined that the detection frame of the detection frame number 19 is the priority appearance / disappearance frame of the subject of interest, the detection frame It is estimated that the subject of interest is at the position of the detection frame of number 19. That is, it is estimated that the subject of interest is reflected in the detection frame of the detection frame number 19. Then, the distance l19 included in the attention subject information AS (19, c19, a19, d19, l19, w19) of the detection frame of the detection frame number 19 is re-estimated as the distance lt to the attention subject, and this process is terminated.

In steps S107-26, the distance lt from the confirmed foot position of the subject of interest to the subject of interest is re-estimated. Specifically, first, the coordinates of the foot position of the confirmed subject of interest are calculated. Hereinafter, the case where the coordinates are calculated as (480, 1280) will be described. At this time, the distance map М is read from the memory unit 312, the distance of the floor 904 corresponding to the value of the y coordinate of the foot position is searched for, the value is set as the distance lt to the subject of interest, and this process is terminated.

In step S107-26, the distance lt to the corresponding subject of interest is searched from the distance map М, but the position information from which the foot position can be estimated may be searched from the pixel area information P. For example, according to the pixel area information P100 (160,680,0,640,0) indicating the left end, which is one of the feature points of the door 906, the coordinates of the upper end of the left end of the door 906 are (160,680). It can be seen that the coordinates of the lower end are (160,1280). In this case, since the y coordinate of the lower left end (160, 1280) of the door 906 coincides with the foot position of the subject of interest (480, 1280), the distance lt'to the subject of interest is the distance l100 to the left end of the door 906. You may infer that it matches.

Returning to FIG. 1D, in step S107-3, the target focus position is corrected from the distance l20 to the attention subject first estimated in step S105 to the distance lt re-estimated by the attention subject position re-estimation process in step S107-2. .. After that, a control signal is sent from the lens control unit 309 to the lens drive unit 202 so that the focal length matches the corrected distance lt, the focus lens of the lens 201 is driven, and this process is completed.

Returning to FIG. 1A, in step S108, the number of loops Ta for which there is no change in the subject is reset, and Ta = 0.

In step S109, an image pickup mode operation process for operating the image pickup element 203 in the image pickup mode is executed. Details of this imaging mode operation process are shown in FIG. 1C.

In FIG. 1C, in step S109-1, high-definition image data is generated by photographing. Specifically, first, the mode control unit 308 sets the mode flag Mf of the mode signal MODE to the imaging mode, and the mode control unit 308 sends the mode signal МОDE to the control circuit 303, the switch 310, and the digital addition unit 311. The switch 310 sets the output destination of the data from the AD conversion unit 304 to the terminal S3 based on the mode signal МODE being set to the imaging mode. Next, the control circuit 303 drives and controls the vertical scanning circuit 302 and the AD conversion unit 304 in the imaging mode. As a result, high-definition image data in which the entire area of the pixel unit 301 is not added is generated from the signal obtained from the pixel 330. The generated high-definition image data is supplied from the AD conversion unit 304 to the image pickup signal processing circuit 210 via the switch 310. The image pickup signal processing circuit 210 detects the subject of interest from the supplied high-definition image data. When the subject of interest can be detected, the appearance rate a of the subject of interest and the disappearance rate d of the subject of interest are calculated.

In step S109-2, the high-definition image data (captured image) obtained in the image pickup signal processing circuit 210 is compared with the high-definition image data obtained in the processing of the previous frame. As a result of this comparison, if there is a difference (change) in the data between frames, the process proceeds to step S109-3, and if not, the process proceeds to step S109-6.

In step S109-3, the contrast AF operation is performed. Specifically, the overall control calculation unit 212 issues a command to the lens drive unit 202 to move the focus position of the lens 201, generate a plurality of high-definition image data in the same manner as in step S109-1, and process the imaging signal. Output to circuit 210. The image pickup signal processing circuit 210 estimates the focusing position of the subject of interest based on the plurality of output high-definition image data. Then, a control signal is sent from the overall control calculation unit 212 to the lens drive unit 202 via the lens control unit 309 so that the focus position is aligned with the estimated focusing position, and the focus lens of the lens 201 is driven.

In step S109-4, the peripheral environment information creation / update process for creating / updating the peripheral environment information is executed based on the plurality of high-definition image data obtained in step S109-3. The details of the surrounding environment information creation / update process are shown in FIG. 1E.

In FIG. 1E, in steps S109-41, the plurality of high-definition image data obtained in step S109-3 are processed by the imaging signal processing circuit 210 to create a distance map М. Then, the created distance map М is stored in the memory unit II216.

In steps S109-43, the peripheral image information is updated. Specifically, the overall control calculation unit 212 reads the distance map М from the memory unit II 216, updates the pixel area information P and the distance l, and sets the difference of the distance l before and after the update as the rate of change c of the distance l. Update. Further, when the attention subject can be detected from the high-definition image data obtained in step S109-1, the appearance rate a and the disappearance rate d of the attention subject in the detection frame in which the attention subject is detected are updated. The sum of the rates c is updated as the weight w. Further, the digital addition information DA is updated based on the updated pixel area information P. By this series of processing, the distance map М, the pixel area information P, the attention subject information AS, and the digital addition information DA, which constitute the surrounding environment information, are created and updated, and the distance l, the appearance rate a, and the disappearance rate are created for each detection frame. When d and the rate of change c are linked, this process ends.

Returning to FIG. 1C, in step S109-5, Ta = 0 is set, and this process is terminated.

On the other hand, in step S109-6, Ta = Ta + 1 is set, and this process is terminated.

Returning to FIG. 1A, in step S110, when Ta ≧ Tth, the process proceeds to step S102. If not, the process returns to step S109, updates the surrounding environment information, shifts to the detection mode again, and executes the detection mode operation process in step S103.

As described above, in the first embodiment, in the system 2 having the event detection function, the weight w and the distance l are associated with each of the plurality of detection frames set in the pixel area of the pixel unit 301. After that, the focus position is first adjusted to the distance l associated with the detection frame detected that the subject of interest is reflected in the detection mode. After that, since the focus position is corrected to the distance l of the detection frame determined to be the priority appearance / disappearance frame of the attention subject based on the weight w, the AF processing can be performed accurately at the same time as the detection of the attention subject. ..

However, in the above method, since the focus position is adjusted to the distance l associated with each detection frame, erroneous AF processing may be performed.

For example, in FIG. 9B, as shown in FIG. 15A, the subject of interest is located in the closest view region in a state of hanging from the ceiling 902, while the subject of interest is reflected in the detection frame of detection frame number 15. think of. In this case, in the first embodiment, as a result of the primary estimation in step S105, the focal length matches the distance l indicating the distant view region associated with the detection frame of the detection frame number 15. Further, even if the foot position of the subject of interest can be confirmed in step S107-22, in the case of this example, since the subject of interest does not have his / her foot on the floor 904, the focus position cannot be corrected correctly based on the foot position.

Therefore, a method capable of performing AF processing with high accuracy even when the subject of interest shown in FIG. 15A is detected will be described in the second embodiment below.

[Example 2]
Hereinafter, this embodiment will be described with reference to FIGS. 1A to 1C, 1E, 2 to 7, 8 (a), 9A, 9C, and 12 to 16.

In this embodiment, the divided area in the first embodiment is an area in which the pixel area of the pixel unit 301 is evenly divided, whereas the divided area is set according to the subject distance. .. That is, since the hardware configuration of the system 2 shown in FIGS. 2 to 7 is the same as that of the first embodiment, the same configuration is designated by the same reference numerals, and duplicate description will be omitted.

First, an example of setting the detection frame in this embodiment will be described with reference to FIGS. 8A and 13A to 13C.

Also in this embodiment, in the detection mode, the addition data shown in FIG. 8A is output from the AD conversion unit 304 of FIG. 3 and input to the digital addition unit 311 via the changeover switch 310.

However, in this embodiment, the digital addition unit 311 performs digital addition corresponding to the division regions of different sizes shown in FIG. 13B in the pixel region of the pixel unit 301, and outputs the input addition data. To.

The digital addition number of each division region in FIG. 13B is shown in FIG. 13A.

Of the divided areas in FIG. 13B, in the right diagonal stripe area (the area painted in the striped pattern from the upper right to the lower left), the addition data is further digitally added by 20 pixels in the horizontal direction and output by the digital addition unit 311. Will be done. Finally, the pixels in the detection frame (right diagonal stripe area) composed of 160 pixels x 160 pixels are added to obtain one output value, and an event occurs when the difference between frames of this output value is equal to or greater than the threshold value. Is detected as. A detection frame of this size is hereinafter referred to as a 1x frame.

Of the divided regions in FIG. 13B, in the vertical stripe region (the region painted in a striped pattern from top to bottom), the digital addition unit 311 further digitally adds 2 pixels in the vertical direction and 40 pixels in the horizontal direction. It is added and output. Finally, the pixels in the detection frame (vertical stripe area) composed of 320 pixels × 320 pixels are added to obtain one output value, and when the difference between the frames of this output value is equal to or greater than the threshold value, it is an event. Detected. A detection frame of this size is hereinafter referred to as a double frame.

Of the divided areas in FIG. 13B, in the left diagonal stripe area (the area painted in the striped pattern from the upper left to the lower right), the digital addition unit 311 further adds 3 pixels in the vertical direction and 60 pixels in the horizontal direction. Is digitally added and output. Finally, the pixels in the detection frame (left diagonal stripe area) composed of 480 pixels × 480 pixels are added to obtain one output value, and an event occurs when the difference between the frames of this output value is equal to or greater than the threshold value. Is detected as. A detection frame of this size is hereinafter referred to as a triple frame.

Further, the surrounding environment information is composed of the distance map М, the pixel area information P, the attention subject information AS, and the digital addition information DA as in the first embodiment, and the imaging signal processing is performed based on a plurality of high-definition image data. It is created and updated by the circuit 210 and the overall control calculation unit 212. Further, as in the first embodiment, the created / updated surrounding environment information is sent to the memory unit 312 in the image sensor 203.

In the above configuration, a case where the event detection process of this embodiment is executed in the same subject environment as in FIG. 9A as in the first embodiment will be described below.

At this time, FIG. 13C shows the allocation of the detection frame number num of each divided area when the pixel area of the pixel unit 301 is divided into 36 as shown in FIG. 13B. Similar to FIG. 9B, the dotted line indicates the boundary line of each divided region. Further, the numbers 1 to 36 described in the upper left of each divided area indicate the detection frame number num of each divided area.

As for the fixed subject, the following is recorded as the pixel area information P as in FIG. 9C.
P100 (160,680,0,640,0) (Left end of door 906)
P101 (320,660,0,460,0) (Right end of door 906)
P102 (1360, 560, 0, 320, 1) (left end of window 907)
P103 (1600,520,0,440,1) (right end of window 907)
P104 (1000, 1000, 300, 300,) (Desk 104)

The event detection process in this embodiment will be described with reference to FIGS. 1A to 1C, FIGS. 1E and 12.

Hereinafter, on the fixed subject shown in FIG. 9A, as shown in FIG. 15A, a subject of interest such as a person hanging from the ceiling with a string and near a near view of the screen appears within the angle of view of the imaging module unit 20. The event detection processing operation in the case will be described as an example.

Among the event detection processes in the present embodiment, the processes shown in FIGS. 1A to 1C and 1E are the same as the event detection processes in the first embodiment. However, in this embodiment, the attention subject position correction process of FIG. 12A is executed instead of the attention subject position correction process of FIG. 1D. This will be described in sequence below.

Since the operations of steps S101 to S103 are the same as those of the first embodiment, the description thereof will be omitted.

In step S104, when the detection result DET is (1, num), the event detection unit 307 determines that an event, that is, the movement of the subject of interest has been detected, and proceeds to step S105. On the other hand, if the detection result DET is (0, −), the process returns to step S103. Hereinafter, as shown in FIG. 15B, when an event is detected in each of the detection frame numbers 18, 19, 25, and 26, that is, the detection result DETs are (1,18) and (1,19). , (1,25), (1,26) will be described.

In step S105, the attention subject information AS (num, cnum, anum, dnum, lnum, wnum) of the detection frame of the detection frame numbers 18, 19, 25, 26 in which the event is detected is read from the memory unit 312.

Next, the average of the distances l18, l19, l25, l26 from the image pickup module 20 of the subject located in the detection frame of the detection frame numbers 18, 19, 25, 26 where the event is detected is the distance lt to the subject of interest. It is first-order estimated to be.

In step S106, the average value of the distances l18, l19, l25, and l26 is first-order estimated as the distance lt to the subject of interest. After that, a control signal is sent from the lens control unit 309 to the lens drive unit 202 so that the focal length matches the primary estimated distance lt to the subject of interest, and the focus position of the focus lens of the lens 201 is moved.

In step S107, the attention subject position correction process for correcting the position of the attention subject is executed. Details of this attention subject position correction process are shown in FIG. 12A.

In FIG. 12A, in step S107-4, a plurality of detection frames (detection frames of detection frame numbers 18, 19, 25, 26 in this embodiment) in which an event (object of interest) was detected in step S104 are adjacent to each other. Judge whether or not. Specifically, the event detection unit 307 reads out the pixel area information P corresponding to the detection frames of the detection frame numbers 18, 19, 25, and 26, and determines whether or not the respective areas are adjacent to each other.

Next, if they are adjacent to each other, the process proceeds to step S107-5, and the target focus position is corrected from the distance l20 to the subject of interest first estimated in step S105 to the distance lt to the subject of interest according to the adjacent method. To do.

FIG. 14 is a diagram showing a method of re-estimating the distance lt to the subject of interest with respect to the adjacent detection frame in step S107-5 of FIG. 12A.

As shown in FIG. 14A, when the 1x frames are adjacent to each other and the size is the same as the 2x frame in any of the vertical, horizontal, and diagonal directions, the imaging module linked to the 2x frame. The distance lt'from the unit 20 is re-estimated as the distance lt to the subject of interest.

As shown in FIG. 14B, when the 1x frames are adjacent to each other and the size is the same as the 3x frame in any of the vertical, horizontal, and diagonal directions, the imaging module linked to the 3x frame. The distance lt'from the unit 20 is re-estimated as the distance lt to the subject of interest.

If the 1x frame and the 2x frame are adjacent to each other as shown in FIG. 14C and the vertical, horizontal, or diagonal directions are the same size as the 3x frame, they are linked to the 3x frame. The distance lt'from the image pickup module unit 20 is re-estimated as the distance lt to the subject of interest.

When an event is detected in the four adjacent 1x frame detection frames of detection frame numbers 18, 19, 25, and 26 as in this embodiment, it is equivalent to the double frame in any of the vertical, horizontal, and diagonal directions. It is the size of. Therefore, the distance lt'from the image pickup module unit 20 associated with the double frame is re-estimated as the distance lt to the subject of interest.

In step S107-5, the distance lt'to the subject of interest first-estimated in step S105 is corrected to the distance lt're-estimated in step S107-4. After that, a control signal is sent from the lens control unit 309 to the lens drive unit 202 so that the focal length matches the corrected distance lt, the focus position of the focus lens of the lens 201 is moved, and this process is completed.

Returning to FIG. 1A, in step S108, the number of loops Ta for which there is no change in the subject is reset, and Ta = 0.

In step S109, an image pickup mode operation process (FIG. 1C) for operating the image pickup element 203 in the image pickup mode is executed.

In FIG. 1C, the operation of steps S109-1 to S109-3 is the same as that of the first embodiment, and thus the description thereof will be omitted.

In step S109-4, the peripheral environment information creation / update process for creating / updating the peripheral environment information is executed based on the plurality of high-definition image data obtained in step S109-3. The details of the surrounding environment information creation / update process are shown in FIG. 12B.

In FIG. 12B, first, the distance map М is created in step S109-41. This operation is the same as the operation of step S109-41 of FIG. 1E of the first embodiment.

In step S109-42, the area setting of the detection frame is updated based on the distance map М created in step S109-41.

A method of updating the area setting of the detection frame, which is executed in step S109-42 of FIG. 12E of this embodiment, will be described with reference to FIG.

First, the distance map М created in step S109-41 is read from the memory unit II216.

Then, as shown in FIG. 16A, the entire pixel region of the pixel unit 301 is divided by a detection frame having a size of a 1x frame. In the case of this embodiment, it is divided into a total of 120 frames, 12 horizontal frames and 10 vertical frames. Then, the average value of the distances from the image pickup module 20 of each detection frame is assigned as the distance lk (k = 0 to 119) from the image pickup module 20 in that region.

Based on the distance lk assigned above, each detection frame is classified into one of the three regions. The three areas are the area farthest from the image pickup module 20 (distant view), the area close to the image pickup module 20 (near view), and the distance from the image pickup module 20 between the distant view and the near view. Distance (middle view). A distant view, a middle view, and a near view are defined as regions in which the distance from the image pickup module unit 20 is in the following range.
Distant view: lf1 ≤ lk
Middle view: lm1 ≤ lk <lm2
Foreground: 0 ≤ lk <ln2
(Ln2, lm1, lm2, lf1 are all 0 or more, and lm1 = ln2, lf1 = lm2)

By the above processing, 120 detection frames that divide the entire pixel region of the pixel unit 301 are classified as shown in FIG. 16B.

In FIG. 16B, the detection frame with I is the detection frame in which the distance lk from the image pickup module 20 is in the range of the distant view, and the detection frame with II is the distance from the image pickup module 20. lk is a detection frame in the range of the middle view. Further, the detection frame with III is a detection frame in which the distance lk from the image pickup module unit 20 is in the near view range.

After all 120 detection frames that divide the entire pixel area of the pixel unit 301 are classified as shown in FIG. 16B, a double frame detection frame is set using the detection frame classified in the middle view. .. At that time, a detection frame classified into a part of the distant view may be included.

Further, the detection frame of the triple frame is set by using the detection frame classified into the near view. At that time, a detection frame classified into a part of the middle view may be included.

By the above processing, the detection area is set as shown in FIG. 13B.

At this time, the distance from the image pickup module unit 20 assigned to the distant view is the average value of the distances from the image pickup module unit 20 of the detection frame designated as I. The distance from the image pickup module 20 assigned to the middle view is the average value of the distances from the image pickup module 20 of the detection frame labeled II. The distance from the image pickup module 20 assigned to the near view is the average value of the distances from the image pickup module 20 of the detection frame labeled III.

Returning to FIG. 12B, the peripheral image information is updated in steps S109-43. This operation is the same as the operation of step S109-41 of FIG. 1E of the first embodiment. After that, this process ends.

After that, the process returns to FIG. 1C, the operations of steps S109-5 and S109-6 are performed in the same manner as in the first embodiment, and then the operation of step S110 of FIG. 1A is executed.

In this embodiment, the area of the detection area is set as shown in FIG. 13B according to the distance from the image pickup module unit 20, but this area takes into consideration the size projected on the angle of view in the distant view and the near view of the subject of interest. And set.

As described above, in this embodiment, the pixel area of the pixel unit 301 is divided into a plurality of detection areas according to the distance from the image pickup module unit 20, and the divided detection area and the focus position are linked to perform detection. AF can be started at the same time.

Further, even if the subject in the near view area is reflected in the detection frame of the detection frame number 15 in FIG. 9B and the foot position is unknown, the subject position is changed to the near view depending on the size of the area in which the subject is captured. It can be determined that there is, and the focus position can be corrected.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

[Other Examples]
Needless to say, the object of the present invention can also be achieved by supplying the device with a storage medium in which the program code of the software that realizes the functions of the above-described embodiment is recorded. At this time, the computer (or CPU or MPU) including the control unit of the supplied device reads and executes the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the program code itself and the storage medium storing the program code constitute the present invention.

As the storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, or the like can be used.

Further, based on the instructions of the above-mentioned program code, the OS (basic system or operating system) running on the device performs a part or all of the processing, and the processing realizes the functions of the above-described embodiment. Needless to say, cases are also included.

Further, the program code read from the storage medium may be written in the memory provided in the function expansion board inserted in the device or the function expansion unit connected to the computer to realize the functions of the above-described embodiment. Needless to say, it is included. At this time, based on the instruction of the program code, the function expansion board, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing.

2 System 20 Image module unit 21 External signal processing unit 201 Lens 202 Lens drive unit 203 Image sensor 210 Image sensor processing circuit 211 Memory unit I
212 Overall control calculation unit 213 Recording medium control I / F unit 214 Recording medium 215 External I / F unit 216 Memory unit II

Claims (14)

A dividing means for dividing the pixel area of the image sensor into a plurality of divided areas, and
In the detection mode, a first generation means for generating a detected image by adding pixels of pixels in the plurality of divided regions for each divided region,
In the detection mode, a detection means that compares the detection images created by the first generation means between frames and detects a difference in each of the plurality of divided regions as a subject of interest.
In the image pickup mode, a second generation means for generating an image pickup image from the image pickup element and
AF means to move the focus position of the focus lens and
The linking means for linking the distances to the fixed subject reflected in each of the plurality of divided areas, and the linking means.
When the attention subject is detected by the detection means, the detection means includes an estimation means for estimating the distance to the attention subject based on the distance associated with the divided region in which the attention subject is reflected.
An imaging device characterized in that the focus position of the focus lens is moved by the AF means so that the focal length matches the estimated distance, and then the mode shifts to the imaging mode. When the dividing means has a first fixed subject over a wide range of perspective within the angle of view, the area of the first fixed subject is set as one of the plurality of divided areas.
The imaging device according to claim 1, wherein the linking means links the distance of a plurality of feature points at which the position of the first fixed subject is specified as the distance to the first fixed subject. The linking means further links weights for each of the plurality of divided regions.
When there is a divided region in which the subject of interest is reflected and the divided region around the divided region in which the weight associated with the subject is equal to or greater than a threshold value, the estimated distance associated with the divided region is used. Further equipped with a first correction means for correcting the distance
The first or second aspect of the present invention, wherein the AF means moves the focus position of the focus lens so that the focal length matches the distance corrected by the first correction means, and then shifts to the imaging mode. Imaging device. When the captured images generated by the second generation means have differences between frames, a distance map showing the distances associated with each of the plurality of divided regions is created, and the plurality of distance maps created. The imaging device according to any one of claims 1 to 3, wherein the distance is updated for each of the divided regions, and then the detection mode is resumed. The invention according to any one of claims 1 to 4, wherein in the cutout mode, a cutout means for cutting out the captured image in the designated divided region among the plurality of divided regions is further provided. Imaging device. When the attention subject is detected by the detection means, the cutout mode is set, and the cutout means sets the pixels of the pixels in the divided region in which the attention subject is reflected and the divided regions around the divided region. An image that has not been added is cut out, the coordinates of the foot position of the subject of interest are calculated from the cut out image, and the estimation is made based on the calculated coordinates and the distance associated with each of the plurality of divided regions. Further equipped with a second correction means for correcting the distance
The imaging according to claim 5, wherein the focus position of the focus lens is moved by the AF means so that the focal length matches the distance corrected by the second correction means, and then the mode shifts to the imaging mode. apparatus. The imaging device according to claim 3, wherein the weight is set based on at least one of the appearance rate, the disappearance rate, and the change rate of the distance of the attention subject. The imaging device according to claim 4, further comprising a memory for storing the distance map. The imaging device according to claim 1, wherein the plurality of divided regions further include setting means for setting the area thereof based on the distance to the fixed subject in which the image is projected. The imaging device according to claim 9, wherein the setting means increases the area of the divided region in which the fixed subject is projected as the distance to the fixed subject is short. 9. or 10, wherein the first generation means performs pixel addition of the pixels and then digitally adds the pixels in the vertical direction and the horizontal direction according to the size of the plurality of divided regions. Imaging device. When the object of interest is simultaneously detected by the detection means in the adjacent divided regions, a third correcting means for correcting the estimated distance according to the method of adjoining the divided regions is further provided.
The ninth to eleventh aspects of the present invention, wherein the AF means moves the focus position of the focus lens so that the focal length matches the distance corrected by the third correction means, and then shifts to the imaging mode. The imaging apparatus according to any one item. A division step of dividing the pixel area of the image sensor into a plurality of division areas, and
In the detection mode, a first generation step of generating a detection image by adding pixels of pixels in the plurality of divided regions to each other,
In the detection mode, a detection step in which the detection images created in the first generation step are compared between frames and a difference is detected as a subject of interest in each of the plurality of divided regions.
In the imaging mode, a second generation step of generating an captured image from the image sensor, and
AF step to move the focus position of the focus lens,
The linking step of linking the distances to the fixed subject reflected in each of the plurality of divided areas, and the linking step.
When the attention subject is detected in the detection step, it has an estimation step of estimating the distance to the attention subject based on the distance associated with the divided region in which the attention subject is reflected.
A drive control method characterized in that the focus position of the focus lens is moved in the AF step so that the focal length matches the estimated distance, and then the mode shifts to the imaging mode. A program comprising executing the drive control method according to claim 13.

Images