JP7278737B2 - IMAGING DEVICE, CONTROL METHOD THEREOF, AND PROGRAM - Google Patents

Description

The present invention relates to an imaging device, its control method, and a program.

In a digital camera, a subject is tracked based on the feature amount of a specific area (feature amount area) of the image from the obtained image data, and shooting conditions such as focus, brightness, color, etc. are adjusted appropriately for the subject. It is common to shoot according to the conditions. At this time, if the size of the feature amount region is set wider as in Patent Document 1, the amount of information used to search for the image area corresponding to the feature amount region in tracking the subject increases, so the tracking performance is improved. Be expected.

JP-A-8-329110

However, the technique of Patent Document 1 does not change the size of the feature amount region based on the image information when searching for the image. Therefore, if the technology according to Patent Document 1 is applied to search for a feature amount region in subject tracking and the size of the feature amount region is increased to improve tracking performance, tracking will fail. Sometimes I end up

For example, if the feature amount area is larger than the subject area, the feature amount area will also include the background of the subject. In this case, if a situation occurs during tracking of the subject, for example, the camera moves significantly or the subject is originally a moving body, the background information in the feature amount region will change over time. That is, the positional relationship between the subject (tracking target) and the background changes significantly, and the same feature amount no longer exists, resulting in failure of tracking.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus capable of always obtaining high tracking performance regardless of tracking conditions by determining the size of a feature amount region based on image information.

An imaging device according to an example of the present invention is an imaging device capable of acquiring a first image and a second image in time series, and includes subject detection means for detecting a subject as a tracking target from the first image. a feature region registering means for registering a feature region including the subject based on the first image; feature quantity region updating means for updating the size of the feature quantity region registered in the region registering means; and subject tracking means for tracking the subject by searching an image area.

According to the present invention, it is possible to provide an imaging apparatus capable of always obtaining high tracking performance regardless of the tracking situation by determining the size of the feature amount region based on the image information.

It is a block diagram which shows the structural example of an imaging device. 3 is a block diagram showing a configuration example of a subject recognition unit; FIG. 4 is a flow chart showing an operation example of the imaging device; 8 is a flowchart showing an example of subject recognition processing; 9 is a flowchart showing an example of subject detection processing; FIG. 10 is a diagram showing examples of evaluation pixels and estimation regions in subject detection processing; FIG. 10 is a diagram showing an example of an evaluation region in subject detection processing; 9 is a flowchart illustrating an example of background motion amount acquisition processing; 8 is a flowchart illustrating an example of update processing of feature amount regions; FIG. 10 is a diagram showing the relationship between the size of a feature amount region and the difference in motion amount; FIG. 10 is a diagram showing an example of changing the size of a feature amount region; FIG. 10 is a diagram showing an example of template matching;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The present invention is a function of setting a feature amount area for a subject, tracking the subject based on the feature amount area, and adjusting shooting conditions such as focus, brightness, color, etc., to a suitable state for the subject. It relates to an imaging device having The imaging apparatus described below is applied to optical equipment such as video cameras, digital still cameras, and interchangeable lenses usable for them.

<Imaging device>
FIG. 1 shows a configuration example of an imaging device.
The imaging device 100 can shoot and record images such as moving images and still images. Each functional block in the imaging device 100 is connected by a bus 160 so as to be in a state of being able to communicate with each other. Each function of the imaging device 100 is realized by the main control unit 151 executing a program and controlling each functional block.

The imaging device 100 includes a subject recognition unit 161 that analyzes an image and estimates a specific subject area from the analysis result. One of the features of this embodiment is that the object recognition unit 161 can be used to change the size of the feature amount region based on image information (image information obtained at the time of shooting).

Here, the image information is information relating to a plurality of images (a plurality of frames) that can be acquired in time series by the photographing lens (lens unit) 101 and the imaging device 141 . The image information includes, for example, the average brightness of the screen, WB (white balance), and position information by the GPS function. As will be described later, the image information is information obtained from the plurality of images, such as the amount of movement of the subject, the amount of movement of the background, and the positions of the subject (tracking target) and the background obtained from them. It also includes the amount of change in relationships.

As described above, in the present embodiment, by enabling the object recognition unit 161 to change the size of the feature amount region, it is possible to realize an imaging apparatus that can always obtain high tracking performance regardless of the tracking situation. . Therefore, the subject recognition unit 161 has a subject detection function, a feature amount region registration function, a background motion amount acquisition function, a feature amount region update function, and a subject tracking function. Details of these functions will be described later.

The taking lens 101 includes a fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, a focus lens 131, a zoom motor 112, an aperture motor 104, and a focus motor 132. . A fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, and a focus lens 131 constitute a photographing optical system.

In the figure, each of the lenses 102, 111, 121, and 131 is shown as one lens for the sake of convenience, but may be composed of a combination of a plurality of lenses. Further, when the imaging device 100 constitutes a camera system, the photographing lens 101 may be integrated with the camera body, or may be configured to be detachable from the camera body.

A diaphragm control unit 105 controls the operation of a diaphragm motor 104 that drives the diaphragm 103 . That is, the aperture control unit 105 changes the aperture diameter of the aperture 103 . A zoom control unit 113 controls the operation of a zoom motor 112 that drives the zoom lens 111 . That is, the zoom control unit 113 changes the focal length (angle of view) of the photographing lens 101 .

The focus control unit 133 acquires the defocus amount and the defocus direction of the photographing lens 101 based on the phase difference between the pair of focus detection signals (the A image and the B image) obtained from the imaging device 141 . The focus control unit 133 then converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 132 . Based on this driving amount and driving direction, the focus control unit 133 controls the operation of the focus motor 132 and drives the focus lens 131 to control the focus state of the photographing lens 101 .

In this manner, the focus control unit 133 performs automatic focus detection (AF) using a phase difference detection method. However, the focus control unit 133 may perform AF using a contrast detection method based on a contrast evaluation value obtained from an image signal instead of AF using a phase difference detection method.

A subject image formed on the imaging plane of the imaging element 141 by the photographing lens 101 is converted into an electrical signal (image signal) by a photoelectric conversion element of each of a plurality of pixels arranged in the imaging element 141 . The imaging element 141 is, for example, a CCD or CMOS image sensor, and has a pixel array of m pixels in the horizontal direction and n pixels in the vertical direction. However, n and m are natural numbers of 2 or more, respectively. Each pixel has two photoelectric conversion elements (photoelectric conversion regions). The imaging control unit 143 controls readout of image signals from the imaging device 141 based on instructions from the main control unit 151 .

An image signal read out from the imaging element 141 is supplied to the image signal processing section 142 . The image signal processing unit 142 applies signal processing such as noise reduction processing, A/D conversion processing, and automatic gain control processing to the image signal, and outputs the image signal to which the signal processing has been applied to the imaging control unit 143. . The imaging control unit 143 stores the image signal received from the image signal processing unit 142 in a RAM (random access memory) 154 .

The image processing unit 152 applies predetermined image processing to the image signal stored in the RAM 154 . The image processing applied by the image processing unit 152 includes, for example, so-called development processing such as white balance adjustment processing, color interpolation (demosaicing) processing, and gamma correction processing. Also, the image processing may include signal format conversion processing, scaling processing, and the like. Examples of the image processing are not limited to these, and processing other than the above may be performed.

The image processing unit 152 can also generate information regarding subject brightness and the like for use in automatic exposure control (AE). Information about the specific subject area is supplied from the subject recognition unit 161 to the image processing unit 152, and used for white balance adjustment processing, for example. Note that when performing AF by the contrast detection method, the image processing unit 152 can also generate an AF evaluation value. The image processing unit 152 stores the processed image signal in the RAM 154 .

The main control unit 151 adds a predetermined header to the image signal temporarily stored in the RAM 154 to generate a data file in a predetermined recording format. At this time, the image compression/decompression unit 153 may encode the data file based on an instruction from the main control unit 151 to compress the amount of information regarding the image signal. The main controller 151 then records the generated data file in a recording medium 157 such as a memory card.

When displaying the image signal stored in the RAM 154, the main control unit 151 instructs the image processing unit 152 to perform scaling processing of the image signal so as to fit the display size of the display unit 150 such as a monitor display. do. Thereafter, the main control unit 151 writes the scaled image signal to the video memory area (VRAM area) of the RAM 154 as display image data. The display unit 150 reads display image data from the VRAM area of the RAM 154 and displays it on a display device such as an LCD or an organic EL display.

The imaging apparatus 100 of the present embodiment immediately displays the captured moving image on the display unit 150 during moving image recording (shooting standby state or during moving image recording), thereby using the display unit 150 as an electronic viewfinder (EVF). make it work. A moving image and its frame images displayed when the display unit 150 functions as an EVF are called live view images or through images. In addition, the imaging device 100 displays the captured still image on the display unit 150 for a certain period of time so that the user can confirm the captured result when the still image is captured. These display operations are also realized under the control of the main control unit 151 .

The operation unit 156 includes switches, buttons, keys, a touch panel, and the like for the user to input instructions to the imaging device 100 . Input from the operation unit 156 is transferred to the main control unit 151 via the bus 160 . The main control unit 151 controls each unit in order to realize the operation according to the input by the operation unit 156 .

The main control unit 151 has one or more programmable processors such as CPU and MPU. The main control unit 151 reads a program stored in a storage unit 155 such as a flash memory to a RAM 154 and executes the program to control each unit and realize each function or each process of the imaging device 100 . Further, the main control unit 151 executes AE processing for automatically determining exposure conditions (shutter speed, accumulation time, aperture value, sensitivity, etc.) based on subject brightness information. Information on subject brightness can be obtained from the image processing unit 152, for example. The main control unit 151 can also determine exposure conditions based on a subject area such as a person's face.

The main control unit 151 fixes the aperture, for example, and determines the electronic shutter speed (accumulation time) and the magnitude of the gain during moving image shooting. The main control unit 151 notifies the imaging control unit 143 of the determined exposure conditions (accumulation time and gain magnitude). The imaging control unit 143 controls the operation of the imaging element 141 so that shooting is performed according to the exposure conditions notified from the main control unit 151 .

The result of the subject recognition unit 161 can be used, for example, for automatic setting of the focus detection area. In this case, a tracking AF function for a specific subject area can be realized. It is also possible to perform AE processing based on the luminance information of the focus detection area, and to perform image processing (eg, gamma correction processing, white balance adjustment processing, etc.) based on the pixel values of the focus detection area. Note that the main control unit 151 may superimpose an index representing the current position of the subject area (for example, a rectangular frame surrounding the subject area) on the display image.

The battery 159 is managed by the power management unit 158 and supplies power to the entire imaging apparatus 100 . The storage unit 155 stores programs to be executed by the main control unit 151, setting values necessary for executing the programs, GUI data, user setting values, and the like. For example, when the operation unit 156 is operated to instruct the transition from the power-off state to the power-on state, the program stored in the storage unit 155 is read into the RAM 154, and the main control unit 151 executes the program.

<Subject Recognition Unit>
FIG. 2 shows a configuration example of a subject recognition unit.
The subject recognition unit 161 includes a subject detection unit 201 , a feature amount region registration unit 202 , a background motion amount acquisition unit 203 , a feature amount region update unit 204 , and a subject tracking unit 205 .

The subject detection unit 201 detects the position and size of the subject (tracking target) based on the image acquired by the imaging device 141 and processed by the image processing unit 152 . For example, the subject detection unit 201 acquires images (frames) in time series from the image processing unit 152 and detects a subject as a tracking target included in each image. Further, the subject detection unit 201 obtains the area occupied by the subject in each image, that is, the subject area, based on the position and size of the subject, and generates information regarding the reliability of the subject area.

A feature amount area registration unit 202 determines a feature amount area including a subject based on the subject area detected by the subject detection unit 201, and registers the feature amount area. Here, the feature amount area is an image area including the subject set in the n frame (first image) when tracking the subject as the tracking target. However, n is a natural number (1, 2, 3, . . . ), and the larger n means the later time. Tracking of the subject is performed by searching for an image area corresponding to the feature amount area in a frame after the n frame, for example, the (n+1) frame (second image), as will be described later.

When registering a feature region based on the subject region detected from the first image by the subject detection unit 201, the feature region registration unit 202 sets the size of the feature region to a default value (initial value). Default values are predetermined. However, the default value may be configured so that the user can freely change it via the operation unit 156 or the like, or it may be configured so that the program can automatically set the optimum value.

When the subject tracking unit 205, which will be described later, finds an image area corresponding to the feature amount area and the tracking is successful, the feature amount area registration unit 202 replaces the already registered feature amount area with The image area is registered as a new feature amount area. On the other hand, when the subject tracking unit 205, which will be described later, cannot find an image area corresponding to the feature amount region and the tracking is unsuccessful, the feature amount area registration unit 202 continues to register the already registered feature amount area. do.

The background motion amount acquisition unit 203 acquires the background motion amount based on a plurality of images sequentially supplied in time series. The background motion amount acquisition unit 203, for example, estimates the background from the plurality of images, and detects a motion vector between the plurality of images based on the estimated background. Accordingly, the background motion amount acquisition unit 203 can acquire the background motion amount based on the motion vector.

In addition, the background motion amount acquisition unit 203 detects the motion of the imaging device 100 using at least one of a gyro sensor and an acceleration sensor, and acquires the background motion amount based on the motion of the imaging device 100. can also Details of the operation of the background motion amount acquisition unit 203 will be described later.

The feature region update unit 204 updates the size of the feature region registered in the feature region registration unit 202 based on the information on the plurality of images that are sequentially supplied in time series. For example, the feature amount region updating unit 204 determines the difference between the subject and the background based on the amount of background motion acquired by the background motion amount acquiring unit 203 and the amount of motion of the subject acquired by the subject tracking unit 205, which will be described later. Get the amount of change in the positional relationship. Then, the feature region update unit 204 updates the size of the feature region registered in the feature region registration unit 202 based on the amount of change.

Here, the size (default value) of the feature amount region that is first registered by the feature amount region registration unit 202 is given by: It is desirable to set the size to correspond to the reference value Δref. A relationship between the reference value Δref and the default value will be described later.

Then, when the amount of change between the subject and the background is larger than the reference value Δref, the feature region updating unit 204 makes the size of the feature region smaller than the default value. Also, when the amount of change is smaller than the reference value Δref, the feature quantity region updating unit 204 makes the size of the feature quantity region larger than the default value. Further, the feature quantity region update unit 204 reduces the size of the feature quantity region as the change amount increases, and increases the size of the feature quantity region as the change quantity decreases. However, when the amount of change is larger than the threshold, the feature amount region updating unit 204 keeps the size of the feature amount region constant.

The subject tracking unit 205 searches for an image area corresponding to the feature amount area updated by the feature amount area updating unit 204 from a plurality of images sequentially supplied in time series, thereby tracking the subject. For example, the subject tracking unit 205 sets the image area having the highest similarity to the feature amount area whose size is updated by the feature amount area updating unit 204 as the image area corresponding to the feature amount area, and determines the subject based on the image area. track.

Here, the degree of similarity is determined, for example, by an evaluation value defined by a predetermined function, and the subject tracking unit 205 determines the region with the lowest evaluation value as the region with the highest degree of similarity. Also, the subject tracking unit 205 can conclude that the image area corresponding to the feature amount area could not be found when the minimum value of the evaluation values is larger than the threshold. The search result includes information such as the object area in the image, reliability, amount of movement of the object, etc., and is used for various processes in the main control unit 151 .

<Example of operation of imaging device>
FIG. 3 shows an operation example of the imaging device.
In the present embodiment, a moving image shooting operation involving subject detection processing, subject tracking processing, and motion amount detection processing for detecting the amount of motion of the subject and background will be described. The motion picture shooting operation is executed during shooting standby or during motion picture recording.

Details such as the resolution of images (frames) to be handled differ between shooting standby and video recording, but subject detection processing, subject tracking processing, and motion amount detection processing for detecting the amount of motion of the subject and background are performed. The content concerned is basically the same. Therefore, the following description will be made without particularly distinguishing between shooting standby and moving image recording.

First, in S301, the main control unit 151 determines whether the power switch of the imaging device 100 is on. If the power switch is off, this flow ends, and if the power switch is on, this flow advances to S302.

Next, in S302, the main control unit 151 controls each unit to execute imaging processing for one frame, and then advances the flow to S303. In the present embodiment, as the imaging process, a pair of parallax images and a captured image for one screen are acquired and stored in the RAM 154 . In S303, the main control unit 151 instructs the subject recognition unit 161 to perform subject recognition processing. Here, the subject recognition processing includes subject detection processing, subject tracking processing, and motion amount detection processing for detecting the amount of motion of the subject and background. Details of the subject recognition processing in the subject recognition unit 161 will be described later.

Thereafter, the position, size, and motion amount of the subject area are notified from the subject recognition section 161 to the main control section 151 and stored in the RAM 154 . The main control unit 151 sets the focus detection area based on the notified object area.

Next, in S304, the main control unit 151 causes the focus control unit 133 to execute focus detection processing. The focus control unit 133 connects a plurality of A signals obtained from a plurality of pixels arranged in the same row among a plurality of pixels included in the focus detection areas of the pair of parallax images to generate an A image, In addition, a plurality of B signals are spliced together to generate a B image. Then, the focus control unit 133 calculates the amount of correlation between the A image and the B image while shifting the relative positions of the A image and the B image, and determines the relative position where the similarity between the A image and the B image is the highest. It is obtained as a phase difference (shift amount) between the A image and the B image. Furthermore, the focus control unit 133 converts the phase difference into a defocus amount and a defocus direction.

Next, in S305, the focus control unit 133 drives the focus motor 132 and moves the focus lens 131 according to the lens drive amount and drive direction corresponding to the defocus amount and defocus direction obtained in S304. Also, when the lens driving process ends, the main control unit 151 returns the flow to S301.

Thereafter, the processes of S302 to S305 are repeatedly executed until it is determined in S301 that the power switch is turned off. As a result, a subject as a tracking target is searched for a plurality of images (frames) sequentially supplied in time series, and a subject tracking function is realized. In FIG. 3, the subject tracking process is performed for each frame that is sequentially supplied, but the subject tracking process may be performed every several frames for the purpose of reducing the processing load and power consumption of the main control unit 151 .

<Subject recognition processing>
FIG. 4 shows an example of subject recognition processing.
This flowchart is a subroutine of S303 in FIG.

First, in S<b>401 , the subject recognition unit 161 acquires an image (frame) from the imaging control unit 143 . After that, in S402, the subject detection unit 201 sets a plurality of evaluation areas with different center positions and sizes for the image from the imaging control unit 143, and detects subjects from each evaluation area. Details of the processing for detecting the subject will be described later.

Next, in step S<b>403 , the feature amount region registration unit 202 registers the feature amount based on the subject area detected by the subject detection unit 201 or the image area corresponding to the feature amount area of the previous frame detected by the subject tracking unit 206 . Register a region. In this embodiment, registration of feature amount regions is performed based on the results of the subject detection unit 201 for the first frame, and based on the results of the subject tracking unit 206 for subsequent frames.

Next, in S404, the background motion amount acquisition unit 203 acquires the background motion amount based on the plurality of images sequentially supplied in time series. The details of the process of acquiring the motion amount of the background will be described later. After that, in S405, the feature amount region updating unit 204 updates the size of the feature amount region based on the amount of motion of the background acquired in S404 and the amount of motion of the subject in the previous frame acquired in S406, which will be described later. I do. Details of the processing for updating the size of the feature amount region will be described later.

Next, in S406, the subject tracking unit 205 searches for an image area corresponding to the feature amount area updated in S405 from a plurality of images sequentially supplied in time series. For example, the subject tracking unit 205 sets an area having the highest degree of similarity to the feature amount area as an image area corresponding to the feature amount area, and tracks the subject based on the image area. The details of the subject tracking process will be described later.

<Subject detection processing>
FIG. 5 shows an example of subject detection processing.
This flowchart is a subroutine of S402 in FIG.

First, in S501, the subject detection unit 201 determines an evaluation region for image evaluation. For example, in FIG. 6, 601 is an input image, 602 is an evaluation pixel, and 603 is an evaluation area candidate. As shown in FIGS. 6A, 6B, and 6C, each evaluation area candidate 603 is set to have a different size. Evaluation pixels 602 are sequentially selected from all pixels of the input image 601 by raster processing.

Next, in S502, the subject detection unit 201 determines an outer evaluation area for the evaluation area. For example, in FIG. 7, 701 is an input image, 702 is an evaluation pixel, 703 is an evaluation area, and 704 is an outer evaluation area. As shown in FIG. 7, the center position of the outer evaluation area 704 is the same as the center position of the evaluation area 703 . Also, the size from the center position of the outer evaluation area 704 to the outer frame is larger than the size from the center position of the evaluation area 703 to the outer frame. That is, the outer evaluation area 704 is a ring-shaped area excluding the area corresponding to the evaluation area 703 . Note that the difference between the size from the center position to the outer frame of the outer evaluation region 704 and the size from the center position to the outer frame of the evaluation region 703 is determined based on the size of the input image 701. 10% of horizontal size.

Next, in S503, the subject detection unit 201 calculates the degree of difference between the image signals corresponding to the evaluation area and the outer evaluation area as an evaluation value.

The evaluation value D is calculated by the following formula.

However, d _H is the difference calculated from the hue information of the evaluation area and the outer evaluation area, p _Hi represents the number of pixels whose hue information is i in the evaluation area, and q _Hi is the outer evaluation area. represents the number of pixels whose hue information is i. d _S , p _Si , q _Si are values related to saturation information, and d _V , p _Vi , q _Vi are values related to luminance information. Also, m is the maximum value that hue information, saturation information, and luminance information can take.

Next, in S504, the subject detection unit 201 determines whether or not there is an unprocessed evaluation area. If there is an unprocessed evaluation area, the subject detection unit 201 returns the flow to S501 and repeats a series of processes. If there is no unprocessed evaluation area, the subject detection unit 201 advances the flow to S505.

Next, in S505, the subject detection unit 201 determines a subject. That is, the subject detection unit 201 determines the area having the highest evaluation value among the evaluation values calculated in S503 as the subject.
It should be noted that the method described here is just an example, and the subject can also be determined by a method other than the flow in FIG. For example, in the flow of FIG. 5, it is also possible to determine the subject in consideration of the depth information of the image.

<Acquisition processing of background motion amount>
FIG. 8 shows an example of background motion amount acquisition processing.
This flowchart is a subroutine of S404 in FIG.

FIG. 8A is an example of obtaining the background motion amount from the estimated background.
First, in S8011, the background motion amount acquisition unit 202 estimates the background based on a plurality of images sequentially acquired in S401. For example, when a subject is detected by the subject detection processing in S402, the background motion amount acquisition unit 202 can estimate an image area other than the subject to be the background.

Next, in S8012, the background motion amount acquisition unit 202 uses, of the plurality of images sequentially acquired in S401, the image related to the current frame and the image related to the previous frame, A background motion vector is calculated from the estimated background (image).

Finally, in S8013, the background motion amount acquisition unit 202 acquires the background motion amount based on the background motion vector.

FIG. 8B is an example of obtaining the motion amount of the background by calculating a plurality of motion vectors by feature point matching and selecting the motion vector of the background from them.
First, in S8021, the background motion amount acquisition unit 202 uniformly distributes a plurality of feature points in the image acquired in S401, and calculates a motion vector for each feature point from the plurality of uniformly distributed feature points. Calculate

Next, in S8022, the background motion amount acquisition unit 202 selects a background motion vector from the plurality of calculated motion vectors. A known technique can be used as a method for selecting the motion vector of the background. For example, the background motion amount acquisition unit 202 histograms the plurality of motion vectors calculated in S8021, and determines the motion vector associated with the maximum number of bins as the background motion vector.

Finally, in S8023, the background motion amount acquisition unit 202 acquires the background motion amount based on the background motion vector.

In calculating the motion vector of the background, the start point of the motion vector is detected from the image of the frame one frame before the image of the current frame, and the end point of the motion vector is detected from the image of the current frame. is desirable. However, if the direction with respect to time of the amount of motion of the background obtained from the motion vector of the background is the same as the direction with respect to time of the amount of motion of the subject, which will be described later, the start point and end point of the motion vector may be reversed.

In this embodiment, a motion vector calculated from an image is used to acquire the amount of motion of the background. The amount of background motion may be determined based on the motion of the device.

<Update Processing of Feature Amount Region>
FIG. 9 shows an example of update processing of feature amount regions.
This flowchart is a subroutine of S405 in FIG.

First, in S901, the feature amount region updating unit 204 calculates the difference between the background motion amount acquired in S8013 or S8023 and the subject motion amount. This difference corresponds to the amount of change in the positional relationship between the subject and the background. Here, the subject tracking unit 205 has already acquired the motion amount of the subject from the image related to the frame before the current frame, and is stored in the RAM 154, for example.

Regarding the difference between the amount of motion of the subject and the amount of motion of the background, the feature amount region updating unit 204 calculates the absolute value of the difference in the horizontal direction of the image and the absolute value of the difference in the vertical direction of the image, One of them or a value obtained by adding them is taken as the difference between the amount of motion of the subject and the amount of motion of the background. For example, the feature amount region updating unit 204 determines the absolute value of the difference in the horizontal direction and the absolute value of the difference in the vertical direction, whichever has the larger value, as the difference between the amount of motion of the subject and the amount of motion of the background.

Next, in S902, the feature amount region updating unit 204 updates the size of the feature amount region registered in the feature amount region registration unit 202 based on the difference between the amount of movement of the subject and the amount of movement of the background. End the flow.

Here, when updating the size of the feature region in S902, the feature region updating unit 204 calculates the difference between the amount of motion of the subject and the amount of motion of the background, that is, the amount of change in the positional relationship between the subject and the background. increases, the size of the feature amount region is reduced. Further, the feature amount region update unit 204 increases the size of the feature amount region as the difference between the amount of movement of the subject and the amount of movement of the background becomes smaller.

FIG. 10 shows the relationship between the size of the feature amount region and the difference in motion amount between the subject and the background.
The default value (initial value) is the size of the feature region registered first in the feature region registration unit 202 . The difference in motion amount corresponding to the default value is the reference value Δref.

According to the figure, when the difference in motion amount is larger than the reference value Δref (ΔH), the size of the feature amount region changes from the default value to a smaller value (arrow A). Also, when the difference in motion amount is smaller than the reference value Δref (ΔL), the size of the feature amount region changes from the default value to a larger value (arrow B). That is, the size of the feature region decreases as the difference in motion amount increases, and the size of the feature region increases as the difference in motion amount decreases. However, when the difference in motion amount is larger than the threshold value Δth, the size of the feature amount region is constant.

FIG. 11 schematically shows the relationship between the size of the feature amount region and the difference in motion amount.
FIG. 8A shows an image of n frames, showing the feature amount region 1101 registered in S403. The size of the feature region 1101 is assumed to have a default value, for example. (B) in the figure is an image related to the (n+1) frame, in which there is no or almost no difference in the amount of motion with respect to the image in (A) in the figure. (D) in the figure is an image relating to the (n+1) frame, and shows a case in which the difference in motion amount is large with respect to the image in (A) in the figure.

In the case of FIG. 4B, in S405, the feature amount region updating unit 204 determines that the difference in motion amount is smaller than the reference value Δref. Therefore, as shown in FIG. 2C, the feature amount region updating unit 204 updates the size of the feature amount region 1102 to a value larger than the size of the feature amount region 1101 shown in FIG.

In this case, the feature amount area 1102 is larger than the subject area and contains information on the subject and the background. That is, when searching for the image region corresponding to the feature amount region 1102 in the image of the (n+2) frame, more information is used for searching, and tracking performance can be improved. Moreover, since the background information included in the feature amount region 1102 does not change or hardly changes, a situation in which tracking fails due to a change in background information does not occur.

In the case of (D) in the figure, in S405, the feature amount region updating unit 204 determines that the difference in motion amount is larger than the reference value Δref. Therefore, as shown in (E) of the same figure, the feature quantity region updating unit 204 updates the size of the feature quantity region 1103 to a smaller value than the size of the feature quantity region 1101 shown in (A) of the same figure.

In this case, the feature amount area 1103 has approximately the same size as the subject area, and does not include the background at all or hardly includes it. That is, when searching for an image area corresponding to the feature amount area 1103 in the image of the (n+2) frame, a situation in which tracking fails due to a change in the background information of the feature amount area 1103 does not occur. do not have.

<Subject Tracking Processing>
Next, the subject tracking processing in S405 will be described with reference to FIG.
The subject tracking unit 205 searches for an image area corresponding to the feature amount area updated in S404. A search result is output as output information of the subject tracking unit 205 .

FIG. 12 shows an example of template matching.
A search for an image region corresponding to a feature amount region can be performed by template matching. Template matching is a technique in which a pixel pattern is set as a template and an area with the highest similarity to the template is searched in the image. As the degree of similarity between the template and the image region, a correlation amount such as the sum of absolute differences between corresponding pixels can be used.

FIG. 12A schematically shows a template 1201 and its configuration example 1202. FIG. When performing template matching, the pixel pattern of the template 1201 is set in advance. In this embodiment, the template 1201 has a size of W pixels in the horizontal direction and H pixels in the vertical direction. Template matching is then performed using the luminance values of the pixels included in template 1201 .

A feature amount T(i, j) of the template 1201 is expressed by the following equation when a coordinate system as shown in FIG. 12A is set.

FIG. 12B schematically shows an image area 1204 within the search area 1203 and its configuration example 1205 . A search area 1203 represents a range in which pattern matching is performed in an image related to a post-subject tracking frame. The search area 1203 may be the entire image or a portion thereof.

Coordinates within the search area 1203 are represented by (x, y). An image area 1204 is for searching an image area having the highest degree of similarity to the template 1201 from the search area 1203 and is shifted within the search area 1203 . Therefore, the size of the search area 1203 is set to be the same as the size of the template 1201 (horizontal pixel count W, vertical pixel count H).

Each time the image area 1204 is shifted, the subject tracking unit 205 calculates the degree of similarity between the brightness value of each pixel included in the image area 1204 and the brightness value of each pixel included in the template 1201 (pattern matching). Here, the feature amount S(i, j) of the image area 1204 is expressed by the following equation when a coordinate system as shown in FIG. 12B is set.

但し、Ｖ（ｘ，ｙ）は、画像領域１２０４の左上頂点の座標（ｘ，ｙ）における評価値として表す。 Also, let V(x, y) be an evaluation value representing the similarity between the template 1201 and the image region 1204. V(x, y) is the sum of absolute differences (SAD) shown in the following equation. is represented by

However, V(x, y) is expressed as an evaluation value at the coordinates (x, y) of the upper left vertex of the image area 1204 .

Then, the subject tracking unit 205 shifts the image area 1204 from the upper left (x=0, y=0) of the search area 1203 to the right (x direction) by one pixel, and the evaluation value V(x, y) is calculated. In addition, when the image area 1204 reaches the right end (x=XW) of the search area 1203, the subject tracking unit 205 returns the image area 1204 to the left end (x=0) and downward (y direction) by one pixel (y=1). Then, the subject tracking unit 205 again shifts the left end (x=0, y=1) of the search area 1203 by one pixel to the right (x direction), and calculates the evaluation value V(x, y) at each position. calculate.

The subject tracking unit 205 repeats the above operation until the search area 1203 reaches the lower right of the search area 1203 (x=XW, y=YH). However, X is the number of horizontal pixels in the search area 1203 (the number of pixels in the x direction), and Y is the number of vertical pixels in the search area 1203 (the number of pixels in the y direction). In addition, the subject tracking unit 205 determines that the image area 1204 at the position V(x, y) having the lowest evaluation value among the evaluation values V(x, y) calculated at each position corresponds to the template 1201. image region with the highest degree of similarity.

In this way, the subject tracking unit 205 determines the position of the pixel area 1204 showing the minimum evaluation value V(x, y) as the position of the pixel area corresponding to the feature amount area, that is, the position of the subject (tracking target). Determined as When the reliability of the search result is low, for example, when the minimum value of the evaluation value V(x, y) is larger than the threshold, the subject tracking unit 205 cannot find the image area corresponding to the feature amount area. You may decide that

In this embodiment, in pattern matching, an example in which the luminance value is used as the feature amount has been described, but elements other than the luminance value (brightness, hue, saturation, etc.) may be used. Also, the evaluation value V(x, y) in pattern matching may be determined in consideration of a plurality of factors (luminance value, lightness, hue, saturation, etc.). Furthermore, the evaluation value V(x, y) can be represented by normalized cross-correlation (NCC), ZNCC, etc. instead of SAD.

<Other embodiments>
A program that implements each function or each process of the above-described embodiment can be installed in advance in the flash memory 155 in the imaging apparatus 100 or alternatively in a ROM (not shown). A program that implements one or more functions or processes of the above-described embodiments can also be supplied to the imaging device 100 via a network or storage medium. In this case, the main control unit 151 in the image capturing apparatus 100, or at least one processor (not shown) instead of this, executes the supplied program, thereby performing each function or each process of the above-described embodiment. Realized.

Moreover, each function or each process of the above-described embodiments can be realized by a program (software) installed in a computer as described above, and can also be realized by hardware (circuits). For example, one or more functions or processes of the above-described embodiments can also be implemented in circuits such as ASICs (Application Specific ICs), FPGAs (Field Programmable Gate Arrays), and the like.

<Conclusion>
As described above, according to the present invention, it is possible to provide an imaging apparatus capable of always obtaining high tracking performance regardless of the tracking situation by determining the size of the feature amount region based on the image information. For example, in tracking under conditions where the positional relationship between the subject and the background changes significantly, the feature amount region is set to be approximately the same size as the subject region, and in tracking under conditions where the positional relationship between the subject and the background changes little, Make the feature amount area larger than the subject area. This makes it possible to always obtain high tracking performance regardless of the tracking situation.

161: subject recognition unit 201: subject detection unit 202: feature amount area registration unit 203: background movement amount acquisition unit 204: feature amount area update unit 205: subject tracking unit

Claims (15)

An imaging device capable of acquiring a first image and a second image in time series,
subject detection means for detecting a subject as a tracking target from the first image;
feature region registration means for registering a feature region including the subject based on the first image;
feature quantity region updating means for updating the size of the feature quantity region registered in the feature quantity region registration means based on information including the amount of motion of the subject relating to the first and second images;
subject tracking means for tracking the subject by searching from the second image for an image area corresponding to the feature amount region whose size is updated by the feature amount area updating means;
An imaging device characterized by: The feature amount region registration means sets the size of the feature amount region to be registered based on the first image as an initial value,
2. The imaging device according to claim 1, wherein: The feature quantity region updating means reduces the size of the feature quantity region to be smaller than the initial value when the amount of change in the positional relationship between the subject and its background is larger than a reference value, and making the size of the feature region larger than the initial value if it is smaller than the reference value;
3. The imaging apparatus according to claim 2, characterized by: The feature quantity region updating means reduces the size of the feature quantity region as the amount of change increases, and increases the size of the feature quantity region as the amount of change decreases.
4. The imaging device according to claim 3, characterized in that: The feature quantity region updating means keeps the size of the feature quantity region constant when the amount of change is greater than a threshold.
5. The imaging device according to claim 4, characterized in that: When the search by the subject tracking means is performed and the tracking is successful, the feature amount area registration means replaces the already registered feature amount area with the feature amount area used in the search. register anew
6. The imaging apparatus according to any one of claims 1 to 5, characterized by: The information includes at least one of average brightness of the screen, white balance, and position information by GPS function.
7. The imaging apparatus according to any one of claims 1 to 6, characterized by: the information includes the amount of movement of the background of the subject ;
7. The imaging apparatus according to any one of claims 1 to 6, characterized by: Further comprising background motion amount acquisition means for acquiring the amount of motion of the background from the first and second images,
the subject tracking means acquires the amount of motion of the subject from the first and second images;
The feature region updating means updates the size of the feature region based on the amount of motion of the subject and the amount of motion of the background.
9. The imaging apparatus according to claim 8, characterized by: The background motion amount acquisition means includes:
estimating the background from the first and second images;
Detecting a motion vector between the first and second images based on the estimated background;
obtaining a motion amount of the background based on the motion vector;
10. The imaging device according to claim 9, characterized by: The background motion amount acquisition means includes:
detecting movement of the imaging device by at least one of a gyro sensor and an acceleration sensor;
obtaining the amount of motion of the background based on the motion of the imaging device;
10. The imaging device according to claim 9, characterized by: further comprising control means for controlling shooting conditions for the subject based on tracking of the subject;
12. The imaging apparatus according to any one of claims 1 to 11, characterized by: an imaging device that captures the first image and the second image in time series;
at least one processor for processing the first image and the second image;
The at least one processor
subject detection means for detecting a subject as a tracking target from the first image;
feature region registration means for registering a feature region including the subject based on the first image;
feature quantity region updating means for updating the size of the feature quantity region registered in the feature quantity region registration means based on information including the amount of motion of the subject relating to the first and second images;
subject tracking means for tracking the subject by searching from the second image for an image area corresponding to the feature amount region whose size is updated by the feature amount area updating means;
An imaging device characterized by: A control method for an imaging device capable of acquiring a first image and a second image in time series,
a subject detection step of detecting a subject as a tracking target from the first image;
a feature amount region registration step of registering a feature amount region including the subject based on the first image;
a feature quantity region updating step of updating the size of the feature quantity region registered in the feature quantity region registration step based on information including the amount of motion of the subject relating to the first and second images;
a subject tracking step of tracking the subject by searching from the second image for an image area corresponding to the feature amount region whose size has been updated in the feature amount region updating step;
A control method characterized by: A program for causing a computer to function as each means of the imaging apparatus according to any one of claims 1 to 13.

Images