JP7223079B2 - IMAGE PROCESSING APPARATUS, CONTROL METHOD THEREOF, AND IMAGING APPARATUS - Google Patents

Description

The present invention relates to an image processing device, its control method, and an imaging device, and more particularly to technology for tracking a specific region between images.

By searching for regions in one or more images taken after time t for regions similar to regions in images taken at time t, movement of regions over time can be detected. For example, by detecting the movement of a specific subject's area (face area) during movie shooting, it is possible to keep the specific subject in focus or dynamically change the exposure conditions so that the specific subject's exposure is appropriate. (Patent Document 1).

JP 2005-318554 A

A technique called matching is generally used when searching for a region similar to a specific image region. For example, in template matching, a pixel pattern in a certain image area is set as a feature amount (template), and the similarity (for example, correlation amount) is calculated for each position while relatively changing the position of the template within the search area of another image. and detect the position with the highest similarity. Then, if it is determined that the similarity at the detected position is sufficiently high, it is estimated that there is an image area with the same pattern as the template at that position.

The accuracy of search by matching greatly depends on how the feature amount used for matching is set. For example, when tracking a face area of a specific person, if a pixel pattern of an area including only a part of the face area is set as a feature amount, erroneous detection is likely to occur because the face feature amount is small. Conversely, if a pixel pattern that includes the entire face area but has a large proportion of the area surrounding the face area (for example, the background area) is set as the feature amount, the similarity of the background will make a large contribution, and erroneous detection will likely occur. .

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus capable of highly accurate area tracking and a control method thereof.

The above object has an extraction means for extracting a feature quantity from an image region, and a search means for searching for a region similar to the image region in a plurality of time-series images using the feature quantity, and extracting In the area having distance information in the peripheral area of the area similar to the image area, the difference from the distance information in the area similar to the image area is within a predetermined range on the short distance side and the long distance side. is not greater than the threshold value, the search means updates the feature value used for the search when the ratio occupies the peripheral region is not greater than the threshold value, and the search means does not update the feature value used for the search when the ratio is greater than or equal to the threshold value. Accomplished by processing equipment.

According to the present invention, it is possible to provide an image processing apparatus capable of highly accurate area tracking and a control method thereof.

1 is a block diagram showing an example functional configuration of a digital camera according to an embodiment; FIG. A diagram showing an example of a pixel array of the image sensor in FIG. Block diagram showing a functional configuration example of the tracking unit in FIG. FIG. 4 is a diagram related to template matching in the first embodiment; Diagram relating to histogram matching in the first embodiment FIG. 4 is a diagram relating to a subject distance acquisition method according to the first embodiment; FIG. 4 is a diagram schematically showing a method of specifying a subject region according to the first embodiment; Flowchart of imaging processing in the first embodiment Flowchart of subject tracking processing in the first embodiment Flowchart of imaging processing in the second embodiment Flowchart of subject tracking processing in the second embodiment FIG. 11 is a diagram schematically showing a feature amount update determination method according to the second embodiment;

A digital camera as an example of an image processing apparatus according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention can also be implemented in electronic equipment that does not have a photographing function. Examples of electronic devices that can implement the present invention include, but are not limited to, digital cameras, mobile phones, tablet terminals, game machines, personal computers, navigation systems, home appliances, robots, and the like.

● <First embodiment>
(Configuration of imaging device)
FIG. 1 is a block diagram showing an example functional configuration of a digital camera 100 according to the first embodiment of the present invention. The digital camera 100 can shoot and record moving and still images. Each functional block within the digital camera 100 is communicably connected to each other via a bus 160 . The operation of the digital camera 100 is realized by the main control unit 151 (central processing unit) executing a program to control each functional block.

The digital camera 100 of this embodiment can acquire distance information of a photographed subject. The distance information may be, for example, a distance image representing the distance of the object corresponding to the pixel value. The distance information may be obtained by any method, but in this embodiment the distance information is obtained based on the parallax images. Although there is no limitation on the method of acquiring the parallax images, in this embodiment, the parallax images are acquired using the imaging device 141 having a plurality of photoelectric conversion elements sharing one microlens. Parallax images may be obtained by using the digital camera 100 as a multi-view camera such as a stereo camera, or parallax image data captured by an arbitrary method may be obtained from a storage medium or an external device.

The digital camera 100 also has a tracking unit 161 that implements a subject tracking function by continuously searching for areas similar to the designated subject area. The tracking unit 161 generates distance information from the parallax image, and uses it for searching the subject area. Details of the configuration and operation of the tracking unit 161 will be described later.

The taking lens 101 (lens unit) has a fixed 1st group lens 102 , a zoom lens 111 , an aperture 103 , a fixed 3rd group lens 121 , a focus lens 131 , a zoom motor 112 , an aperture motor 104 , and a focus motor 132 . A fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, and a focus lens 131 constitute a photographing optical system. Although the lenses 102, 111, 121, and 131 are illustrated as one lens for convenience, they may each be composed of a plurality of lenses. Also, the photographing lens 101 may be configured as a detachable interchangeable lens.

A diaphragm control unit 105 controls the operation of a diaphragm motor 104 that drives the diaphragm 103 to change the aperture diameter of the diaphragm 103 .
A zoom control unit 113 controls the operation of a zoom motor 112 that drives the zoom lens 111 to change the focal length (angle of view) of the photographing lens 101 .

The focus control unit 133 calculates the defocus amount and defocus direction of the photographing lens 101 based on the phase difference between the pair of focus detection signals (the A image and the B image) obtained from the imaging device 141 . The focus control unit 133 then converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 132 . The focus control unit 133 controls the operation of the focus motor 132 based on the drive amount and the drive direction, and drives the focus lens 131 to control the focus state of the photographing lens 101 . In this manner, the focus control unit 133 performs automatic focus detection (AF) using a phase difference detection method. Note that the focus control unit 133 may perform contrast detection AF based on a contrast evaluation value obtained from an image signal obtained from the imaging device 141 .

A subject image formed on the imaging plane of the imaging element 141 by the imaging lens 101 is converted into an electric signal (image signal) by a photoelectric conversion element of each of a plurality of pixels arranged on the imaging element 141 . In this embodiment, m pixels in the horizontal direction and n pixels in the vertical direction (where n and m are plural) are arranged in a matrix in the image sensor 141, and each pixel has two photoelectric conversion elements (photoelectric conversion regions). ) is provided. Signal reading from the imaging device 141 is controlled by the sensor control section 143 according to instructions from the main control section 151 .

(Pixel array of image sensor 141)
FIG. 2 is a diagram schematically showing an arrangement example of pixels in the imaging element 141, and representatively shows an area composed of 16 pixels, 4 pixels in the horizontal direction and 4 pixels in the vertical direction. Each pixel of the imaging element 141 is provided with one microlens 210 and two photoelectric conversion elements 201 and 202 that receive light via the microlens 210 . Since the two photoelectric conversion elements 201 and 202 are arranged in the horizontal direction in the example of FIG. 2, each pixel has the function of dividing the pupil region of the photographing lens 101 in the horizontal direction.

In addition, the imaging element 141 is provided with a color filter in a primary color Bayer arrangement in which four pixels of 2 pixels in the horizontal direction×2 pixels in the vertical direction are repeated units. The color filter has a configuration in which rows in which R (red) and G (green) are repeatedly arranged in the horizontal direction and rows in which G and B (blue) are repeatedly arranged in the horizontal direction are alternately arranged. A pixel 200R provided with a red filter is called a red pixel, a pixel 200G provided with a G (green) filter is called a green pixel, and a pixel 200B provided with a B (blue) filter is called a blue pixel.

In the following description, the first photoelectric conversion element 201 is called an A pixel, the second photoelectric conversion element 202 is called a B pixel, a signal read from the A pixel is called an A signal, and a signal read from the B pixel is called a B signal. be. An image composed of A signals obtained from a plurality of pixels included in a certain region and an image composed of B signals form a set of parallax images. Therefore, the digital camera 100 can generate two parallax images by one shot. Also, by adding the A signal and the B signal for each pixel, it is possible to obtain a signal similar to that of a general pixel that does not have a pupil division function. Hereinafter, this added signal is sometimes referred to as an A+B signal, and an image composed of the A+B signal is sometimes referred to as a captured image.

In this way, from one pixel, the output (A signal) of the first photoelectric conversion element 201, the output (B signal) of the second photoelectric conversion element 202, and the first photoelectric conversion element 201 and the second photoelectric conversion element 201 are output. Three types of signals, ie, the addition output (A+B signal) of the conversion element 202 can be read. The A signal (B signal) may be obtained by subtracting the B signal (A signal) from the A+B signal instead of reading.

Note that the photoelectric conversion elements may be divided and arranged in the vertical direction, and pixels in which the photoelectric conversion elements are divided in different directions may be mixed. Also, the photoelectric conversion element may be divided in both the vertical and horizontal directions. Also, it may be divided into three or more in the same direction.

Returning to FIG. 1, the image signal read out from the imaging element 141 is supplied to the signal processing section 142 . The signal processing unit 142 applies signal processing such as noise reduction processing, A/D conversion processing, and automatic gain control processing to the image signal, and outputs the image signal to the sensor control unit 143 . The sensor control unit 143 stores the image signal received from the signal processing unit 142 in a RAM (random access memory) 154 .

The image processing unit 152 applies predetermined image processing to the image data accumulated in the RAM 154 . Image processing applied by the image processing unit 152 includes so-called development processing such as white balance adjustment processing, color interpolation (demosaicing) processing, and gamma correction processing, as well as signal format conversion processing, scaling processing, subject detection processing, subject recognition processing, and the like. There are, but not limited to, The image processing unit 152 can also generate information about subject brightness for use in automatic exposure control (AE). The results of subject detection processing and subject recognition processing may be used for other image processing (for example, white balance adjustment processing). It should be noted that the image processing unit 152 may generate the AF evaluation value when the contrast detection AF is performed. The image processing unit 152 stores the processed image data in the RAM 154 .

When recording the image data stored in the RAM 154, the main control unit 151 adds, for example, a predetermined header to the image processing data to generate a data file according to the recording format. At this time, the main control unit 151 encodes the image data in the compression/decompression unit 153 as necessary to compress the amount of information. The main controller 151 records the generated data file in a recording medium 157 such as a memory card.

When displaying the image data stored in the RAM 154, the main control unit 151 scales the image data in the image processing unit 152 so as to fit the display size of the display unit 150, and then uses the RAM 154 as a video memory. Write to the area (VRAM area).
The display unit 150 reads image data for display from the VRAM area of the RAM 154, and displays it on a display device such as an LCD or an organic EL display.

The digital camera 100 of the present embodiment functions as an electronic viewfinder (EVF) by immediately displaying the captured moving image on the display unit 150 during moving image shooting (shooting standby state or during moving image recording). Let A moving image and its frame images displayed when the display unit 150 functions as an EVF are called a live view image or a through image.
Further, when a still image is captured, the digital camera 100 displays the last captured still image on the display unit 150 for a certain period of time so that the user can confirm the result of the capturing. These display operations are also realized under the control of the main control unit 151 .

An operation unit 156 includes switches, buttons, keys, a touch panel, and the like for the user to input instructions to the digital camera 100 . An input through the operation unit 156 is detected by the main control unit 151 through the bus 160, and the main control unit 151 controls each unit in order to realize operations according to the input.

The main control unit 151 has one or more programmable processors such as CPU and MPU, for example, and controls each unit by reading a program stored in the storage unit 155 into the RAM 154 and executing it, thereby realizing the functions of the digital camera 100. do. The main control unit 151 also executes AE processing for automatically determining exposure conditions (shutter speed or accumulation time, aperture value, sensitivity) based on subject brightness information. Information on subject brightness can be acquired from the image processing unit 152, for example. The main control unit 151 can also determine exposure conditions based on a specific subject area, such as a person's face.

The main control unit 151 fixes the aperture during moving image shooting, and controls the exposure based on the electronic shutter speed (accumulation time) and gain. The main control unit 151 notifies the sensor control unit 143 of the determined sense of accumulation and the magnitude of the gain. The sensor control unit 143 controls the operation of the imaging device 141 so that shooting is performed according to the notified exposure conditions.

Note that in this embodiment, a total of three images, a set of parallax images and a captured image, can be acquired in one shot, and the image processing unit 152 processes each image and writes them to the RAM 154 . The tracking unit 161 obtains distance information of a subject from a set of parallax images, and uses the information for subject tracking processing for captured images. When subject tracking is successful, the tracking unit 161 outputs information about the position of the subject area in the captured image and information about reliability.

The result of subject tracking can be used, for example, for automatic setting of the focus detection area. As a result, it is possible to realize a tracking AF function for a specific subject area. Also, AE processing can be performed based on the luminance information of the focus detection area, and image processing (for example, gamma correction processing, white balance adjustment processing, etc.) can be performed based on the pixel values of the focus detection area. Note that the main control unit 151 may superimpose an index indicating the position of the current subject area (for example, a rectangular frame surrounding the area) on the display image.

A battery 159 is managed by a power management unit 158 and supplies power to the entire digital camera 100 .
The storage unit 155 stores programs to be executed by the main control unit 151, setting values necessary for executing the programs, GUI data, user setting values, and the like. For example, when the operation unit 156 is operated to instruct the transition from the power OFF state to the power ON state, the program stored in the storage unit 155 is read into a part of the RAM 154, and the main control unit 151 executes the program.

(Configuration and operation of tracking unit)
FIG. 3 is a block diagram showing a functional configuration example of the tracking unit 161. As shown in FIG. The tracking unit 161 has a matching unit 1610 , a feature extraction unit 1620 and a distance map generation unit 1630 . The tracking unit 161 identifies an image area (object area) to be tracked from the designated position, and extracts a feature amount from the object area. Then, in each supplied captured image, using the extracted feature amount, a region having a high degree of similarity with the subject region of the previous frame is searched as the subject region. Also, the tracking unit 161 acquires distance information from the pair of parallax images and uses it to identify the subject area.

The matching unit 1610 searches for a subject area in the supplied image using the feature amount of the subject area supplied from the feature extraction unit 1620 . Although there is no particular limitation on the method of searching for regions based on image feature amounts, matching section 1610 uses at least one of template matching and histogram matching.

Template matching and histogram matching are described below.
Template matching is a technique in which a pixel pattern is set as a template and an area with the highest similarity to the template is searched in the image. A correlation amount such as the sum of absolute differences between corresponding pixels can be used as the degree of similarity between the template and the image region.

FIG. 4A schematically shows a template 301 and its configuration example 302. FIG. When template matching is performed, the feature extraction unit 1620 supplies color (hue) information to be used for the template to the matching unit 1610 as a feature amount. Here, the template 301 has a horizontal pixel count of W and a vertical pixel count of H, and binarization is performed to replace pixels that match the feature amount and pixels that do not match the feature amount with different fixed values. . A matching unit 1610 performs pattern matching using the binarized template 301 .

Therefore, the feature amount T(i,j) of the template 301 used for pattern matching can be expressed by the following formula (1) when the coordinates in the template 301 are represented by the coordinate system shown in FIG. 4(a).
T(i, j) = {T(0, 0), T(1, 0), ..., T(W-1, H-1)} (1)

FIG. 4(b) shows an example of a search area 303 of a subject area and its configuration 305. FIG. A search area 303 is a range in which pattern matching is performed within the image, and may be the entire image or a portion of the image. Coordinates within the search area 303 are represented by (x, y). Also in the search area 303, binarization is performed in which pixels that match the feature quantity and pixels that do not match the feature quantity are replaced with different fixed values. A region 304 has the same size (the number of horizontal pixels W and the number of vertical pixels H) as the template 301, and is a target for calculating the degree of similarity with the template 301. FIG.

The feature amount S(i,j) of the region 304 used for pattern matching can be expressed by the following equation (2) when the coordinates in the template 301 are represented by the coordinate system shown in FIG. 4B.
S(i, j) = {S(0, 0), S(1, 0), ..., S(W-1, H-1)} (2)

ここで、V(x, y)は、領域３０４の左上頂点の座標(x, y)における評価値を表す。 Collating section 1610 calculates a Sum of Absolute Difference (SAD) value shown in Equation (3) below as an evaluation value V(x, y) representing the similarity between template 301 and region 304 .

Here, V(x, y) represents the evaluation value at the coordinates (x, y) of the upper left vertex of the area 304 .

The matching unit 1610 moves the region 304 from the upper left to the right of the search region 303 by one pixel, and when x=(X-1)-(W-1) is reached, then x=0 and downward by one pixel. , and the evaluation value V(x, y) is calculated at each position. The coordinate (x, y) at which the calculated evaluation value V(x, y) indicates the minimum value indicates the position of the region 304 having the pixel pattern most similar to the template 301 . The matching unit 1610 detects the area 304 having the minimum evaluation value V(x, y) as a subject area existing within the search area. Note that when the reliability of the search result is low (for example, when the minimum value of the evaluation value V(x, y) exceeds the threshold), it may be determined that the subject region has not been found.

Here, for pattern matching, an example is shown in which a template that is binarized according to whether it is one of the colors corresponding to the feature amount is used. A valued template may also be used. Also, a feature amount based on brightness or saturation may be used instead of the color feature amount. Moreover, although an example using SAD as an evaluation value of similarity has been shown, other evaluation values such as normalized cross-correlation (NCC) and ZNCC may be used.

Next, the details of histogram matching will be described.
FIG. 5A shows an example of a subject area 401 and its histogram 402. FIG. When performing histogram matching, the feature extracting unit 1620 supplies color (hue) information to be used for the color histogram to the matching unit 1610 as a feature amount. Assuming that the number of bins in the color histogram is M (M is an integer equal to or greater than 2), the color histogram p(m) 402 generated by the matching unit 1620 can be expressed by Equation (4) below.
p(m) = {p(0), p(1), ..., p(M-1)} (4)
Note that p(m) is a normalized histogram. This color histogram p(m) has only bins corresponding to colors included in the features. That is, if the number of bins is M, the number of colors supplied as features is also M.

FIG. 5B shows an example of the search area 403 of the object area and the color histogram 405. FIG. Assuming that the number of bins is M, the color histogram q(m) 405 of the area 404 is expressed by the following equation (5).
q(m) = {q(0), q(1), ..., q(M-1)} (5)
It is assumed that q(m) is a normalized histogram. This color histogram q(m) is also a histogram having only bins corresponding to colors included in the feature amount.

ここで、D(x, y)は、領域４０４の左上頂点の座標(x, y)における評価値を表す。 The tracking unit 161 uses the Bhattacharyya coefficient shown in the following equation (6) as the similarity evaluation value D(x, y) between the color histogram p(m) of the subject region 401 and the color histogram q(m) of the region 404. can be calculated.

Here, D(x, y) represents the evaluation value at the coordinates (x, y) of the upper left vertex of the area 404 .

The matching unit 1610 calculates the evaluation value D(x, y) while shifting the area 404 within the search area 403 in the same manner as in template matching. The coordinates (x, y) at which the calculated evaluation value D(x, y) indicates the maximum value indicate the position of the region 404 that is most similar to the subject region 401 . The matching unit 1610 detects the area 404 having the maximum evaluation value D(x, y) as a subject area existing within the search area.

Although an example of using a color feature amount for histogram matching is shown here, a hue or saturation feature amount may be used. Also, although an example of using the Bhattacharyya coefficient as the similarity evaluation value has been shown, other evaluation values such as a histogram intersection may be used.

A distance map generation unit 1630 calculates the subject distance from a set of parallax images and generates a distance map. A distance map is one type of distance information in which each pixel represents an object distance, and is sometimes called a depth map, a depth image, or a distance image. Note that the distance map may be generated without using the parallax image. For example, by obtaining the position of the focus lens 131 at which the contrast evaluation value is maximized for each pixel, the subject distance for each pixel may be obtained and a distance image may be generated.

A method of calculating the subject distance will be described with reference to FIG. In FIG. 6, assuming that an A image 1151a and a B image 1151b have been obtained, the light flux is refracted as indicated by the solid line based on the focal length of the photographing lens 101 and the distance information between the focus lens 131 and the image sensor 141. I understand. Therefore, it can be seen that the subject in focus is at the position 1152a. Similarly, when the B image 1151c is obtained for the A image 1151a, it is found that there is a subject in focus at a position 1152b, and when the B image 1151d is obtained, there is a subject at a position 1152c. As described above, for each pixel, the distance information of the subject at that pixel position can be calculated from the relative position of the A image including that pixel and the corresponding B image.

For example, assume that an A image 1151a and a B image 1151d are obtained in FIG. In this case, the distance 1153 from the midpoint pixel 1154 corresponding to half the image shift amount to the object position 1152c or the defocus amount corresponding to the distance 1153 is stored as the pixel value of the pixel 1154 . In this way, the distance information of the object can be calculated for each pixel, and the distance map can be generated.

Note that the distance map may be generated by dividing the image into minute areas and calculating the defocus amount for each minute area. An A image and a B image are generated from the pixels included in the minute area, the phase difference (image shift amount) is detected by correlation calculation, and converted into a defocus amount. In this case as well, each pixel in the generated distance map indicates the subject distance, but the pixels included in the minute area indicate the same subject distance. The distance map generator 1630 supplies the generated distance map to the feature extractor 1620 .

Note that the distance map may be generated for the entire image, or may be generated only for a partial area designated for extracting the feature amount.

The feature extraction section 1620 extracts feature amounts used for tracking (searching) the subject area from the subject area.
When subject tracking is performed, the user is generally asked to specify a position in the image to be tracked before tracking is started. For example, in the shooting standby state, the user can designate a position within the image displayed on the display unit 150 through the operation unit 156 . For example, if the display unit 150 is a touch display, the main control unit 151 acquires the coordinates of the tap operation, or the coordinates of the position specified by the cursor that can move on the image through the operation of the operation unit 156 . Information on the designated position is input from the main control unit 151 to the feature extraction unit 1620 .

A method for specifying a subject region from which the feature amount is to be extracted by the feature extraction unit 1620 will be described with reference to FIG. FIG. 7A shows a captured image, and a specified position 503 indicates coordinates within a person's face 501 . It is also assumed that the background house 502 has similar color information to the person's face 501 .

The feature extraction unit 1620 generates a color histogram H _in within the subject area, using a predetermined area including the specified position 503, for example, a predetermined rectangular area centered on the specified position 503 as a temporary subject area. Also, the feature extraction unit 1620 uses all areas other than the temporary object area as a reference area, and generates a color histogram H _Out for this reference area. A color histogram represents the frequency of colors included in an image. Here, as an example, pixel values are converted from the RGB color space to the HSV color space to generate a color histogram for hue (H). However, other types of color histograms may be generated.

Then, feature extraction section 1620 calculates information amount I(a) represented by the following equation (7).
I(a) = _-log2 ( _Hin (a) / _Hout (a)) (7)
where a is an integer indicating the bin number. The absolute value of the information amount I(a) becomes smaller as the ratio of the number of pixels of the color corresponding to the bin included in the virtual subject area to the number of pixels of the color corresponding to the bin included in the reference area increases. . That is, the smaller the value of the information amount I(a), the larger the ratio of the color corresponding to the information amount I(a) included in the provisional subject area than the reference area. It is considered that there is a high possibility that it is a characteristic color of the subject area. The feature extractor 1620 calculates the amount of information I(a) for all bins.

The feature extraction unit 1620 replaces each of the calculated information amounts I(a) with any value within a specific range (for example, the range of 8-bit values (0 to 255)). At this time, the feature extraction unit 1620 replaces the smaller value of the information amount I(a) with a larger value. Then, the feature extraction unit 1620 replaces the value of each pixel included in the captured image with a value obtained by replacing the information amount I(a) corresponding to the color of the pixel.

Through such processing, feature extraction section 1620 generates a subject map based on color information. FIG. 7B shows an example of a subject map, in which pixels close to white are highly likely to be pixels of the subject, and pixels close to black are less likely to be pixels of the subject. Although the object map is shown as a binary image in FIG. 7B for convenience, it is actually a multi-tone image. Since part of the house 502 as the background of the captured image has a color similar to that of the person's face 501, the subject map based on the color information is not sufficient to identify the person's face 501. FIG. A rectangular area 504 shown in FIG. 7C is an example of a finally set (updated) subject area based on, for example, an area in the subject map in which pixel values are equal to or greater than a predetermined threshold.

If the feature amount extracted from such a subject area is used, the possibility that the person's face 501 can be tracked with high accuracy is low. Therefore, in this embodiment, the distance map generated by the distance map generation unit 1630 is used in order to improve the accuracy of the subject area set based on the color information. FIG. 7(d) shows the distance map generated for the captured image shown in FIG. 7(a), with the object distance corresponding to the specified position 503 as a reference, the smaller the difference in object distance, the whiter, and the larger the difference, the blacker. Here is an example converted to be Although the distance map is shown as a binary image in FIG. 7D for convenience, it is actually a multi-tone image.

The feature extraction unit 1620 generates a subject map with the distance information added, for example, by multiplying the values of the corresponding pixels of the distance map and the subject map based on the color information. FIG. 7(e) shows an example of a subject map with distance information (that is, based on both color information and distance information). In the subject map shown in FIG. 7E, the person's face 501 and the background house 502 can be distinguished with high accuracy. A rectangular area 505 shown in FIG. 7(f) is an example of a subject area set based on an area having pixel values equal to or greater than a predetermined threshold in the subject map shown in FIG. 7(e), for example. A rectangular area 505 is a rectangular area circumscribing the person's face 501 and contains very few background pixels. Using the feature amount extracted from such a subject area increases the possibility that the person's face 501 can be tracked with high accuracy.

In this way, by referring to the distance information in addition to the color information about the specified range including the specified position, it is possible to set the subject area with higher accuracy and extract the feature quantity suitable for accurate tracking. become.

At the time when the position of the tracked object is specified, effective (reliable enough to refer to) distance information may not be obtained for the specified position and its neighboring area. For example, if the distance map is generated only for a specific area (for example, focus detection area) and the specified position is outside the specific area, or if the specified position is out of focus, the reliability of the distance information is low. Such cases can be considered.

Therefore, if the feature extraction unit 1620 obtains distance information with sufficient reliability to refer to the vicinity of the designated position (temporary subject area), the feature extraction unit 1620 sets the subject area by referring to the distance information in addition to the color information. do. On the other hand, if distance information with sufficient reliability to refer to the vicinity of the specified position (temporary subject area) is not obtained, the feature extraction unit 1620 extracts the subject area based on the color information without referring to the distance information. set. Note that the distance information with sufficient reliability to refer to is, for example, the distance obtained when the provisional subject area is in focus or in a state close to focus (i.e., when the defocus amount is equal to or less than a predetermined threshold). It may be information, but is not limited to this.

(Processing flow of imaging device)
A moving image shooting operation accompanied by subject tracking processing by the digital camera 100 of the present embodiment will be described with reference to the flowcharts of FIGS. 8 and 9. FIG. The moving image shooting operation is executed during shooting standby or movie recording. Details such as the resolution of images (frames) handled during shooting standby and during video recording are different, but the content of the processing related to subject tracking is basically the same, so the following description will be given without any particular distinction. .

In S801, the main control unit 151 determines whether the power of the digital camera 100 is ON, ends the process if it is not determined to be ON, and advances the process to S802 if it is determined to be ON.
In S802, the main control unit 151 controls each unit, executes imaging processing for one frame, and advances the processing to S803. Note that here, a set of parallax images and a captured image for one screen are generated and stored in the RAM 154 .

In S803, the main control unit 151 causes the tracking unit 161 to execute subject tracking processing. Details of the processing will be described later. By subject tracking processing, the tracking unit 161 notifies the main control unit 151 of the position and size of the subject area. The main control unit 151 sets the focus detection area based on the notified object area.

In S804, the main control unit 151 causes the focus control unit 133 to execute focus detection processing. The focus control unit 133 connects a plurality of A signals obtained from a plurality of pixels arranged in the same row among the plurality of pixels included in the focus detection region in the pair of parallax images to form a plurality of A images. B signals are spliced together to generate a B image. Then, the focus control unit 133 calculates the amount of correlation between the A image and the B image while shifting the relative positions of the A image and the B image, and determines the relative position where the similarity between the A image and the B image is the highest. It is obtained as a phase difference (shift amount) between the A image and the B image. Furthermore, the focus control unit 133 converts the phase difference into a defocus amount and a defocus direction.

In S805, the focus control unit 133 drives the focus motor 132 according to the lens drive amount and drive direction corresponding to the defocus amount and the defocus direction obtained in S804, moves the focus lens 131, and returns the process to S801.

Thereafter, the processes of S802 to S805 are repeatedly executed until it is no longer determined in S801 that the power switch is ON. As a result, a subject region is searched for a plurality of time-series images, and a subject tracking function is realized. In FIG. 8, the subject tracking process is performed every frame, but it may be performed every several frames for the purpose of reducing the processing load and power consumption.

(subject tracking processing)
Next, details of the subject tracking processing in S803 will be described with reference to the flowchart of FIG.
In S901, the tracking unit 161 determines whether or not an instruction to start subject tracking has been detected. Note that the start instruction may be, for example, a designation input of a tracking position from the operation unit 156 . Information on the specified position is notified from the main control unit 151 . At this point, there is a high possibility that the distance information for the specified position has not been obtained, or that the specified position is out of focus and therefore the reliability of the distance information is low. Therefore, the processing contents are different from those after the focus detection processing is performed for the specified position.

In S902, the tracking unit 161 (feature extraction unit 1620) determines whether or not effective (highly reliable) distance information has been obtained for the specified position and its vicinity. , the process advances to S903 if it is not determined to have been obtained.

In S903, the tracking unit 161 (feature extraction unit 1620) identifies the subject area from the specified position using only the color information as described above, extracts the feature amount of the subject area, and advances the process to S905.

In S904, the tracking unit 161 (feature extraction unit 1620) identifies the subject area from the specified position using both the color information and the distance information as described above, and extracts the feature amount (pixel pattern or histogram) of the subject area. The process proceeds to S905.

In S905, the tracking unit 161 (collation unit 1610) performs matching processing on the search area of the captured image using the feature amount extracted in S903 or S904, and searches for an area with the highest similarity of the feature amount. . The tracking unit 161 notifies the main control unit 151 of information regarding the position and size of the searched area as a tracking result, and terminates the tracking process.

On the other hand, in S906, the tracking unit 161 (feature extraction unit 1620) determines whether or not the most recently extracted feature amount is extracted from the subject area specified using both the color information and the distance information. Then, if the tracking unit 161 (feature extraction unit 1620) determines that the most recently extracted feature amount is extracted from the subject area specified using both the color information and the distance information, the process proceeds to S905. If not, the process proceeds to S907.

In S907, the tracking unit 161 (feature extraction unit 1620) determines whether or not effective distance information has been obtained for the subject area detected by the previous collation. If it is determined not to have been obtained, the process proceeds to S905.

In S908, the tracking unit 161 (feature extraction unit 1620) identifies (updates) the subject area again from the designated position using both the color information and the distance information in the same manner as in S904, and extracts the feature amount of the updated subject area. The process proceeds to S905. It should be noted that the feature amount extracted in S908 may be added with the feature amount extracted in the past (for example, extracted in the immediately preceding process of S903).

In the matching process executed in S905 during the continuation process, if the feature amount has been updated in S908, the updated feature amount is used, and if the feature amount has not been updated in S908, the most recently extracted feature amount is continued. used.

For example, even if the focus detection process for the subject area detected by the previous collation has started, the reliability of the distance information cannot be said to be high unless the defocus amount is equal to or less than a predetermined threshold value. In such a case, processing is performed in the steps of S901, S906, S907, and S905.
If the defocus amount of the tracked subject area is equal to or less than a predetermined threshold value, highly reliable distance information can be obtained for the subject area. In such a case, processing is performed in the steps of S901, S906, S907, S908, and S905.
When the subject area can be specified using not only the color information but also the highly reliable distance information, the subject area and the feature amount are updated, and the updated feature amount is used in subsequent tracking processing. In this case, the process is performed in steps S901, S906, and S905.

As described above, according to the present embodiment, when specifying an image region (subject region) to be tracked based on a specified position in an image, distance information is used in addition to color information of the image to determine the subject region. can improve the accuracy of Therefore, it is possible to improve the accuracy of the tracking process using the feature amount extracted from the subject area.

If the reliability of the distance information is not high, the subject area is specified based on the color information until the reliability becomes high, and the distance information is updated when the reliable distance information becomes available. It is further used to re-identify (update) the subject area. Therefore, even if a position for which distance information is not obtained or a position for which the reliability of distance information is low is specified as a tracking target, the accuracy of tracking processing can be improved over time.

● <Second embodiment>
In the first embodiment, the feature amount is not updated when the feature amount can be extracted from the subject area specified based on the highly reliable distance information and color information. This makes it possible to avoid the accumulation of drift and achieve subject tracking that is resistant to occlusion. On the other hand, for example, when the environment in which the subject exists changes, the tracking accuracy of the subject may decrease when the brightness or hue of the subject changes from when the feature amount is extracted.

Therefore, in this embodiment, when the difference in the distance information between the subject area and its surrounding area satisfies a predetermined condition, the feature amount extracted using the highly reliable distance information is also updated. . Since the present embodiment can be implemented with the digital camera 100 having the configuration shown in FIG. 1 as in the first embodiment, the differences in operation from the first embodiment will be mainly described below.

A moving image shooting operation accompanied by subject tracking processing by the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. 10 .
S1001 to S1003 and S1005 to S1006 in FIG. 10 are the same as S801 to S805 in FIG. The present embodiment differs from the first embodiment in that after subject tracking processing is performed in S1003, feature amount update processing is performed in S1004.

Next, details of the feature amount update processing performed in S1004 of FIG. 10 will be described using the flowchart of FIG.
In S1101, the tracking unit 161 (feature extraction unit 1620) determines whether the difference in distance information between the subject region and its surrounding regions is large, based on the subject region searched in the matching process (S905) and the obtained distance information. determine whether or not

FIGS. 12(a) and 12(c) are different captured images, and FIGS. 12(b) and 12(d) are generated for the captured images of FIGS. 12(a) and 12(c), respectively. Schematically shows a distance map that has been generated. In FIG. 12(a), a house 1202 exists as a background with a distance behind the person 1201, and another person 1206 exists in front of the person 1205 in FIG. 12(c).

In the distance map of FIG. 12B, the distance information of each pixel is shown in white when the difference with respect to the distance information corresponding to the person 1201 to be tracked is small, and in black when the difference is large. Similarly, the distance map of FIG. 12B shows the distance information of each pixel in white as the difference with respect to the distance information corresponding to the person 1205 to be tracked is smaller, and in black as the difference is larger. Although FIGS. 12(b) and 12(d) show the distance maps as binary images for drawing purposes, they are actually multi-valued grayscale images. Note that the reference distance information corresponding to the subject area may be the average value of the distance information, the most frequently used distance information, or the like.

Area 1203 in FIG. 12(b) and area 1207 in FIG. 12(d) are subject areas identified by subject tracking processing in S1003, and areas 1204 and 1208 are areas surrounding areas 1203 and 1207, respectively. . Here, the peripheral area of the subject area is obtained by enlarging the subject area by the same amount in the vertical and horizontal directions, and excluding the subject area from an area whose size in the horizontal direction and the vertical direction is three times the size of the subject area. defined as a hollow region. However, this is just an example, and other methods may be used.

The tracking unit 161 (feature extraction unit 1620) extracts an area having distance information similar to (within a predetermined range of difference) the distance information in the main subject area from the surrounding area, and extracts the extracted area as the surrounding area. It is determined whether or not the ratio in the area is equal to or greater than a predetermined threshold. The tracking unit 161 (feature extraction unit 1620) ends the feature update processing if the ratio is determined to be equal to or greater than the threshold, and advances the processing to S1102 if the ratio is not determined to be equal to or greater than the threshold.

The determination in S1101 will be described. If the proportion of the surrounding area that has distance information similar to the distance information in the main subject area is low (for example, if it is less than a threshold), the subject area to be tracked and the background area can be clearly distinguished. It is believed that there is. Therefore, even if the feature amount is updated based on the captured image that satisfies this condition, the effect of the background on the updated feature amount is considered to be small.

Conversely, if a portion of the surrounding area has distance information similar to the distance information in the main subject area at a high rate (e.g., if it is equal to or greater than the threshold), it is difficult to distinguish between the subject area to be tracked and the background area. situation.

In the examples of FIGS. 12(b) and (d), the areas shown in white are areas having distance information similar to the distance information corresponding to the main subject area. The threshold used in S1101 can be determined experimentally, for example. Here, in the example shown in FIG. In the example shown in 12(d), it is determined to be greater than or equal to the predetermined threshold.

In S<b>1102 , the tracking unit 161 (feature extraction unit 1620 ) extracts a new feature amount from the object region searched for in the matching process, based on the evaluation value (Equation (3)) calculated in the matching process. determines whether or not the degree of similarity with the feature quantity used for searching the object region is low. Specifically, the feature extraction unit 1620 determines whether the new evaluation value calculated by the matching unit 1610 is higher than the update threshold, or whether the evaluation value based on the Bhattacharyya coefficient (equation (6)) is determined by another update. Determine whether or not it is lower than the threshold.

If a feature quantity with low similarity to the feature quantity used for the search is extracted from the searched subject region, the subject region could be searched, but the appearance of the subject region has changed, and the feature quantity is It is considered that there is a high need for updating. On the other hand, if a feature quantity having a high degree of similarity to the feature quantity used for searching is extracted from the searched subject region, the change in the appearance of the subject region is small, and the need to update the feature quantity is low. it is conceivable that.

Therefore, if the tracking unit 161 (feature extraction unit 1620) determines that the similarity is low in S1102, the process proceeds to S1103, and if the similarity is not determined to be low, the feature update process ends.

In S<b>1103 , the tracking unit 161 (feature extraction unit 1620 ) updates the feature amount used for matching processing with the new feature amount extracted from the searched subject area, as in S<b>908 . There are no particular restrictions on the method of updating. For example, the feature quantity used in the matching process until then may be completely replaced with a new feature quantity, or the feature quantity after updating using the feature quantity used in the matching processing and the new feature quantity may be calculated. For example, if the evaluation value (equation (3)) is based on the sum of absolute differences, the updated feature amount can be obtained according to equation (8).
T(i, j) = Tpre(i, j) × α + Tnow(i, j) × (1-α) , 0 ≤ α ≤ 1 (8)
Here, Tpre (i, j) is the feature amount used in the matching process, Tnow (i, j) is the new feature amount, and T(i, j) is the updated feature amount.

Further, if the evaluation value (equation (6)) is based on the Bhattacharyya coefficient, the updated feature amount can be obtained according to equation (9).
p(m) = ppre(m) x α + pnow(m) x (1-α) , 0 ≤ α ≤ 1 (9)
Here, ppre (m) indicates the feature amount used in the matching process, pnow (m) indicates the new feature amount, and p(m) indicates the updated feature amount.

In both equations (8) and (9), α=0 indicates an update that completely replaces with new extracted features, and α=1 indicates that the features are not updated. The update degree α can be adaptively determined according to at least one of the magnitude of the difference in distance information determined in S1101 and the similarity determined in S1102, for example.

For example, after satisfying the determination conditions in S1101 and S1102, the greater the difference in distance information and the lower the degree of similarity, the smaller the value of the update degree α (the greater the contribution of the new feature amount). The updated feature amount can be calculated. Further, after satisfying the determination conditions in S1101 and S1102, the smaller the difference in the distance information and the higher the similarity, the larger the value of the degree of update α (the smaller the contribution of the new feature amount). The updated feature amount can be calculated.

Furthermore, when an operation to determine the focus distance or exposure (for example, an operation corresponding to an instruction to prepare for shooting or an instruction to start shooting, and an operation of the shutter button included in the operation unit 156) is detected, the subject is positioned at that point. It is considered highly probable that the tracking process was successful. Therefore, when an operation to confirm the focus distance or exposure is detected, the determinations in S1101 and S1102 are made so as to facilitate updating with new feature amounts extracted from the subject area detected at that time. You may make it change the threshold value to be used.

As described above, according to the present embodiment, the feature amount can be updated when the feature amount can be accurately extracted from the subject area using the distance information. Therefore, even if the appearance of the tracked subject area changes, it is possible to update the feature amount without deteriorating the tracking accuracy, and it is possible to further improve the subject tracking performance.

(Other embodiments)
In the above-described embodiment, the case where subject tracking is performed during shooting has been described, but if distance information can be obtained, subject tracking can be performed in the same way during playback of moving images. In this case, the distance information recorded in the frames of the moving image may be acquired, or if each frame is recorded in the form of a set of parallax images, the distance information is generated from the parallax images, and the parallax images are generated. are combined to generate a video frame for playback. Of course, you may acquire distance information by another method.

When subject tracking is performed during playback, the tracking result can be used, for example, to control the display method of moving images. For example, it is possible to control so that the subject area being tracked is displayed in the center of the screen, or to be displayed after being scaled so that the size of the subject area being tracked is constant. . Also, an index (for example, a circumscribing rectangular frame of the subject area) that specifies the subject area being tracked may be superimposed and displayed. Note that these are merely examples, and the tracking results may be used for other purposes.

The superimposed display of the index for specifying the subject area being tracked may be different between when the subject area is specified by referring to the distance information and when it is specified by using only the color information. For example, if the subject area is specified using only color information, the accuracy of the subject area may be low, so an index of fixed position and size is displayed. Further, when the subject area is specified by referring to the distance information, the position and size of the index are dynamically changed according to the position and size of the subject area.

In addition, the present invention is applicable not only to moving images, but also to shooting and playing back a plurality of images in time series, such as continuous shooting and interval shooting.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Moreover, the above-described embodiments are merely specific examples for the purpose of helping understanding of the present invention, and are not intended to limit the present invention to the above-described embodiments in any sense. All embodiments falling within the scope defined by the claims are included in the present invention.

DESCRIPTION OF SYMBOLS 100... Digital camera 101... Shooting lens 131... Focus lens 132... Focus motor 133... Focus control part 141... Imaging element 151... Main control part 152... Image processing part 161... Tracking part

Claims (8)

an extraction means for extracting a feature amount from an image region;
searching means for searching for a region similar to the image region in a plurality of time-sequential images using the feature amount;
The extracting means is
Among the distance information in the peripheral area of the area similar to the image area, an area having distance information whose difference from the distance information in the area similar to the image area is within a predetermined range on the short distance side and the long distance side. updating the feature quantity used by the searching means for searching when the ratio of the surrounding area is not equal to or greater than a threshold ;
The image processing apparatus according to claim 1, wherein, when the ratio is equal to or greater than the threshold value, the feature quantity used by the searching means for searching is not updated . The extracting means extracts the above-described distance information according to the ratio of the areas in the peripheral area having distance information in which the difference from the distance information in the area similar to the image area is within a predetermined range on the short distance side and the long distance side . 2. The image processing apparatus according to claim 1 , wherein the searching means changes the degree of update of the feature quantity used for searching. The extraction means is
If the ratio is not equal to or greater than the threshold value, determining the degree of similarity between the feature amount used for searching by the searching means and the feature amount extracted from a region similar to the image region;
When the similarity is determined to be low, updating the feature amount used for searching by the searching means,
3. The image processing apparatus according to claim 1, wherein when the similarity is not determined to be low, the search means does not update the feature amount used for the search. an image processing apparatus according to any one of claims 1 to 3 ;
focus detection means for performing focus detection on an area searched by the search means and including an area similar to the image area;
An imaging device characterized by comprising: 5. The imaging method according to claim 4, wherein the threshold is changed so that the feature quantity used by the search means for searching is easily updated when an operation to determine the focal distance or the exposure is detected. Device. an imaging device having a function of dividing a pupil region of a photographing lens;
generating means for generating the distance information from the parallax image obtained from the imaging element;
6. The imaging device according to claim 5, further comprising: an extracting step in which the extracting means extracts the feature amount from the image region;
a searching step of searching for a region similar to the image region in a plurality of time-series images using the feature amount,
In the extraction step,
Among the distance information in the peripheral area of the area similar to the image area, an area having distance information whose difference from the distance information in the area similar to the image area is within a predetermined range on the short distance side and the long distance side. updating the feature amount used for searching in the searching step when the ratio of the surrounding area is not equal to or greater than a threshold ;
A control method for an image processing apparatus , wherein the feature amount used for searching by the searching means is not updated when the ratio is equal to or greater than the threshold value . A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 3 .

Images