JP6452481B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus and a control method thereof, and more particularly to a technique for synthesizing a plurality of images.

  A technique for acquiring an image with reduced noise or acquiring an image with an expanded dynamic range by aligning and synthesizing a plurality of images with overlapping shooting ranges is known. In such an image synthesis technique, a method of geometrically transforming an image to be synthesized and aligning it with a reference image is known, and affine transformation and projective transformation are used as typical geometric transformations.

  A coefficient defining the geometric transformation (geometric transformation coefficient) can be calculated from a plurality of motion vectors representing the direction and magnitude of the motion of the subject included in both the image to be synthesized and the reference image. The movement of the subject between images can be broadly divided into those that occur in the entire image due to changes in the shooting position and / or shooting direction as in hand-held shooting, and those that occur when the subject itself moves relative to the background. The motion vector representing the former movement is called a global motion vector, and the motion vector representing the latter movement is called a moving object motion vector.

  In order to align a stationary subject between images, it is necessary to use a geometric transformation coefficient calculated only by a global motion vector. Further, in order to align the moving subject, it is necessary to use a geometric transformation coefficient calculated from the motion vector of the moving object for each subject.

  In Patent Document 1, it is determined whether each block is a background or a moving subject from the degree of coincidence between a global motion vector and a local motion vector calculated for each block by dividing an image into a plurality of blocks. A method for determining a motion vector to use is disclosed.

  Patent Document 2 discloses a method of calculating a frequency distribution of motion vectors obtained for each of a plurality of templates and determining a motion vector having the maximum frequency as a global motion vector.

JP 2012-142828 A JP 2006-94560 A

  The amount of movement of the subject in the captured image due to the movement of the camera varies depending on the subject distance, and the closer the subject, the larger the amount of movement in the image. However, in Patent Literature 1 and Patent Literature 2, since the subject distance is not considered in the calculation of the global motion vector, there is a possibility that a motion vector related to a short-distance still subject is erroneously determined as a motion vector of a moving subject.

  In addition, since the motion vector of the moving object becomes dominant when the area occupied by the moving subject in the image is large, the motion vector of a large number of moving objects may be erroneously determined as a global motion vector.

  The present invention provides an image processing apparatus capable of accurately discriminating between a stationary subject and a moving subject in an image by considering the distance of the subject, and a method for controlling the same, in view of the problems of the prior art. With the goal.

  The above-described object is based on a specific distance based on a specific distance, a grouping unit that groups image regions using distance information, a detection unit that detects a motion relative to another image for each group, and a magnitude of the motion. And a normalizing unit that normalizes the image and a determination unit that determines, for each group, whether the region is a stationary subject region or a moving subject region based on a motion whose size is normalized. Achieved by processing equipment.

  With such a configuration, according to the present invention, it is possible to provide an image processing apparatus capable of accurately discriminating between a stationary subject and a moving subject in an image by considering the distance of the subject, and a control method therefor. .

1 is a block diagram illustrating a functional configuration example of a digital camera as an example of an image processing apparatus according to an embodiment; The figure which shows the structural example of the imaging part 205 of FIG. The figure for demonstrating operation | movement of the distance information calculation part 210 of FIG. Flowchart for explaining the alignment process according to the first embodiment. Flowchart for explaining the alignment process according to the first embodiment. The figure which shows the imaging | photography scene for demonstrating 1st Embodiment. The figure for demonstrating the grouping process in 1st Embodiment The figure for demonstrating the calculation process of the motion vector in embodiment The figure which shows the example of the representative motion vector and normalization representative motion vector in 1st Embodiment The figure for demonstrating the production | generation process of the estimation global motion vector in embodiment The figure for demonstrating the motion determination process in embodiment Flowchart for explaining redetermination processing in the embodiment The figure for demonstrating the redetermination process in embodiment The figure for demonstrating the example of the method of calculating a geometric transformation coefficient from the other geometric transformation coefficient in embodiment. The figure which shows the imaging | photography scene for demonstrating 2nd Embodiment. Flowchart for explaining the operation of the alignment processing according to the second embodiment. The figure for demonstrating the grouping process of the 1st step | paragraph in 2nd Embodiment. The figure for demonstrating the grouping process of the 2nd step | paragraph in 2nd Embodiment. The figure which shows the example of the representative motion vector in 2nd Embodiment The figure which shows the example of the normalization representative motion vector in 2nd Embodiment.

● (first embodiment)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Here, as an example of an image processing apparatus capable of acquiring subject distance information, each pixel of the image sensor has a plurality of photoelectric conversion regions, and a plurality of parallax images can be acquired by one photographing (exposure). An example in which the present invention is applied to a digital camera having various configurations will be described. However, the parallax image may be acquired by other methods such as using a multi-lens camera. Further, if the distance information of the subject can be acquired, the parallax image is not necessarily used.

  Furthermore, in the present invention, an imaging function and a function for generating subject distance information are not essential. For example, a configuration for acquiring a plurality of pieces of image data to be combined and corresponding subject distance information from a storage device in which image data and distance information of the subject in the image represented by the image data are stored in association with each other It may be. The distance information may be information in a format called a distance image, a depth image, a depth map, or the like.

FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 200 according to the present embodiment.
The system control unit 201 is, for example, a CPU, reads out a control program for each block included in the digital camera 200 from the ROM 202, develops it in the RAM 203, and executes it to control the operation of each block and realize various functions of the digital camera 200 To do.

  The ROM 202 is a rewritable nonvolatile memory, and stores parameters necessary for the operation of each block, GUI data, and the like in addition to the control program for each block included in the digital camera 200. The ROM 202 also stores lens information such as the exit pupil distance.

  The RAM 203 is a rewritable volatile memory, and is used as a work area for the system control unit 201 and a temporary storage area for data output from each block in addition to loading a control program.

  The optical system 204 includes a stop and a focus lens, and forms a subject image on the imaging unit 205. The imaging unit 205 (imaging means) is an imaging element such as a CCD or a CMOS sensor, for example, photoelectrically converts the subject image formed by the optical system 204 at each pixel, and A / D converts the obtained analog image signal The data is output to the unit 206.

  As described above, the digital camera 200 of the present embodiment can acquire a plurality of parallax images by one shooting (exposure). Various cameras such as a stereo camera such as a stereo camera are known as such cameras. In this embodiment, the imaging unit 205 (imaging device) is placed on the light receiving surface of the exit pupil of the optical system 204. Is provided with a microlens array (hereinafter referred to as MLA). Then, by detecting the phase difference between a plurality of parallax images obtained by such an image sensor, distance information (depth information) of the subject can be obtained. Moreover, a normal captured image can also be obtained by adding parallax images.

  The A / D conversion unit 206 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data. For example, the system control unit 201 stores the digital image data in the RAM 203.

  An image processing unit 207 (image processing means) applies image processing such as gamma correction processing, white balance adjustment processing, and color interpolation processing to the image data stored in the RAM 203. Note that the image processing unit 207 generates a normal image by adding parallax image data, and performs encoding processing and decoding processing of image data as necessary.

  The recording medium 208 is, for example, a removable memory card or the like, and image data processed by the image processing unit 207, image data output by the A / D conversion unit 206, and the like are recorded as an image file in a predetermined format. The

A bus 209 connects the blocks so that control signals and data can communicate with each other.
A distance information calculation unit 210 (fourth calculation unit) reads the parallax image data processed by the image processing unit 207 and calculates distance information for each pixel of the image. Details of the operation of the distance information calculation unit 210 will be described later.

  The conversion coefficient calculation unit 211 acquires reference image data, image data to be combined with the reference image data (combined image data), and distance information corresponding to the combined image data. Then, the conversion coefficient calculation unit 211 (grouping means) extracts subject areas in the synthesized image using the distance information, and groups the subject areas. Alternatively, the conversion coefficient calculation unit 211 classifies the region of the combined image into groups for each distance range. A conversion coefficient calculation unit 211 (first calculation unit) calculates a motion vector for each subject group, and uses a geometric conversion coefficient for aligning the subject region (subject group region) of the subject group with the reference image as a motion vector. Calculate based on

  The conversion coefficient calculation unit 211 (third calculation unit) further obtains determination information output from the conversion coefficient determination unit 212 described later, and uses only the motion vector of the subject group area determined as global movement from the determination information. The geometric transformation coefficient for the entire image is calculated.

  The conversion coefficient determination unit 212 (generation unit) generates a representative motion vector for each subject group using the geometric conversion coefficient calculated by the conversion coefficient calculation unit 211. Then, the transform coefficient determination unit 212 (normalization unit and estimation unit) normalizes the representative motion vector by a specific distance and converts it into a motion that does not depend on the subject distance, and then global motion of the reference image and the synthesized image. Is estimated. Then, the conversion coefficient determination unit 212 (determination unit) determines whether the movement of the subject group area is a global movement (a stationary subject) or an individual movement (a moving subject), and sets determination information representing the determination result for each subject group. Output to. The conversion coefficient calculation unit 211 and the conversion coefficient determination unit 212 constitute detection means for detecting a motion with respect to another image for each group.

The geometric conversion unit 213 (conversion unit) acquires the combined image data and the geometric conversion coefficient for each subject group, converts the subject region belonging to each group in the combined image based on the corresponding geometric conversion coefficient, Data of a composite image aligned with the reference image is generated.
Alternatively, the geometric conversion unit 213 acquires the combined image data and the geometric conversion coefficient for the entire combined image, converts the combined image based on the geometric conversion coefficient, and converts the combined image aligned with the reference image. Generate data.
The geometric conversion unit 213 (synthesizing unit) may further perform a process of combining the generated combined image data with the reference image data.

  Next, a method for generating subject distance information in the present embodiment will be described. First, a configuration example of the imaging unit 205 will be described with reference to FIGS. 2 (a) to 2 (d). FIG. 2A schematically shows a state in which the imaging unit 205 is viewed from the front and side of the digital camera 200. An MLA 141 is formed on the light-receiving surface of the pixel group 143 included in the imaging unit 205, and each pixel constituting the pixel group 143 includes two microlenses 142 and two as shown in FIGS. It is composed of two photodiodes (photoelectric conversion regions) 143a and 143b. Hereinafter, the photodiode 143a is referred to as an A image photodiode (A pixel), and the photodiode 143b is referred to as a B image photodiode (B pixel).

  FIG. 2D conceptually shows the exit pupil 144 of the optical system 204. The A image pupil 145a and the A pixel 143a, and the B image pupil 145b and the B pixel 143b have a conjugate relationship by the microlens 142, respectively. . Accordingly, each pixel of the imaging unit 205 has a pupil division function, and the light beam that has passed through the A image pupil 145a in the right half of the exit pupil 144 is present in the A pixel 143a, and the left half of the exit pupil 144 is present in the B pixel 143b. The light beam that has passed through the B image pupil 145b enters. Therefore, an image composed of the A pixel group and an image composed of the B pixel group are parallax images.

  For a plurality of pixels, an image signal composed of an A pixel group is an A image, an image signal composed of a B pixel group is a B image, and a deviation amount between the A image and the B image is detected. The defocus amount and defocus direction can be detected. Accordingly, automatic focus detection (AF) using the phase difference detection method can be realized from the signal output from the imaging unit 205.

  FIGS. 2E to 2G schematically show the principle of focus detection by the phase difference detection method. 2 (e) shows how the subject is in focus before the subject, FIG. 2 (f) shows how the subject is in focus, and FIG. 2 (g) shows the subject behind the subject. Represents the state of suitability. Reference numerals 147a and 147b schematically represent an A image and a B image obtained from the pixel group in the focus detection area set in the imaging unit 205, respectively. In the case of FIG. 2F, there is no deviation between the A image 147a and the B image 147b, and the subject is in focus. In the case of FIG. 2E, the A image 147a is separated from the center on the left side, and the B image 147b is separated on the right side with respect to the center. In the case of FIG. In addition, the B image 147b is separated to the left with respect to the center. The distance information of the subject can be obtained from the deviation amount (defocus amount) of the A image and the B image, the deviation direction from the center, the focal length of the optical system 204, and the distance between the imaging unit 205 and the focus lens.

  Next, a distance information calculation method performed by the distance information calculation unit 210 will be described with reference to FIG. FIG. 3 schematically shows a method for calculating the subject distance. Assuming that an A image 151a and a B image 151b are obtained, it can be seen from the focal length of the optical system 204 and the distance information between the focus lens and the imaging unit 205 that the light beam is refracted as shown by a solid line. Therefore, it can be seen that the subject in focus is at the position 152a. Similarly, when the B image 151c is obtained for the A image 151a, it can be seen that there is a subject in focus at the position 152b, and when the B image 151d is obtained, the position 152c is in focus. As described above, in each pixel, distance information of the subject at the pixel position can be calculated from the relative position between the A image signal including the pixel and the corresponding B image signal.

  For example, when the A image 151a and the B image 151d are obtained in FIG. 3, the distance 153 from the intermediate point pixel 154 to the subject position 152c corresponding to half of the image shift amount is used as the subject distance information in the pixel 154. Remember. In this way, it is possible to hold subject distance information for each pixel. The distance information may be stored as a depth image or a distance image.

  Next, with reference to the flowcharts shown in FIG. 4A and FIG. 4B, the overall alignment process of the composite image in this embodiment will be described. The following alignment process is sequentially applied to each of the plurality of images excluding the reference image. Here, it is assumed that a plurality of images to be combined are acquired in advance by continuous shooting or the like.

S1500
The system control unit 201 uses the parallax images constituting the composite image, calculates distance information by the distance information calculation unit 210 using the method described above, and stores the distance information in, for example, the RAM 203. Although this is also described above, the distance information calculation process is not essential, and S1500 can be omitted when the distance information calculated in advance is acquired.

The system control unit 201 uses the conversion coefficient calculation unit 211 and the conversion coefficient determination unit 212 in steps S1501 to S1512 to calculate a geometric conversion coefficient for alignment and determine a stationary subject and a moving subject.
S1501
The conversion coefficient calculation unit 211 groups subjects in the combined image based on the distance information. A specific example of grouping will be described with reference to FIGS.
FIG. 5A is a diagram illustrating an example of a shooting scene. A stationary subject A 100, a stationary subject B 101, a moving subject a 104, a stationary subject C 102, and a stationary subject D 103 exist in the order from the digital camera 200. The background portion may also be handled as one or more subjects, but will be omitted in the following description.

  FIG. 6A schematically shows the distance from the subject included in the shooting scene of FIG. 5A and the digital camera 200. FIG. 6B is a histogram of distance information for each pixel calculated for the composite image, and the horizontal scale indicating the subject distance corresponds to FIG.

  In the present embodiment, the transform coefficient calculation unit 211 determines a group of distance information having a frequency equal to or higher than a predetermined threshold, including a maximum frequency, in a frequency distribution (histogram) of distance information of a combined image as one subject. Group subjects as a group. The boundary between the groups can be determined as a minimum frequency or a frequency boundary that is less than the threshold value, but two subject groups existing at close distances may be combined into one subject group. By these methods, the subject areas in the image are grouped so that the areas included in the same distance range belong to the same group.

5 to 6, the stationary subject A 100, the stationary subject B 101, and the stationary subject D 103 are grouped into independent subject groups A, B, and D, respectively, and the moving subject a 104 and the stationary subject that are close to each other are grouped. C 102 is grouped into one subject group C.
FIG. 5B shows the result of grouping as a region of the combined image. Subject groups A to D correspond to subject group regions 500 to 503, respectively.

  As a result of grouping, the conversion coefficient calculation unit 211 represents information (for example, coordinate information) that represents a representative distance (for example, distance information having the highest frequency) of each subject group and an image region (subject group region) corresponding to each subject group. Are stored in the RAM 203, for example.

S1502
Next, the conversion coefficient calculation unit 211 calculates a motion vector for each subject group region. In this embodiment, the conversion coefficient calculation unit 211 calculates a motion vector based on template matching.

A motion vector calculation method will be described with reference to FIG. 7A and 7B are close-up views of the area of the subject A 100 (subject group area 500) in the combined image and the reference image.
The conversion coefficient calculation unit 211 sets a rectangular area of n × n pixels as the target area 600 in the subject group area 500 in the composite image. The center coordinates of the target area 600 are the start point coordinates of the motion vector.

  Next, the conversion coefficient calculation unit 211 sets a search area 601 in the subject group area 500 in the reference image. The search area 601 includes an area corresponding to the target area 600 and is wider than the target area 600. Then, the transform coefficient calculation unit 211 sets the coordinates of the center of the region having the highest degree of coincidence with the target region 600 in the search region 601 as the end point coordinate of the motion vector. Since detection of a motion vector using template matching is known, the description of further details is omitted. The conversion coefficient calculation unit 211 sets target areas at a plurality of locations for each subject area, and detects a motion vector for each target area. Conditions can be set in advance for the size, position, and number of target areas to be set, but the conditions are set uniformly in the subject area.

FIG. 7C is a diagram schematically showing motion vectors 700 to 703 calculated for the subject group regions 500 to 503 in the combined image shown in FIG.
Each of the plurality of motion vectors 700 calculated in the subject group area 500 corresponding to the subject group A indicates the amount of movement in length and the direction of movement in the direction of the arrow. Similarly, a plurality of motion vectors 701 to 703 are shown for the other subject groups B to D.

S1503
The conversion coefficient calculation unit 211 first selects the first subject group (i = 0th). For the sake of explanation, it is assumed that numbers are assigned in alphabetical order (or in order of distance) of the group A, such as the 0th group A and the first group B.

S1504
The conversion coefficient calculation unit 211 determines whether the number of motion vectors calculated for the area of the i-th subject group is greater than or equal to a first predetermined value. If so, the process proceeds to S1508. Here, the first predetermined value is a number necessary for calculating the geometric transformation coefficient for the subject group region, and is 4 when the geometric transformation is performed by projective transformation. When other methods such as affine transformation are used, the number depends on the method.

ここで（ｘ０，ｙ０）は動きベクトルの始点の座標であり、（ｘ，ｙ）は動きベクトルの終点の座標、ａ〜ｉは射影変換係数である。なお、本実施形態では光学系２０４の光軸と撮像部２０５との交点を原点（０，０）とした直交座標系とするが、他の予め定めた位置（例えば左上隅）を原点としてもよい。 S1505
The conversion coefficient calculation unit 211 calculates a geometric conversion coefficient for the subject group region using the motion vector calculated for the i-th subject group region.
A calculation method when the geometric transformation coefficient is a projective transformation coefficient will be described. Projective transformation can be expressed by Equation 1 below.

Here, (x0, y0) is the coordinates of the start point of the motion vector, (x, y) is the coordinates of the end point of the motion vector, and a to i are projective transformation coefficients. In the present embodiment, an orthogonal coordinate system having the origin (0, 0) as the intersection point of the optical axis of the optical system 204 and the imaging unit 205 is used, but another predetermined position (for example, the upper left corner) may be used as the origin. Good.

変換係数算出部２１１は、ｉ番目の被写体グループ領域について算出された個々の動きベクトルについて、始点及び終点の座標（ｘ，ｙ）、（ｘ０，ｙ０）をそれぞれ式（２）、式（３）に代入する。変換係数算出部２１１は、これにより得られる９変数の方程式を解くことで幾何変換係数を算出し、例えばＲＡＭ２０３に保存する。 Equation (1) can be transformed into Equation (2) and Equation (3) below.

The conversion coefficient calculation unit 211 sets the coordinates (x, y) and (x0, y0) of the start point and the end point of the individual motion vectors calculated for the i-th subject group region, respectively, using the equations (2) and (3). Assign to. The conversion coefficient calculation unit 211 calculates a geometric conversion coefficient by solving an equation of nine variables obtained thereby, and stores it in the RAM 203, for example.

S1506
The conversion coefficient determination unit 212 performs coordinate conversion of the reference point using the geometric conversion coefficient calculated by the conversion coefficient calculation unit 211 in S1505, and creates a motion vector having the coordinates before conversion as the start point and the coordinates after conversion as the end point. To do. This motion vector is called a representative motion vector of the i-th subject group region. In the present embodiment, the reference point is the coordinate origin (0, 0). 8A to 8D show representative motion vectors 800 to 803 obtained for the subject group regions 500 to 503 (subject groups A to D).

S1507
The conversion coefficient determination unit 212 normalizes the representative motion vector of the i-th subject group region based on the distance information of the specific subject group, and generates a normalized representative motion vector. By normalizing with specific distance information, the size of the normalized representative motion vector of a subject group composed only of still subjects becomes equivalent regardless of the subject distance.

A method for generating a normalized representative motion vector will be described.
First, the conversion coefficient determination unit 212 determines the distance in the optical axis direction for normalization. In this embodiment, the distance of the subject group A from the digital camera 200 (representative distance of the subject group A) is set as a normalized distance.
When the normalized distance is DA (FIG. 6B) and the representative distance of the subject group to be normalized is D, the size of the representative motion vector of the subject group to be normalized is expressed by the following equation (4). Is normalized.
｜ Normalized representative motion vector | = | Representative motion vector | × D / DA Expression (4)

  FIGS. 8E to 8H show normalized representative motion vectors 804 to 807 of the representative motion vectors 800 to 803 of the subject groups shown in FIGS. 8A to 8D. Here, since normalization is performed by the distance of the subject group A, the normalized representative motion vector 804 of the subject group A shown in FIG. 8E is equal to the representative motion vector 800 before normalization. In FIGS. 8E to 8H, reference numerals 808 to 811 denote angles (directions of movement) formed by the normalized region motion vectors 804 to 807 with the x axis.

  The conversion coefficient determination unit 212 stores the calculated normalized representative motion vector information in association with the subject group number (i), for example, in the RAM 203, and advances the process to S1509.

S1508
In S1504, when it is determined that the number of calculated motion vectors is less than the first predetermined value, the conversion coefficient calculation unit 211 uses, for example, a subject group number i as information for specifying a subject group having no conversion coefficient. Are stored in the RAM 203, and the process advances to step S1509.

S1509
The system control unit 201 determines whether the normalized representative motion vector is calculated or the group number is stored for all subject group regions. If there is an unprocessed subject group region, the process proceeds to step S1510, and there is no unprocessed subject group region. If so, the process advances to S1511.

S1510
The system control unit 201 increments the group number i and returns the process to S1504.

S1511
The transform coefficient determination unit 212 generates an estimated global motion vector related to the change amount and change direction of the shooting position of the reference image and the combined image from the calculated normalized representative motion vector. Here, only the subject group region for which the geometric transformation coefficient is calculated is set as the processing target.
First, the transform coefficient determination unit 212 creates a histogram regarding the magnitude and direction of the normalized representative motion vector. Here, the direction is expressed by angles 808 to 811 (FIGS. 8E to 8H) formed by the x-axis and the normalized representative motion vector.

  FIGS. 9A and 9B show examples of normalized and representative motion vector magnitude and angle histograms. Then, the transform coefficient determination unit 212 sets the vector having the highest frequency 900 and the direction 901 as the estimated global motion vector. The estimated global motion vector is shown in FIG. The estimated global motion vector 1000 has a mode value 900 and an angle 901 of the normalized representative motion vector.

S1512
Next, the conversion coefficient determination unit 212 determines whether the movement is a global movement (stationary subject) or an individual movement (moving subject) for each subject group region. Again, only the subject group region for which the geometric transformation coefficient has been calculated is set as the processing target.

Details of this determination processing will be described with reference to the flowchart of FIG.
S1600
The conversion coefficient determination unit 212 selects a normalized representative motion vector of the first subject group region (i = 0).

S1601
The transform coefficient determination unit 212 calculates a difference absolute value between the size of the normalized representative motion vector and the size of the estimated global motion vector, and determines whether the difference absolute value is less than a second predetermined value. The conversion coefficient determination unit 212 advances the process to S1602 if the absolute difference value is less than the second predetermined value, and advances to S1604 if it is greater than or equal to the second predetermined value. For example, the second predetermined value is 1 [pixel].

S1602
The transform coefficient determination unit 212 calculates a difference absolute value between the direction (angle) of the normalized representative motion vector and the direction (angle) of the estimated global motion vector, and determines whether the difference absolute value is less than a third predetermined value. . The conversion coefficient determination unit 212 advances the process to S1603 if the difference absolute value is less than the third predetermined value, and advances to S1604 if the difference absolute value is greater than or equal to the second predetermined value. For example, the third predetermined value is 1 [degree].

S1603
The conversion coefficient determination unit 212 determines that the subject group area to be processed is moving globally (still subject), and advances the processing to step S1605.

S1604
The conversion coefficient determination unit 212 determines that the subject group area to be processed does not move globally (moving subject), and advances the process to step S1605.

S1605
The system control unit 201 determines whether determination processing has been performed for all subject group areas (geometric transformation coefficients have been calculated). If there is an unprocessed subject group region, the process proceeds to S1606. If there is no unprocessed subject group area, the system control unit 201 stores information representing the determination result for each subject group area in, for example, the RAM 203, ends the determination process, and advances the process to S1513 (FIG. 4B).

S1606
The system control unit 201 increments the group number i and returns the process to S1601.

S1513
In step S1512, the system control unit 201 determines whether or not to perform re-determination for the subject group area that is determined not to be a global movement (moving subject) by increasing the grouping accuracy.

  There is a high possibility that the subject group region including both the stationary subject region and the moving subject region is determined not to be a global movement in the determination processing in S1512. For this reason, when it is considered that the subject group area determined not to be a global movement (moving subject) includes a plurality of subject areas, re-determination processing is performed by regrouping into individual subject areas.

  When the histogram of the distance information corresponding to the subject group region that is determined not to be a global movement (moving subject) has a high frequency at discrete distances, the system control unit 201 has a plurality of subject regions. Is determined to be included, and it is determined that re-determination processing is performed. More specifically, the system control unit 201 determines that a plurality of subject areas are included when the histogram of the distance information of the subject group area has a plurality of frequency maximum values. Note that this determination method is an example, and for example, it may be determined whether or not a plurality of subject regions are included based on the distribution (coordinates) in the image of pixels corresponding to the distance information of the subject group region.

  If there is a subject group area determined to be redetermined, the system control unit 201 advances the process to step S1514. If the subject group area determined to be redetermined does not exist or if all subject group areas are determined to be global movements, the system control unit 201 advances the process to step S1515.

S1514
The system control unit 201 performs redetermination processing by the conversion coefficient calculation unit 211 and the conversion coefficient determination unit 212. Details of the redetermination process will be described with reference to the flowchart of FIG.

S2300
The conversion coefficient calculation unit 211 regroups the subject group areas determined to be redetermined into a plurality of groups.
For example, the conversion coefficient calculation unit 211 generates a plurality of subject group areas having discrete distances as representative distances in the corresponding distance information histogram. Alternatively, if the subject group area to be processed is a group of a plurality of subject groups in S1501, the subject group detection result in S1501 may be used.

For example, the distance information histogram can be regrouped by dividing the histogram into a plurality of distance ranges with the distance corresponding to the maximum value having a frequency equal to or higher than a predetermined threshold as a representative distance.
7 to 8, the normalized representative motion vector of the subject group C (subject group region 502) is greatly different in magnitude and direction from the estimated global vector. Therefore, in the determination process of S1512, it is determined that the subject group area 502 is not a global movement.

  The histogram of distance information corresponding to the subject group area 502 (FIG. 6B) includes a portion corresponding to the moving subject a 104 and a portion corresponding to the stationary subject C102. Therefore, the conversion coefficient calculation unit 211 regroups the histogram of distance information corresponding to the subject group area 502 into subject groups CA and CB corresponding to the respective subjects.

The result of regrouping is shown in FIG. The moving subject a 104 is regrouped as a subject group CA, and the stationary subject C 102 is regrouped as a subject group CB. As a result, the subject group area 502 is divided into two subject group areas 1100 and 1101.
Thereafter, the conversion coefficient calculation unit 211 and the conversion coefficient determination unit 212 perform the same processing as S1503 to S1510 in FIG. 4A and S1512 in FIG. 4B for each subject group region, and the process proceeds to S1515. Since the estimated global vector has already been calculated, the re-determination process does not perform the process corresponding to S1511.

  12B and 12C show the representative motion vectors 1200 and 1201 of the subject group areas 1100 and 1101 (subject groups CA and CB), and FIGS. 12D and 12E show the normalized representative motion vectors 1202 and 1203, respectively. Respectively. Reference numerals 1203 and 1204 denote angles formed by the normalized region motion vectors 1202 and 1203 with the x axis.

S1515
The system control unit 201 determines whether there is a subject group area for which a geometric transformation coefficient has not been calculated. If there is, the process proceeds to S1516, and if not, the process proceeds to S1518.

S1516
The system control unit 201 uses the conversion coefficient calculation unit 211 to generate a geometric conversion coefficient for a subject group region for which no geometric conversion coefficient has been calculated. In this embodiment, the conversion coefficient calculation unit 211 generates a geometric conversion coefficient for a subject group area for which no geometric conversion coefficient has been calculated based on the relationship between the distances of the subject group areas from the calculated geometric conversion coefficient.

  First, the conversion coefficient calculation unit 211 selects two or more subject group areas for which a geometric conversion coefficient has been calculated that have been determined to be global motion. Here, it is assumed that two are selected, the subject group region 1 and the subject group region 2 are represented in order from the closest representative distance from the digital camera 200, and the respective geometric transformation coefficients are Pi and Pj.

FIG. 13A schematically shows an example of the relationship between the distance relationship between the subject group region, the subject group region 1 and the subject group region 2 for which the geometric transformation coefficient is not calculated, and the geometric transformation coefficient. The conversion coefficient calculation unit 211 acquires geometric conversion coefficients Pi 1301 and Pj 1302.
Next, the conversion coefficient calculation unit 211 calculates a difference Δdist 1303 of the representative distance between the subject group area 1 and the subject group area 2 and a difference Δparam 1304 of the corresponding geometric conversion coefficient.

The conversion coefficient calculation unit 211 calculates a difference dist 1305 between the representative distance of the subject group area for which the geometric conversion coefficient could not be calculated and the representative distance of the selected subject group area closest to the digital camera 200.
Then, the conversion coefficient calculation unit 211 calculates the geometric conversion coefficient P 1300 of the subject group area for which the geometric conversion coefficient has not been calculated, using the following equation (5).
P = Pi + dist × Δparam / Δdist (5)
Note that the calculation using the equation (5) is performed for each of the geometric transformation coefficients a to i shown in the equation (1).

  Here, the case where two subject group regions with calculated geometric transformation coefficients that are closer to the digital camera 200 than the subject group region for which the geometric transformation coefficients could not be calculated has been illustrated. However, the calculation can be performed in the same manner even when one closer and farther than the subject group region for which the geometric transformation coefficient could not be calculated are selected one by one, or when two far away are selected.

S1517
The conversion coefficient determination unit 212 determines whether the subject group area for which the geometric conversion coefficient is calculated in S1516 is a global motion. The basic determination method is the same as the motion determination process described with reference to FIG. 10, but the vectors for comparing the magnitude and direction (angle) are different.

  A specific method will be described with reference to FIG. FIG. 13B shows the subject group region 1400 for which the geometric transformation coefficient has been calculated in S1516, and the calculated motion vector 1402. Although only one motion vector 1402 is shown here, other motion vectors may be calculated.

  The transformation coefficient determination unit 212 uses the geometric transformation coefficient calculated in S1516 using the start point 1401 of one calculated motion vector (here, motion vector 1402) as a reference point, and performs the motion vector 1405 in the same manner as in S1506. Is calculated. Here, it is assumed that the motion vector 1405 of the start point 1401 and the end point 1404 is calculated by the geometric transformation coefficient.

  Next, the transform coefficient determination unit 212 determines the difference between the magnitude and direction (angle) of the motion vector 1402 and the motion vector 1405 as in S1601 to S1604. Specifically, the transform coefficient determination unit 212 determines that the subject group area is the absolute value of the magnitude difference is less than the fourth predetermined value and the absolute value of the direction (angle) is less than the fifth predetermined value. It is determined that 1400 is moving globally (stationary subject). On the other hand, if any of the conditions is not satisfied, the conversion coefficient determination unit 212 determines that the subject group area 1400 is moving individually (moving subject). For example, the fourth predetermined value is 1 [pixel], and the fifth predetermined value is 1 [degree].

  When the determination process is performed for all the subject group areas for which the geometric transformation coefficients have been calculated in S1516, the system control unit 201 stores information representing the determination result for each subject group area in, for example, the RAM 203, and the process proceeds to S1518. .

S1518
The system control unit 201 applies a geometric transformation process to the synthesized image by the geometric transformation unit 213 to generate an image aligned with the reference image. The geometric transformation process may be any of the following methods.
(First method)
A geometric transformation process is applied to each subject group region of the composite image using the geometric transformation coefficient calculated for each.
(Second method)
One geometric transformation coefficient is calculated from a motion vector obtained for a subject group region of the composite image that is determined to have a global motion, and is applied to the entire composite image. In this case, the geometric transformation coefficient can be obtained by the transformation coefficient calculation unit 211 as in S1505.

Since the first method uses a geometric transformation coefficient for each subject area, the alignment accuracy is higher than that of the second method. However, since it is necessary to switch the geometric transformation coefficient for each subject area and perform transformation, the processing load and processing It is disadvantageous over the second method in terms of time.
Since the second method requires only one conversion process, the second method is more advantageous than the first method in terms of processing load and processing time, but the accuracy of alignment is lower than that of the first method. In particular, there is a possibility that the accuracy of positioning of the moving subject will be greatly reduced.

  Which of these methods is used may be determined as appropriate according to the performance of the image processing apparatus, available resources, processing load, and the like, and may be switched dynamically.

  In this embodiment, the overall motion between images is estimated using a motion vector normalized using distance information, and each subject region is moving globally (a stationary subject) or moving globally. It is determined whether it is present (moving subject). Therefore, the difference in the amount of movement in the image according to the subject distance and the influence of the moving subject in the image can be suppressed, and it can be accurately determined whether or not the subject is a stationary subject, and the global motion vector estimation accuracy is improved. Therefore, the alignment accuracy can be improved.

● (Second Embodiment)
Next, a second embodiment of the present invention will be described. The present embodiment relates to an alignment process in a scene in which there are still subjects having a large depth straddling a plurality of subject groups in the first embodiment, and there are still subjects and moving subjects at the same distance. Since the contents other than the contents of the alignment process may be the same as those in the first embodiment, this embodiment will also be described using the configuration of the digital camera 200.

  FIG. 14 is a diagram schematically showing an example of a shooting scene in the present embodiment. A moving subject b 1700 and a moving subject c 1701 are added to the scene shown in FIG. The moving subject b 1700 has a depth that spans the distance range of the stationary subjects A 100 and B 101, and the moving subject c 1701 exists at the same distance as the stationary subject D 103. The background portion may also be handled as one or more subjects, but will be omitted in the following description.

  Next, the alignment process in this embodiment is demonstrated using the flowchart of FIG. In FIG. 15, the same reference numerals as those in FIG. 4A and FIG.

S2400
The conversion coefficient calculation unit 211 groups subjects in the combined image based on the distance information. In this embodiment, two-stage grouping is performed. The grouping performed in this step is basically the same as the grouping in S1501.
FIG. 16 schematically shows the distance from the digital camera 200 of the subject included in the shooting scene of FIG. 14 and the frequency distribution of the distance information of the combined image, as in FIG.
The conversion coefficient calculation unit 211 uses a group of distance information including a frequency that is a maximum in the frequency distribution (histogram) of the distance information of the combined image as a subject group, Group.

  The subject group C is the same as in the first embodiment. The subject group D newly includes a moving subject c 1701. On the other hand, the addition of the moving subject b 1700 having a large depth makes the separation of the distance information corresponding to the stationary subjects A 100 and B 101 unclear, and the subject groups A and B use the minimum value of a series of frequency distributions as a boundary. Grouped.

S2401
Next, the transform coefficient calculation unit 211 performs second-stage grouping that is characteristic of the present embodiment. Specifically, the conversion coefficient calculation unit 211 performs grouping based on regions in the image corresponding to each subject group.

  For example, the conversion coefficient calculation unit 211 (grouping unit) specifies pixel coordinates from distance information corresponding to each subject group, and extracts a closed region corresponding to the subject group. For extraction of the closed region, for example, a known method such as a dynamic contour model (Snake method) can be used. Of the extracted closed regions, the adjacent regions or the regions whose intervals are within the threshold are collectively set as one subject group.

  FIG. 17 schematically shows the result of grouping in S2401. Each of the subject groups A to D grouped in S2400 is further grouped into two subject groups in S2401. Specifically, the subject group A is regrouped into a subject group AA corresponding to the stationary subject A 100 and a subject group AB corresponding to a part on the short distance side of the moving subject b 1700. Reference numerals 2000 and 2001 denote corresponding subject group areas.

  Similarly, the subject group B is regrouped into a subject group BA corresponding to the stationary subject B 101 and a subject group BB corresponding to a part of the moving subject b 1700 on the far side. Reference numerals 2002 and 2003 denote corresponding subject group areas.

The subject group C is regrouped into a subject group CA corresponding to the moving subject a 104 and a subject group CB corresponding to the stationary subject C 101. Reference numerals 2004 and 2005 denote corresponding subject group areas.
The subject group D is regrouped into a subject group DA corresponding to the stationary subject D 103 and a subject group DB corresponding to the moving subject c 1701. Reference numerals 2006 and 2007 denote corresponding subject group areas.

  Next, the transform coefficient determination unit 212 calculates normalized motion vectors for the subject groups AA to DB in the same manner as in the first embodiment.

S2402
The conversion coefficient determination unit 212 first selects the first subject group (i = 0th). For the sake of explanation, it is assumed that numbers are assigned in alphabetical order (or in order of distance) of the group AA as 0th and the group AB as 1st.

S2403
The conversion coefficient determination unit 212 performs coordinate conversion of the reference point using the geometric conversion coefficient calculated by the conversion coefficient calculation unit 211 in S1505, and creates a motion vector having the coordinates before conversion as the start point and the coordinates after conversion as the end point. To do. The content of the process is the same as S1506. This motion vector is called a representative motion vector of the i-th subject group region. In the present embodiment, the reference point is the coordinate origin (0, 0). 18A to 18H show representative motion vectors 2100 to 2107 calculated for the areas of the subject groups AA to DB.

S2404
The conversion coefficient determination unit 212 normalizes the representative motion vector of the i-th subject group region based on the distance information of the specific subject group, and generates a normalized representative motion vector. The content of the process is the same as S1507. In this embodiment, the distance of the subject group AA from the digital camera 200 (representative distance of the subject group AA) is set as the normalized distance DA, which is the same as S1507.

  FIGS. 19A to 19H show normalized representative motion vectors 2200 to 2207 of the representative motion vectors 2100 to 2107 of the subject groups shown in FIGS. 18A to 18H. Here, since normalization is performed by the distance of the subject group AA, the normalized representative motion vector 2200 of the subject group AA shown in FIG. 18A is equal to the representative motion vector 2100 before normalization. Note that 2208 to 2215 in FIGS. 19A to 19H are angles (directions of motion) formed by the normalized region motion vectors 2200 to 2207 with the x axis.

S2405
The transform coefficient determination unit 212 generates an estimated global motion vector related to the change amount and change direction of the shooting position of the reference image and the combined image from the calculated normalized representative motion vector. Here, only the subject group region for which the geometric transformation coefficient is calculated is set as the processing target. Since the content of the process is the same as S1511, detailed description is omitted.
Since the determination processing in S1512 and subsequent steps are the same as those in the first embodiment, description thereof will be omitted.

  As described above, according to the present embodiment, the subjects are grouped based on the frequency distribution of distance information, and then each group is further grouped based on the corresponding image region. Therefore, even when there is a subject with a large depth, or even in a scene where a stationary subject and a moving subject exist at the same distance, it is possible to accurately determine whether or not the subject is a stationary subject, improving the accuracy of global motion vector estimation. To do. Therefore, the alignment accuracy can be improved.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 200 ... Digital camera, 201 ... System control part, 204 ... Optical system, 205 ... Imaging part, 206 ... Conversion part, 207 ... Image processing part, 208 ... Recording medium, 209 ... Bus, 210 ... Distance information calculation part, 211 ... Conversion coefficient calculation unit 212, conversion coefficient determination unit, 213, geometric conversion unit

Claims (15)

Grouping means for grouping image regions using distance information;
Detecting means for detecting movement relative to other images for each group;
Normalizing means for normalizing the magnitude of the movement based on a specific distance;
Determining means for determining, for each of the groups, a region of a stationary subject or a region of a moving subject based on the movement whose size has been normalized;
An image processing apparatus comprising: The detecting means;
First calculation means for calculating a conversion coefficient for aligning with the other image for each group;
Generating means for generating a representative motion vector representing the motion for each of the groups using the transform coefficient;
Have
The normalizing means normalizes the magnitude of the motion by normalizing the magnitude of the representative motion vector by a ratio of the specific distance to a distance corresponding to a group.
The image processing apparatus according to claim 1. The detecting means detects a motion vector representing the motion for each group;
The first calculation means includes
For the group in which the number of detected motion vectors is less than a predetermined value determined in advance, the conversion coefficient is not calculated,
For the group in which the number of detected motion vectors is equal to or greater than a predetermined value, the conversion coefficient is calculated based on the motion vectors.
The image processing apparatus according to claim 2.   Furthermore, it has the 2nd calculation means which calculates using the said conversion coefficient about the other group calculated by the said 1st calculation means about the group in which the said conversion coefficient was not calculated by the said 1st calculation means. The image processing apparatus according to claim 3. Transform means for applying the transform coefficient for each group and generating an image aligned with the other image;
Synthesizing means for synthesizing the image aligned with the other image with the other image;
The image processing apparatus according to claim 2, further comprising: Based on the motion vector detected by the detection means for the group determined as the area of the stationary subject by the determination means, a third conversion coefficient is calculated for aligning the entire image with the other image. A calculation means;
Transform means for applying the transform coefficient to the entire image and generating an image aligned with the other image;
Synthesizing means for synthesizing the image aligned with the other image with the other image;
The image processing apparatus according to claim 2, further comprising: The determination means is
Estimating means for estimating the overall motion of the image relative to the other image based on the normalized motion of the group for each group;
A group in which the difference between the motion with the normalized size and the overall motion is less than a predetermined value is determined as a still subject region, and the motion with the normalized size and the overall motion are A group in which the difference between the two is greater than or equal to a predetermined value is determined as a moving subject area;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The image processing according to claim 7, wherein the estimation unit estimates an overall motion of the image based on a motion magnitude and a direction frequency in which the size is normalized for each group. apparatus.   The image processing apparatus according to claim 1, wherein the grouping unit performs the grouping so that regions included in the same distance range belong to the same group.   The image processing apparatus according to claim 1, wherein the grouping unit performs the grouping based on a frequency distribution of the distance information.   The image processing apparatus according to claim 10, wherein the grouping unit further groups the grouped areas for each closed area.   The image processing apparatus according to claim 1, wherein the distance information is information representing a distance for each pixel of the image. Imaging means capable of acquiring parallax images;
Fourth calculation means for generating the distance information from the parallax image;
Image processing means for generating the image from the parallax image;
The image processing apparatus according to any one of claims 1 to 12,
An imaging device comprising: A grouping means for grouping image regions using distance information;
A detecting step for detecting movement of the other image for each of the groups;
A normalizing step in which normalizing means normalizes the magnitude of the movement based on a specific distance;
A determination step of determining, for each of the groups, a region of a stationary subject or a region of a moving subject based on the movement whose size has been normalized;
A control method for an image processing apparatus, comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 12.

Images