JP4693709B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, such as a digital camera, a digital video camera, a mobile phone with a camera, and a personal computer, which detect a positional shift of these images from a plurality of captured images.

  With the current camera, all the important tasks for shooting such as determining exposure and focusing are automated, and it is very unlikely that people who are unskilled in camera operation will fail to shoot. Recently, a system for preventing camera shake applied to the camera has been studied, and the cause of the photographer's shooting mistake has almost disappeared. As a method for preventing camera shake, a correction lens is used to detect vibration due to camera shake using a sensor that detects acceleration, angular acceleration, angular velocity, angular displacement, etc., and correct the change in the optical axis due to camera shake according to the detected value. There is an optical shift type image stabilization to be displaced. In addition, by repeating shooting with a short exposure time that does not cause camera shake, and combining the positions of a plurality of shot images, an image that compensates for the lack of exposure while correcting the difference in composition for each image Synthetic image stabilization is also known.

  For example, Patent Document 1 discloses an alignment method in image synthesis type camera shake correction. In this method, a motion vector is calculated using a peripheral area (background) of the frame that avoids an area where the main subject is likely to be located. This is because, when a person is assumed as the main subject, there is a possibility of performing alignment that is affected by subject shake due to blinking of eyes or complicated movements of hands.

  A color image processing method is described in Patent Document 2, for example. In this method, alignment calculation is performed for each color component, particularly for color components (G component) that have a large influence on the resolution in terms of human visual characteristics, alignment calculation is performed with high accuracy, and other color components (R , B component), the alignment calculation process is simplified.

JP 2004-219765 A JP 2002-112007 A

  However, in the image composition-type image stabilization, since the composition is performed by aligning a plurality of images with slightly different compositions, the composition processing is not appropriately performed when the alignment accuracy is poor, and the composite image Will be blurred. In particular, when a motion vector calculation operation is performed on a region with a large amount of blur in an image signal, it is difficult to perform alignment accurately, and thus image synthesis may not be performed appropriately. For this reason, in an image signal in which the amount of blur is greatly different between the main subject and the background, a high-precision motion vector cannot be obtained due to the influence of the motion vector calculation result of the background region, and the main subject is blurred. was there.

  For example, in Patent Document 1, although the intended purpose can be achieved, since the motion vector is calculated using the frame peripheral region, the position with low accuracy is required for an image signal with a large amount of background blur. There is a problem that only the combined result can be obtained. Also, in Patent Document 2 described above, since the accuracy of the alignment calculation is changed using only the color component of the image signal, for example, the same problem is caused for a scene with a lot of blue such as a blue sky in the background. It will occur.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an image processing method capable of detecting a positional deviation with high accuracy.

  As a result of intensive studies to solve the above problems, the present inventor has come up with various aspects of the invention described below.

First image processing apparatus according to the present invention, first with respect to each of the plurality of captured images to identify the focus detection area focused at the time of shooting, before Symbol within a predetermined range from the focus detection area region and the region determination unit distance to the Utsushitai is to divided into at least two regions of the farthest second region, and the displacement detection means for detecting a positional deviation between before Symbol plurality of captured images, the And when the focus detection areas match between the plurality of captured images, the positional deviation detection means assigns a weight to the first area that is greater than or equal to a weight set for the second area. The positional deviation is detected after setting.
The second image processing apparatus according to the present invention specifies a focus detection area focused at the time of shooting for each of a plurality of captured images, and the first image processing apparatus is within a predetermined range from the focus detection area. A region determination unit that divides the region into at least two regions, the second region having the longest distance to the subject, and a displacement detection unit that detects a displacement between the plurality of captured images, If the focus detection area does not match among a plurality of photographed images, the positional deviation detection means assigns a weight that is greater than or equal to the weight set for the first area to the second area. The positional deviation is detected after setting for.

The first image processing method according to the present invention, first with respect to each of the plurality of captured images to identify the focus detection area focused at the time of shooting, before Symbol within a predetermined range from the focus detection area area and the area determination step for dividing into at least two regions of the distance to the Utsushitai is farthest second region, and the positional deviation detection step of detecting a positional deviation between before Symbol plurality of captured images, the And when the focus detection areas match between the plurality of captured images, the first area is given a weight greater than or equal to a weight set for the second area in the misalignment detection step. The positional deviation is detected after setting.
In the second image processing method according to the present invention, a focus detection area focused at the time of shooting is specified for each of a plurality of captured images, and the first image processing method is within a predetermined range from the focus detection area. A region determination step for dividing the region into at least two regions, the second region having the longest distance to the subject, and a displacement detection step for detecting displacement between the plurality of captured images, In the case where the focus detection area does not match between a plurality of captured images, in the misalignment detection step, the second area is weighted more than the weight set for the first area. The positional deviation is detected after setting for.

  According to the present invention, since the weighting amount to be given to the first and second regions is adjusted according to the result of the region determination division reflecting the distance measurement result, an appropriate positional deviation amount is set. Can be detected. Therefore, by performing image composition type camera shake correction using this positional deviation amount, it is possible to prevent problems such as out-of-focus and blurring of the main subject.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the accompanying drawings.

(First embodiment)
First, a first embodiment of the present invention will be described with reference to FIGS. Here, a digital single-lens reflex camera having seven focus detection areas (hereinafter referred to as ranging points) by the phase difference detection type automatic focus adjustment function will be described as an example. The phase difference detection type automatic focusing function is known.

  First, the configuration of the imaging device in the present embodiment will be described. FIG. 1 is a side sectional view showing a schematic configuration of a camera according to the first embodiment of the present invention.

  The camera according to this embodiment is a single-plate digital color camera using an image sensor such as a CCD or CMOS sensor, and obtains an image signal representing a still image by driving the image sensor continuously or once. Here, the image sensor is an area sensor of a type that converts received light into an electrical signal for each pixel, accumulates charges according to the amount of light, and reads the accumulated charges.

  As shown in FIG. 1, a plurality of lens devices 102 having different focal lengths can be attached to and detached from the camera body 101, and shooting screens with various angles of view can be obtained by replacing the lens devices 102.

  The lens device 102 has a drive mechanism (not shown), and this drive mechanism adjusts the focus by moving a focusing lens, which is a part of the imaging optical system 104, in the direction of the optical axis L1.

  Further, in the lens device 102, there are disposed a diaphragm for adjusting the light quantity of the photographing light flux by changing the opening area of the light passage opening, and a drive mechanism (all not shown) for driving the diaphragm.

  In the camera body 101, a main mirror 105, a sub mirror 106, a focal plane shutter 107, an optical low-pass filter 108, and an image sensor 108 are disposed along the optical axis L1.

  The main mirror 105 is composed of, for example, a half mirror. The main mirror 105 transmits a part of the light beam from the lens device 102 and guides it to an AF unit 114 (described later) via the sub mirror 106 and reflects the remaining light beam to a finder optical system 103 (described later). . The main mirror 105 and the sub mirror 106 are movable, and can be switched between a state of being obliquely provided on the photographing optical path (on the optical axis L1) and a state of being retracted from the photographing optical path. As described above, the photographing light beam is divided into the viewfinder optical system 103 and the AF unit 114 in a state where the photographing light path is obliquely provided. However, in a state where the photographing light beam is retracted from the photographing optical path, the photographing light beam is directly transmitted to the imaging element 109 described later. It comes to lead.

  A focal plane shutter (hereinafter referred to as a shutter) 107 has a front curtain and a rear curtain composed of a plurality of light shielding blades. The shutter 107 shields the photographic light flux by covering the aperture serving as a light passage opening with the front curtain or the rear curtain when not photographing, and images the photographic light flux by traveling while the front curtain and the rear curtain form a slit during photographing. Let it pass to the surface side.

  The optical low-pass filter 108 limits the high-frequency component of the incident light beam so that a spatial frequency component higher than necessary for the subject image is not transmitted to the image sensor 109 described later.

  The image sensor 109 is housed in, for example, a package, and includes, for example, a CMOS process compatible sensor (CMOS sensor) that is one of amplification type solid-state image sensors. The CMOS sensor has a feature that random access to an arbitrary pixel is possible, readout that is thinned for display display is easy, and real-time display can be performed at a high display rate. The subject image captured by the image sensor 109 is displayed on the display unit 115 through predetermined image processing. The display unit 115 is attached to the back surface of the camera body 101 so that a photographer can directly observe an image displayed on the display unit 115. As the display unit 115, for example, a liquid crystal panel is provided.

  The AF unit 114 is an AF unit having seven distance measuring points, and detects a focus adjustment state by a phase difference detection method.

  The viewfinder optical system 103 is provided with a focusing screen 110, a pentaprism 111, an AE unit 112, and an eyepiece lens 113.

  The focusing screen 110 is disposed, for example, on the planned image plane of the subject image. The pentaprism 111 reflects the light beam from the main mirror 105 a plurality of times (converts it into an erect image) and guides it to the eyepiece lens 113. The eyepiece 113 is a lens for observing a subject image formed on the focusing screen 110, and is composed of, for example, about three lenses. The AE unit 112 is a sensor for performing automatic photometry.

  Furthermore, a release button 116 for taking a still image that can be pressed in two stages is provided. When the release button 116 is half pressed, a shooting preparation operation (focus adjustment operation, photometry operation, etc.) is started, and when the release button 116 is fully pressed, a shooting operation is started. In the following description, the half-pressed state of the release button 116 is expressed as a state where the switch SW1 is turned on, and the full-pressed state of the release button 116 is expressed as a state where the switch SW2 is turned on.

  Next, a functional configuration of the camera (imaging device) according to the present embodiment will be described. FIG. 2 is a block diagram illustrating a functional configuration of the camera according to the first embodiment.

  As shown in FIG. 2, the light beam incident from the photographing lens 201 (lens device 102) passes through the stop 202 and reaches the mirror box 203. The mirror box 203 includes a main mirror 105 and a sub mirror 106. By driving the mirror box 203, the incident light is divided into transmitted light and reflected light, and each is guided to the AF unit 114 and the AE unit 112, and the incident light is directly guided to the imaging unit 209 (imaging device 109). You can switch to the state. As described above, the imaging unit 209 is configured by a CMOS sensor, for example.

  The photographing lens 201 is driven by a lens driving unit 207 for focus adjustment, the diaphragm 202 is driven by the diaphragm driving unit 205, the mirror box is driven by the mirror driving unit 204, and the shutter 107 is driven by the shutter driving unit 206.

  The imaging control unit 208 determines the aperture value and shutter speed at the time of exposure based on the photometric information from the AE unit 112, and controls the shutter drive unit (SH drive unit) 206 and the aperture drive unit 205. Further, the imaging control unit 208 determines a lens driving amount necessary for focusing based on a signal from the AF unit 114 and controls the lens driving unit 207.

  At the time of exposure, the main mirror 105 and the sub mirror 106 are retracted from the photographing optical path by the mirror driving unit 204, the shutter 107 is driven by the shutter driving unit 206, and exposure is performed for a predetermined exposure time. As a result, the light flux from the photographing lens 201 forms an image on the imaging unit 209.

  The video signal output from the imaging unit 209 is converted into a digital signal by the A / D conversion unit 210 and input to the signal processing unit 211. The signal processing unit 211 performs signal processing such as generation of a luminance signal and / or a color signal, and as a result, a color video signal is generated.

  The display unit 215 and the recording unit 216 display, record, and save the captured image, and display, record, and save the color video signal generated by the signal processing unit 211.

  In this embodiment, a mode switch (not shown) is provided, and the photographer can arbitrarily set the normal shooting mode and the image composition mode. The above-described units operate regardless of which operation mode is selected, but the image detection mode is selected for the following shift detection unit 212, coordinate conversion unit 213, image storage unit 214, and image composition unit 217. Works if. Note that the normal shooting mode is an operation mode that is selected when shooting a subject whose brightness does not require image stabilization. The image composition mode is, for example, a dark subject and a long exposure time. Therefore, the operation mode is selected when there is a risk of camera shake. When the image composition mode is selected, the shooting control unit 208 determines the number of shots based on information from the AE sensor 112, and controls shooting for the number of shots. In addition, the shooting control unit 208 passes the number of shots and information on the distance measuring points selected at the time of shooting each image to the shift detection unit 212.

  The deviation detection unit 212 performs a two-dimensional correlation calculation based on the plurality of image signals received from the signal processing unit 211 and the information input from the imaging control unit 208, and the relative positional deviation amount between the images ( Motion vector) is calculated. The coordinate conversion unit 213 performs coordinate conversion of each image in accordance with the motion vector calculated by the deviation detection unit 212. Then, the image storage unit 214 stores the images after coordinate conversion, and the stored images are combined into one image by the image combining unit 217. At this time, the combined image is displayed and stored on the display unit 215 and the recording unit 216.

  Next, the operation of the camera according to the first embodiment configured as described above will be described. 4A to 4C are flowcharts illustrating the operation of the camera according to the first embodiment.

  First, as shown in FIG. 4A, when the camera is turned on, the imaging control unit 208 initializes various settings in step S401, and proceeds to step S402.

  In step S <b> 402, the shooting control unit 208 determines whether the shooting mode of the camera is set to an image synthesis mode for combining a plurality of images after shooting or a normal shooting mode. If the image composition mode is set, the process proceeds to step S403. If the normal shooting mode is set, the process proceeds to step S419.

  In step S403, the imaging control unit 208 stands by until the user presses the release button 116 halfway. When release button 116 is half-pressed, the process proceeds to step S404.

In step S404, focus adjustment and photometry are automatically performed. Further, in accordance with the automatic focus adjustment operation at this time, the imaging control unit 208 classifies the distance measuring points into the following five types.
Type 1: Focused distance measuring point Type 2: Distance measuring point that detected a distance measurement result close to Type 1 distance measuring point Type 3: Distance measuring object farthest from all distance measuring points Ranging point that detected the result Type 4: Ranging point that detected the ranging result close to the Ranging point of Type 3 Type 5: Ranging point not included in any of the above types

  For classification determination of type 2 ranging points, for example, threshold values α and β are held in the camera in advance, and ranging points satisfying the determination formula “α ≦ | defocus1-defocus | ≦ β” are set as type 2. Classify. Here, “defocus1” is the distance measurement result of the type 1 distance measurement point, and “defocus” is the distance measurement result of the distance measurement point of interest.

  In classification determination of type 4 ranging points, as in type 2, for example, ranging points satisfying the determination formula “α ≦ | defocus3-defocus | ≦ β” are classified as type 4. Here, “defocus3” is a distance measurement result of a type 3 distance measurement point.

  According to such a classification standard, a ranging point classified as type 1 or 2 corresponds to a focusing point in focus, and a ranging point classified as type 3 or 4 corresponds to a distance measurement in the background portion. It corresponds to a point.

  Then, the imaging control unit 208 determines the type-specific classification for all the distance measuring points, stores the result in the buffer memory, and proceeds to step S405.

In step S405, the imaging control unit 208 determines the scheduled imaging number N according to the photometric result of the AE unit 112 obtained in step S404. Number shoot N, for example, when the original exposure time required when shooting one in the scene T _S, the exposure time per one when plural shot was T _M, "T _S = T _M × N ”is satisfied. Further, the exposure time T _M is set to be equal to 1 / f second (f is a focal length), for example, which is a guide for the exposure time without camera shake. However, the exposure time T _M and the number of shots N are not limited to those described above.

  Next, in step S406, the shooting control unit 208 determines whether or not the release button 116 has been fully pressed by the user. If the release button 116 has been fully pressed, n, which is a count value of the current number of shots, is set to n = 0. Then, the process proceeds to step S407, and N consecutive images are started. If the release button 116 is not fully pressed, the process returns to step S403.

  In step S407, the shooting control unit 208 stores the focal length f and the aperture value F at the time of shooting in the buffer memory.

  In step S408, the number of shots n is counted up after performing the shooting process.

  Thereafter, in step S409, the shooting control unit 208 determines whether or not the current number of shots has reached the scheduled number of shots N determined in step S405. If the number has reached N, that is, the required number of shots has been completed. If YES, go to step S411. On the other hand, if the planned number of images N has not been reached, an automatic focus adjustment operation is performed in step S410, and the process returns to step S407. In the automatic focus adjustment operation in step S410, as in step S404, the distance measuring points are classified according to the aforementioned types and stored in the buffer memory. As described above, in this embodiment, each time the automatic focus adjustment operation is performed, the ranging points are classified and recorded by type.

  In step S411, the deviation detection unit 212 calculates the amount of positional deviation between a plurality of captured images. Here, an example of a specific calculation procedure will be described.

  First, the deviation detection unit 212 divides each of a plurality of image signals received from the signal processing unit 211 into a plurality of blocks as shown in FIG. FIG. 3 shows an example in which the long side direction of the image signal is divided into eight and the short side direction is divided into six, and divided into a total of 48 blocks. However, the number of divided blocks is arbitrary. In the following description, as shown in FIG. 3, the upper left block is represented as A1, the right block is represented as B1, and the lower right block is represented as H6. That is, each block is represented by symbols A1 to H6.

  Then, the deviation detection unit 212 performs a two-dimensional correlation calculation between the blocks at the same position among a plurality of image signals, and calculates a motion vector that is a positional deviation amount for each block. Then, a histogram of the calculated motion vectors of 48 blocks is obtained, and the mode value is used as the motion vector of the entire image.

  According to such a method for calculating the amount of positional deviation, the motion vector calculation result of a specific block is prioritized by weighting the motion vector calculation result of each block later to calculate the final motion vector of the entire image signal. It becomes easy to reflect in the result. It is also easy to calculate the motion vector of the entire image using only a specific block without using all blocks.

  In this embodiment, as will be described later, in order to determine the final motion vector of the entire image by assigning weights to each block, in step S411, the processing up to the calculation of the motion vectors of all the blocks is performed. Then, after recording the result in the buffer memory, the process proceeds to step S412.

In step S412, the shift detection unit 212 determines a weighting amount W to be given to a block that preferentially considers the motion vector calculation result when calculating the final motion vector of the entire screen according to the depth of field at the time of shooting. . As will be described later, in this embodiment, each block is classified into the following three groups.
Group A: Blocks to be prioritized when calculating the final motion vector of the entire screen Group B: Blocks to be prioritized when calculating the final motion vector of the entire screen Group C: Total of the final screen Blocks that are not particularly prioritized or considered when calculating motion vectors

  In the present embodiment, the motion vector calculation result of the block in the focused area, that is, the area in the vicinity of the type 1 or 2 distance measuring point is prioritized, and the motion vector of the entire image is obtained. For this reason, in step S412, a weighting amount corresponding to the depth of field is determined for each group in advance.

  At this time, considering the depth of field at the time of shooting, when the depth of field is shallow, the amount of blur in the background increases, and the accuracy when the motion vector is calculated using the blocks in the background region is low. End up. Therefore, a small weighting amount is given to the blocks belonging to the group C, and conversely, a large weighting amount is given to the focused blocks belonging to the groups A and B. As a result, it is possible to calculate the motion vector with high accuracy. On the other hand, when the depth of field is deep, the difference in the amount of blur between the area near the focusing point that is in focus and the other areas is smaller than when the depth of field is shallow. There is no need to make a large difference in quantity.

Thus, by considering the depth of field when determining the weighting amount W, it is possible to calculate a more accurate motion vector. The depth of field decreases as the focal length f of the lens at the time of shooting increases and the aperture value F decreases, and conversely increases as the focal length f decreases and the aperture value increases. Therefore, for the determination of weighting factors W, it is preferable to prepare uniquely W _A from the focal length f and the aperture value F as shown in FIG. 5, a table specifying the W _B and W _C in the camera .

In the example shown in FIG. 5, the closer to the upper left of the table, the larger the focal length f and the smaller the aperture value F, so that the depth of field is shallow. Therefore, a large value is given as the weighting amounts W _A and W _B , and a small value is given as the weighting amount W _C. On the other hand, the closer to the lower right of the table, the smaller the focal length f and the larger the aperture value F, so that the depth of field is deeper. Accordingly, values that are not so different are prepared as the weighting amounts W _A , W _B, and W _C. Thus, the weighting amount W can be easily determined by having a table as shown in FIG. 5 within a range that satisfies W _A ≧ W _B ≧ W _C. When step S412 is completed, the process proceeds to step S413.

  In step S413, as illustrated in FIG. 4B, the shift detection unit 212 determines the final type of all the distance measurement points from the classification information of the distance measurement point types in all the captured images. For example, in the case where the distance measurement point arrangement as shown in FIG. 3 is set and the four types of images are continuously captured and the classification by type of each distance measurement point is as shown in Table 1, the processing is as follows. become that way.

  When the results shown in Table 1 are obtained, the ranging points 1, 3, 5, and 6 have the same classification according to the type of the ranging points in all the captured images. The final type classification shall be the same type. On the other hand, with respect to the distance measuring points 2, 4 and 7, the types classified only at the time of photographing one of the images are different. Such a distance measuring point including a different type classified by type during photographing is generally classified into type 5 (a distance measuring point having no particular feature). However, with regard to ranging points including at least one type 3, one ranging point is always classified as type 3 because of the subsequent processing. Therefore, the distance measuring point having the largest number of times (number of sheets) classified into type 3 is forcibly classified into type 3. When the distance measuring points are classified according to such rules, the results shown in Table 2 are finally obtained.

  On the other hand, in the case where the distance measurement point arrangement as shown in FIG. 3 is set and the four types of distance measurement points are classified as shown in Table 3 when four images are continuously captured, the processing is as follows. become that way.

  When the results shown in Table 3 are obtained, the ranging points 2, 3, 4, 6, and 7 have the same classification according to the type of the ranging points in all captured images. The final type classification is the same type. On the other hand, with respect to the distance measuring points 1 and 5, the types classified only at the time of photographing one of the images are different. Such a distance measuring point including a different type classified by type during photographing is generally classified into type 5 (a distance measuring point having no particular feature). When the ranging points are classified according to the type according to such rules, the results shown in Table 4 are finally obtained.

  Further, in step S413, the deviation detection unit 212 reads the distance measuring points classified as type 1 for all the captured images, and determines whether or not they are all the same. If the type 1 distance measurement points coincide with each other, the process proceeds to step S414. If there is any type 1 distance measurement point, the process proceeds to step S415. That is, the presence of type 1 in the final classification means that the focused distance measuring point is the same in all the captured images, and therefore it can be determined that the main subject has not moved. . On the other hand, if Type 1 does not exist in the final classification, it can be determined that the main subject has moved. In steps S414 and S415, processing according to these circumstances is executed.

  This means that if the type 1 distance measurement points match, the main subject has not moved, but if the type 1 distance measurement points are different even once, the main subject will not move. This is because the subject is considered to be moving during continuous shooting. That is, in the block in the vicinity of the type 1 distance measuring point, the image differs depending on the shooting timing, and there is a high possibility that an accurate motion vector cannot be calculated without matching during correlation calculation. Therefore, in such a case, the motion vector of the entire image is calculated with priority given to the motion vector of the block in the vicinity of the type 3 distance measuring point, which is considered the background, over the block in the vicinity of the type 1 distance measuring point. .

  Therefore, if the type 1 ranging points are the same for all images, the process proceeds to step S414 in which the motion vector of the entire image is calculated with priority given to the motion vectors of the blocks near the type 1 ranging points. On the other hand, if they are not the same, the process proceeds to step S415 where the motion vector of the entire image is calculated with priority given to the motion vector of the block near the type 3 distance measuring point.

  For example, when the results shown in Table 1 and Table 2 are obtained, the distance measurement points classified into Type 1 are distance measurement points 5, and the process proceeds to step S414. On the other hand, when the results shown in Tables 3 and 4 are obtained, the distance measuring points classified as Type 1 are distance measuring points 5 in the first, third and fourth images. On the other hand, since the distance measurement point is 1 in the second image, the process proceeds to step S415.

  In steps S414 and S415, the deviation detection unit 212 classifies the blocks in the screen into the groups A, B, and C described above. However, in step S414 and step S415, in order to perform processing according to the presence / absence of the movement of the main subject, the reference to be referred for classification is different.

  In step S414, processing is performed on the assumption that the main subject has not moved. For this reason, the deviation detection unit 212 performs classification in which a block near the main subject is heavily weighted. Therefore, a block near the type 1 distance measuring point is set as the group A, a block near the type 2 distance measuring point is set as the group B, and other blocks are set as the group C. For example, when the result shown in Table 2 is obtained, blocks F3 and F4 corresponding to the distance measuring point 5 are set in the group A (see FIG. 3). In group B, blocks D3, D4, E3 and E4 corresponding to distance measuring point 1 are set. In group C, all other blocks are set.

  On the other hand, in step S415, processing is performed on the assumption that the main subject has moved. For this reason, the shift detection unit performs classification in which a block that is considered to be the background is heavily weighted. Therefore, a block near the type 3 distance measuring point is set as the group A, a block near the type 4 distance measuring point is set as the group B, and other blocks are set as the group C. For example, when the result shown in Table 4 is obtained, blocks D2 and E2 corresponding to the distance measuring point 2 are set in the group A. In group B, blocks B3, B4, C3 and C4 corresponding to distance measuring points 4 or 6 are set. In group C, all other blocks are set.

  Then, after the grouping of each block is completed in step S414 or S415, the process proceeds to step S416.

In step S416, the deviation detection unit 212 assigns the weighting amount obtained in step S412 to each block in FIG. 3 based on the classification result in step S414 or S415. That is, the classified blocks in the group A given amount of weight W _A, the classification block to a group B given amount of weight W _B, the classified blocks in the group B provide a weighting amount W _C. Then, after adding these weighting amounts W _A , W _B, or W _C to the motion vector obtained in step S411, a motion vector histogram is created.

For example, when the classification in step S414 is performed, the block near the type 1 distance measurement point belongs to the group A, the block near the type 2 distance measurement point belongs to the group B, and the other group belongs to the group C. The block belongs to. On the other hand, when the classification in step S415 is performed, the block near the type 3 distance measurement point belongs to the group A, the block near the type 4 distance measurement point belongs to the group B, and the other group belongs to the group C. The block belongs to. Then, a weight amount W _X (W _A ≧ W _B ≧ W _C ) is given to the blocks included in the group X (X is any one of A, B, and _C ) to calculate the final motion vector of the entire screen. Use.

  As weighting, for example, in a block having a weighting amount of W, W is counted for each motion vector. Then, the appearance frequency is made into a histogram. After creating the histogram, the process proceeds to step S417.

  In step S417, the deviation detection unit 212 selects the motion vector that appears most frequently in the histogram created in step S416, and determines the motion vector as the motion vector of the entire image.

  Next, in step S418, the coordinate conversion unit 213 performs coordinate conversion, the image storage unit 214 stores the image after conversion by the coordinate conversion unit 213, and the image composition unit 217 performs image composition. However, if the images are simply combined, images with different compositions will be combined, resulting in an area where the images do not overlap. Therefore, in step S418, the area where the images do not overlap is cut, and only the overlapping area between the plurality of images is combined, and then the diffusion complement process is performed to obtain the original frame size. Then, the synthesized image is displayed on the display unit 215 and recorded on the recording unit 216. In this way, a series of processing ends.

  On the other hand, when the process proceeds from step S402 to step S419, in step S419, the imaging control unit 208 waits until the release button 116 is half-pressed by the user. When release button 116 is half-pressed, the process proceeds to step S420.

  In step S420, focus adjustment and photometry are automatically performed. However, unlike step S404, the ranging points are not classified.

  Next, in step S421, the shooting control unit 208 determines whether or not the release button 116 has been fully pressed by the user. If the release button 116 has been fully pressed, the current shooting number n is set to n = 0, Proceeding to step S407, continuous shooting of N sheets is started. If the release button 116 is not fully pressed, the process returns to step S419.

  Then, in step S422, after performing a photographing process, the photographed image is displayed on the display unit 215 and stored in the recording unit 216. In this way, a series of processing ends.

  As described above, in the first embodiment, when the composite image mode is selected, the weighting amount is adjusted according to the presence or absence of the movement of the block belonging to the type 1 indicating the focused distance measuring point. ing. For this reason, it is possible to perform weighting according to the presence or absence of the movement of the main subject, and the overall motion vector can be calculated with high accuracy. As a result, it is possible to obtain a composite image with very little subject blur.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, motion vectors are calculated for all blocks obtained by previously dividing a captured image, and then the motion vectors of the entire image are calculated after weighting is performed according to the type of ranging point type. is doing. On the other hand, in the second embodiment, the distance measurement points are first classified by type, and then the motion vector is calculated only for the block to be calculated with priority given to the motion vector, thereby calculating the motion vector of the entire image. To do. Hereinafter, parts different from the first embodiment will be described in detail. FIGS. 6A to 6C are flowcharts illustrating the operation of the camera according to the second embodiment.

  In the second embodiment, as shown in FIG. 6A, when the image composition mode is set, the processes in steps S401 to S409 are performed in the same manner as in the first embodiment. However, if the current number of shots has reached the scheduled number of shots N in step S409, that is, if the required number of shots has been taken, the process proceeds to step S412 without calculating the motion vectors of all blocks, and deviation detection is performed. The unit 212 calculates the weighting amount W.

  Next, as illustrated in FIG. 6B, in step S413, the shift detection unit 212 determines the final types of all the distance measurement points from the classification information of the distance measurement point types in all the captured images. It is determined whether or not the type 1 distance measuring points are the same for all the captured images. Further, in step S413, the deviation detection unit 212 reads the distance measuring points classified as type 1 for all the captured images, and determines whether or not they are all the same. If the type 1 distance measuring points match, the process proceeds to step S501. If there is a distance that does not match, the process proceeds to step S502.

  In steps S501 and S502, the deviation detection unit 212 classifies the blocks in the screen into the groups A and B described above. However, in step S501 and step S502, in order to perform processing in accordance with the presence or absence of the movement of the main subject, the reference to be referred for classification is different.

  In step S501, processing is performed on the assumption that the main subject has not moved. For this reason, the deviation detection unit 212 performs classification in which a block near the main subject is heavily weighted. Therefore, a block near the type 1 distance measuring point is set as the group A, and a block near the type 2 distance measuring point is set as the group B. For example, when the result shown in Table 2 is obtained, blocks F3 and F4 corresponding to the distance measuring point 5 are set in the group A (see FIG. 3). In group B, blocks D3, D4, E3 and E4 corresponding to distance measuring point 1 are set.

  On the other hand, in step S502, processing is performed on the assumption that the main subject has moved. For this reason, the shift detection unit performs classification in which a block that is considered to be the background is heavily weighted. Therefore, a block near the type 3 distance measuring point is set as the group A, and a block near the type 4 distance measuring point is set as the group B. For example, when the result shown in Table 4 is obtained, blocks D2 and E2 corresponding to the distance measuring point 2 are set in the group A. In group B, blocks B3, B4, C3 and C4 corresponding to distance measuring points 4 or 6 are set.

  Then, after grouping of each block is completed in step S501 or S502, the process proceeds to step S503.

  In step S503, the deviation detection unit 212 performs a two-dimensional correlation calculation only on the blocks classified into the group A or the group B, and calculates a motion vector.

  Next, in step S504, the deviation detection unit 212 creates a motion vector histogram while taking the weighting amount W into consideration.

  Thereafter, the same processing as in the first embodiment is performed. Also, when the normal shooting mode is set, the same processing as in the first embodiment is performed.

  In the second embodiment, as described above, the motion vector calculation process is performed only on the blocks to be preferentially considered when calculating the final motion vector of the entire image. For this reason, in addition to the same effect as that of the first embodiment, an effect that the processing time required for motion vector calculation can be further reduced can be obtained.

  In the description of the first and second embodiments, the configuration based on the phase difference AF method is described. However, the present invention is not limited to this, and automatic focus detection using a contrast detection method is performed. The present invention can also be applied to a camera to be performed. The contrast detection method focuses on a specific region in the frame (hereinafter, this region is referred to as a contrast evaluation region), and searches for a lens position where the contrast evaluation value of that portion is the highest. In a camera in which such an autofocus detection method is a contrast detection method, the present invention can be applied if weighting is performed so as to give priority to the motion vector calculation results of blocks in the vicinity of the contrast evaluation region.

  In the description of the first and second embodiments, a case is described in which a single-lens reflex camera equipped with a phase difference AF method is used to capture a plurality of images with the autofocus function turned on. However, since the focus detection information from the AF unit 114 can be acquired even when a plurality of images are taken with the autofocus function turned off (manual focus state), it is possible to determine a region near the focused region. The present invention can be applied.

  Furthermore, in the description of the first and second embodiments, a method of calculating a motion vector that is a composition shift of a plurality of still images has been described. However, the present invention may be applied to a moving image. it can. For moving image blur correction, a technique is known in which a specific image is used as a reference, a positional deviation between this and another continuous image is detected, and an image obtained by correcting this is reproduced. This technique is described in, for example, JP-A-11-187303. Combining the present invention with this technique makes it possible to increase the accuracy of motion vector calculation and shorten the processing time, and is therefore effective during playback and recording of moving images. In addition, in order to avoid the adverse effects of moving the main subject, etc. with regard to the image stabilization of the moving image, the motion vector of the block near the distance measurement point where the distance measurement result is obtained in the area (background) where there is no main subject is obtained. It is desirable to give priority.

  Further, a CCD sensor may be used as the image sensor instead of the CMOS sensor.

  In these descriptions, the motion vector is calculated in the camera or video camera, but the present invention is not limited to this. For example, if the information at the time of automatic focus detection is an image written in an EXIF area or the like, the image processing apparatus such as a PC executes the processing of the present invention for a plurality of images received from the camera or the network. By doing so, a motion vector can be calculated.

  That is, the embodiment of the present invention can be realized by, for example, a computer executing a program. Also, means for supplying a program to a computer, for example, a computer-readable recording medium such as a CD-ROM in which such a program is recorded, or a transmission medium such as the Internet for transmitting such a program is also applied as an embodiment of the present invention. Can do. The above program can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.

1 is a side sectional view showing a schematic configuration of a camera according to a first embodiment of the present invention. It is a block diagram which shows the functional structure of the camera which concerns on 1st Embodiment. It is a figure which shows the example of the division | segmentation of a block. It is a flowchart which shows operation | movement of the camera which concerns on 1st Embodiment. FIG. 4 is a flowchart illustrating the operation of the camera according to the first embodiment, following FIG. 4-1. FIG. 4 is a flowchart illustrating the operation of the camera according to the first embodiment, following FIG. 4-1. It is a figure which shows the determination table of weighting amount. It is a flowchart which shows operation | movement of the camera which concerns on the 2nd Embodiment of this invention. FIG. 6 is a flowchart illustrating the operation of the camera according to the second embodiment, following FIG. FIG. 6 is a flowchart illustrating the operation of the camera according to the second embodiment, following FIG.

Explanation of symbols

114: AF unit 208: Imaging control unit 212: Deviation detection unit 213: Coordinate conversion unit 214: Image storage unit 217: Image synthesis unit

Claims (7)

For each of the plurality of captured images to identify the focus detection area focused at the time of shooting, the farthest distance to the first region and the Utsushitai preceding Symbol within a predetermined range from the focus detection area A region determination means for dividing the second region into at least two regions ;
And displacement detection means for detecting a positional deviation between before Symbol plurality of captured images,
Have
When the focus detection areas match between the plurality of captured images, the misregistration detection means sets a weight for the first area that is greater than or equal to the weight set for the second area . An image processing apparatus that detects the positional deviation above. The area determination means includes, for each captured image, an area indicating a distance measurement result within a predetermined range from a distance measurement result in the first area within an area other than the first and second areas. As a third area,
The misregistration detection means sets weighting within a range in which the weights set for the first and second areas are set as upper and lower limits for the information indicating the misalignment in the third area, respectively. the image processing apparatus according to claim 1, wherein the detecting the positional deviation in terms of the. For each of the plurality of captured images, a focus detection area focused at the time of shooting is specified, and the first area within the predetermined range from the focus detection area and the distance from the subject to the second are the longest. A region determination means for dividing the region into at least two regions;
A displacement detection means for detecting displacement between the plurality of captured images;
Have
When the plurality of captured images include those in which the focus detection areas do not match, the misregistration detection means assigns a weight greater than or equal to a weight set for the first area to the second area. images processor you and detects the positional deviation on set for region. The area determination means includes, for each of the captured images, an area indicating a distance measurement result within a predetermined range from a distance measurement result in the second area in an area other than the first and second areas. As a third area,
The misregistration detection means sets weighting within a range in which the weights set for the first and second regions are set as the lower limit and the upper limit, respectively, for the information indicating the misregistration in the third region. The image processing apparatus according to claim 3 , wherein the position shift is detected. The positional deviation detection means, the image processing apparatus according to any one of claims 1 to 4, characterized in that to change the ratio of the weighted according to the depth of field at the time of imaging of the captured image. For each of the plurality of captured images to identify the focus detection area focused at the time of shooting, the farthest distance to the first region and the Utsushitai preceding Symbol within a predetermined range from the focus detection area An area determination step for dividing the area into at least two areas with the second area ;
A positional deviation detection step of detecting a positional deviation between before Symbol plurality of captured images,
Have
When the focus detection areas match between the plurality of captured images, in the misregistration detection step, a weight equal to or higher than the weight set for the second area is set for the first area . An image processing method characterized by detecting the positional deviation above. For each of the plurality of captured images, a focus detection area focused at the time of shooting is specified, and the first area within the predetermined range from the focus detection area and the distance from the subject to the second are the longest. An area determination step for dividing the area into at least two areas;
A displacement detection step for detecting displacement between the plurality of captured images;
Have
If the plurality of captured images include those in which the focus detection areas do not coincide with each other, in the misregistration detection step, a weight equal to or higher than a weight set for the first area is set. images processing how to and detecting the positional deviation on set for region.

Images