JP2015012482A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which aligns images by suppressing impact of any erroneous detection of a mobile vector when aligning three or more images, and aligns images by predominantly suppressing blurring of one particular part of a main object.SOLUTION: An image processing apparatus comprises: calculating means that calculates a mobile vector for every divided area in three or more images acquired by imaging means; aligning means that aligns the plurality of images on the basis of the mobile vector calculated by the calculating means; and counting means that counts the number of times the mobile vector used by the aligning means in which the mobile vector is in every divided area. The aligning means identifies a mobile vector used for the alignment of images by using the number of times counted by the counting means from the mobile vector calculated by the calculating means.

Description

  The present invention relates to an image processing apparatus and an image processing method for aligning and synthesizing a plurality of images obtained by photographing.

  There is panning as a shooting technique that expresses the speed of moving subjects. The purpose of the photographing technique is to make the photographer pan the camera in accordance with the movement of the subject so that the moving subject is stopped in the photographed image and the background flows. In general panning, shooting is performed by adjusting the shutter speed slower than usual in accordance with the moving speed of the subject (main subject) to be shot. However, since the shutter speed is slow, the main subject is often blurred due to camera shake or the difference between the moving speed of the main subject and the panning speed. In view of the above problem, facilitation of panning using an image processing technique is desired.

  For example, in Patent Document 1, image processing for detecting the start position with respect to the direction of motion of a moving object included in a plurality of images and adjusting the position of the moving object so that the detected start position is the same position on each image. An apparatus is disclosed. In Patent Document 2, first, a feature point is extracted from a certain image, and a vector from the feature point to a reference position is calculated. Next, feature points are extracted from another image, and the pixel position indicated by the corresponding vector is calculated based on the result calculated from the first image. An alignment device is disclosed in which the number of pixel position calculations is totaled for each pixel, and the positions of the two images are aligned based on the total result.

JP 2012-109899 A JP-A-10-91788

  However, in the prior art disclosed in Patent Document 1 described above, since the position is adjusted based on the leading position of the moving object, if there is an error in motion detection at the leading position, it is easily affected. Also, it is not disclosed how to handle other than the moving object head position.

  The prior art disclosed in Patent Document 2 is based on the premise of alignment between two images, and does not disclose the relationship between images in alignment of three or more images.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing apparatus and an image processing method for aligning three or more images by aligning the positions while suppressing the influence of erroneous movement vector detection. It is another object of the present invention to provide an image processing apparatus and an image processing method that enable alignment while focusing on blurring of a part of a main subject.

  In order to achieve the above object, an image processing apparatus of the present invention calculates a movement vector for each divided area in at least three or more images captured by an imaging unit, and calculates by the calculation unit. Positioning means for aligning the plurality of images based on the movement vector to be performed, and counting means for counting the number of times the movement vector used in the alignment means is used for each of the divided areas. The alignment means specifies a movement vector used for alignment between images using the number of times counted by the counting means from the movement vector calculated by the calculation means.

  According to the present invention, in the alignment of three or more images, it is possible to align the positions while suppressing the influence at the time of erroneous detection of the movement vector. In addition, it is possible to perform alignment while mainly suppressing the shake of a part of the main subject.

1 is a basic configuration diagram of an imaging apparatus according to a first embodiment. Flowchart of CPU processing according to the first embodiment Explanatory drawing of a main subject and a search area according to the first embodiment Flow chart of alignment according to the first embodiment Explanatory drawing of the alignment concerning 1st Embodiment Flowchart of CPU processing according to the second embodiment Flow chart of alignment according to the second embodiment Explanatory drawing of the alignment concerning 2nd Embodiment FIG. 4 is an image diagram of an image generated by panning shooting according to an embodiment of the present invention.

(First embodiment)
FIG. 1 is a block diagram of an imaging apparatus as an example of an image processing apparatus according to the first embodiment of the present invention.

  The imaging apparatus 100 may be any electronic device having an imaging function, such as a digital camera and a digital video camera, as well as a mobile phone with a camera function, a computer with a camera, and a scanner. In addition, a part or all of the imaging apparatus 100 in the present embodiment can be used as the image processing apparatus in the present embodiment. The image processing apparatus does not necessarily have an imaging function, and may have a function of processing an image output from the imaging element 102 or an image stored in each storage device.

  The optical system 101 includes a lens, a shutter, and a diaphragm. The optical system 101 guides a light beam from a subject to the image sensor 102 and forms an optical image of the subject on the image sensor 102. Information such as the focal length, shutter speed, and aperture value is transmitted to the central processing unit (hereinafter referred to as CPU) 103.

  An image sensor 102 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) converts an optical image formed by the optical system 101 into an electrical signal. Thereafter, it is digitized through an AD converter and then stored in the primary storage device 104. In this embodiment, the pixel array of the image sensor 102 is a Bayer array of RGB pixels, but the application of the present invention is not limited to this pixel arrangement. For example, it may be an array of complementary color filter pixels, or a functional pixel or the like may be provided for the purpose of color measurement, distance measurement, and the like in addition to such shooting pixels. The electrical gain (hereinafter referred to as ISO sensitivity) of the image sensor 102 is set by the CPU 103.

  An angular velocity sensor 105 such as a gyro sensor detects a shake, converts it as an electrical signal, and transmits it to the CPU 103.

  The CPU 103 functioning as a control unit realizes each function of the imaging apparatus 100 by controlling each part of the imaging apparatus 100 according to an input signal or a prestored program. In the following description, at least a part of the functions realized by the CPU 103 executing the program may be realized by dedicated hardware such as ASIC (Application Specific Integrated Circuit).

  The primary storage device 104 is a volatile device such as a RAM (Random Access Memory), and is used for the work of the CPU 103. Information stored in the primary storage device 104 is used by the image processing unit 106 or recorded on the recording medium 107.

  The secondary storage device 108 is a nonvolatile storage device such as an EEPROM, for example, stores a program (firmware) for controlling the imaging device 100 and various setting information, and is used by the CPU 103.

  The recording medium 107 records data of an image obtained by photographing that is stored in the primary storage device 104. Note that the recording medium 107 can be detached from the imaging apparatus 100, for example, like a semiconductor memory card, and the recorded data can be read out by other devices such as a personal computer. That is, the imaging apparatus 100 has a mechanism for attaching and detaching the recording medium 107 and a read / write function.

  The display unit 109 has a function of displaying information stored for display in the temporary storage device 104 according to an instruction from the CPU 103. The present embodiment has a live view (viewfinder) display function for sequentially displaying at least part of images continuously acquired from the image sensor 102. In addition, it has a display function such as a GUI (Graphical User Interface) for interactive display and playback and display of recorded images recorded on the recording medium 107 and the like after shooting.

  The operation unit 110 is an input device group that receives user operations and transmits input information to the CPU 103. For example, the operation unit 110 may be an input device using voice, line of sight, etc. as well as buttons, levers, touch panels, and the like. Note that the imaging apparatus 100 according to the present embodiment has a plurality of image processing patterns that the image processing apparatus 106 applies to the captured image, and the patterns can be set as the imaging mode from the operation unit 110.

  The image processing unit 106 performs image processing called so-called development processing, and also adjusts color tone according to the shooting mode. For example, interpolation processing such as demosaicing processing, white balance processing, correction processing such as aberration and distortion, sharpness, gamma processing, matrix calculation, color conversion processing using a lookup table, and the like can be given. The image processing unit 106 also performs display processing such as resizing and gamma conversion for display on the display unit 109 and recording processing such as encoding and compression during recording on the recording medium 107. The image processing unit 106 also performs processing for generating a panning image in the present embodiment. A plurality of images to be combined and image data generated in the process are stored in the primary storage device 104, for example. Note that at least a part of the functions of the image processing unit 106 may be realized by the CPU 103 as software.

  FIG. 2 is a flowchart relating to processing of the CPU 103 relating to shooting and recording in the panning shooting mode.

  Note that in this embodiment, there are two steps for capturing and recording an image. S1 of the operation unit 110 that appears after that means an instruction to prepare for shooting, and S2 of the operation unit 110 is This means an instruction to capture and record the main image. In the present embodiment, S1 is associated with half-pressing of the shutter button of the operation unit 110, and S2 is associated with full-pressing of the shutter button so that the user can input. In addition, when the operation unit 110 is a touch panel, it is conceivable to change the touch operation on the touch panel, and any operation can be assigned to S1 and S2.

  In step S <b> 201, the CPU 103 receives a user input from the operation unit 110.

  In step S202, the CPU 103 adjusts settings such as the focal length, shutter speed, and aperture value of the optical system 101 based on the input information.

  In step S <b> 203, the CPU 103 adjusts settings such as the ISO sensitivity of the image sensor 102 based on the input information.

  In step S <b> 204, the CPU 103 presents information regarding the changed setting to the user on the display unit 109.

  In step S <b> 205, the CPU 103 receives information on the angular velocity of the imaging apparatus 100 detected by the angular velocity sensor 105. Further, by always executing S205 in the same manner as S202 and S203, information on angular velocity can be embedded as image information together with information on focal length, shutter speed, aperture value, ISO sensitivity, and the like. Then, the reprocessed image in the camera and the post-processing by the PC application are facilitated for the captured image. The information on the angular velocity to be recorded may be the angular velocity itself, the angle moved between images, or the angular displacement.

  The order of S201 to S205 is not limited to this, and the order can be freely changed according to the process.

  In S206, the presence or absence (ON / OFF) of the input of S1 of the operation unit 110 is determined, and the CPU 103 repeats the operations of S201 to S205 unless S1 of the operation unit 110 is input.

  The subsequent processing is processing after S1 of the operation unit 110 is input (ON) in S206.

  In step S <b> 207, the CPU 103 measures brightness using a photometric sensor that is a part of the optical system 101. In the case of an AE (Auto Exposure) mode in which automatic exposure control is performed, exposure is automatically adjusted using the shutter speed, aperture value, and ISO sensitivity.

  In S208, in the AF (Auto Focus) mode in which automatic focus adjustment control is performed, the CPU 103 measures the subject distance using a distance measuring sensor disposed in a part of the optical system 101 or in the image sensor 102, and Focus adjustment is performed based on the defocus amount.

  Note that the order of S207 and S208 is not limited to this, and the order can be freely changed according to the processing.

  In S209, the presence or absence (ON / OFF) of the input of S2 of the operation unit 110 is determined, and the CPU 103 repeats the operations of S201 to S208 unless S2 of the operation unit 110 is input. Further, it may be configured to detect whether or not the input of S1 of the operation unit 110 is continued in S209 and to return to step S201 when the input of S1 of the operation unit 110 is not performed (OFF). Good.

  The subsequent processing is processing after S2 of the operation unit 110 is input (ON) in S209.

  In S210, in response to the shooting instruction in S2 from the operation unit 10, the number of images necessary for the panning process is shot. At this time, in the panning mode, it is assumed that the user pans the imaging device 100 (or a part thereof including the imaging device 102) to obtain a panning effect. The number of images used for panning may be set in advance by the user, or may be a number that is automatically calculated from the speed of the main subject, the amount of panning, a setting that blurs the background area, and the like.

  In S211, the image processing unit 106 performs the development process as described above on the image data captured in S210.

  In S212, the CPU 103 detects a main subject region by detecting a movement vector for each divided region between a plurality of images, and performs alignment processing in the detected main subject region. Here, in the present embodiment, the main subject area refers to a subject area that is detected separately from a background area, which will be described later, in the captured image, and the subject in the area is composed of, for example, a plurality of persons. It may be. In the present embodiment, a moving subject area (moving body area) is detected as a main subject area for panning. That is, a stationary object (stationary object) among a plurality of shootings is handled as a background. However, as described above, when it is assumed that shooting is performed by panning, a subject that moves corresponding to the panning amount in the opposite direction of the panning direction is stationary when compared between images. The subject. A subject that moves in the panning direction by the same amount as the panning amount is determined as a moving object. Details will be described later. On the other hand, when it is assumed that shooting is performed with a tripod or the like fixed, an area where a motion vector can be largely detected is considered as a main subject area.

  In S213, the CPU 103 combines the plurality of images that have been aligned so that the main subject areas match in S212 by the image processing unit 106, and generates a composite image as a panning image. A plurality of methods are conceivable as a method for generating a composite image having a visual effect similar to that of panning. In this embodiment, the background area of a plurality of images obtained by shooting is subjected to a blurring process (filtering process or the like) with a blurring amount (number of taps) based on the amount of background movement between images, and the main subject area is shot. Position alignment is performed so that the plurality of images coincide with each other, and the result is synthesized by addition averaging. Here, the amount of movement of the background can be calculated from the angular velocity obtained by the angular velocity sensor 105. Not limited to this, for example, alignment is performed so that the main subject region matches between a plurality of captured images, and addition is performed. Further, the main subject region of the combined image, that is, a moving subject region other than the moving subject region exists. Blur processing may be applied to the background area. Further, an image obtained by further combining the image obtained by blurring the combined image as described above and the image before blurring may be used as the final combined image.

  In S <b> 214, the CPU 103 displays, on the display unit 109, image data obtained by performing display processing by the image processing unit 106 on the combined image data generated in S <b> 213 and the original image data before combining.

  In step S215, the CPU 103 stores the image data obtained by performing encoding processing, compression processing, and the like on the composite image data generated in step S213 on the recording medium 107 and the original pre-combination image data. Record. In the present embodiment, as the setting of the recording image, it is possible to set recording of a RAW image that has not been subjected to development processing or the like, or recording of a JPEG image that has been subjected to development processing and conforms to the standard. Depending on these settings, the development process in S210 and the encoding process in S216 may not be applied.

  In this embodiment, the main subject detection process in S212 and the panning composition process in S213 are performed on the image after the development process in S211. However, the present invention is not limited to this, and the image before the development process is processed. Each process can also be applied.

  In the main subject area detection processing in S212, it is assumed that the position of the main subject is within the search area for detecting the movement vector. If the position of the main subject in each image does not fall within the search area, The position of the main subject needs to be within the search area. In this embodiment, in S212, the motion vector is detected using a search block large enough to reduce the image and pick up the feature points of the low-contrast subject, and the position of the main subject fits in the search region even with low accuracy. After that, more accurate movement vector detection is performed. FIG. 3 shows a state in which a user is detected as a main subject from three images taken by the user while panning with the vehicle as the main subject, and the images are aligned with the main subject. An image 301, an image 304, and an image 305 show a state in which images taken in this order are displayed as they are, and an image 306 shows an example of a block-unit movement vector required for specifying a main subject. The movement vector 306 is detected between the image 301 and the image 304 as in the image 306, the main subject of the image 301 is detected as 302 a, and the region where the main subject 302 a is located in the image 301 is set as the search region 303. In the images 304 and 305, it can be seen that the main subjects 302b and 302c are not contained in the search area 303 set in the same area as the angle of view of the image 301. In such a case, the image 304 and the image 305 are moved in the memory like the image 308 and the image 309 so that the main subject 302b and the main subject 302c fit in the search area 303. The dotted lines in the images 308 and 309 indicate the edges of the images 304 and 305 before movement. In FIG. 3, since the image 301 is the reference image, the image 307 is not moved.

  FIG. 4A is a flowchart showing details of detection of the main subject area (moving body area) and registration processing between images in the main subject area performed in S212 of FIG.

  In step S401, the CPU 103 sets a search block obtained by dividing an image into a plurality of areas, and detects a movement vector between images in units of search blocks. Among the detected movement vectors, the number of the same vector is counted. FIG. 4B shows the state after counting as a histogram of the vector size on the horizontal axis and the number (frequency) on the vertical axis. The horizontal axis distinguishes the direction of the movement vector from 0 as a boundary. First, the main subject component in which the movement vector is separated from the background movement amount as shown in the drawing with reference to the background movement amount 410 based on the shake detected by the angular velocity sensor 105 (that is, the movement information of the imaging means between a plurality of images). 411 and the background component 412 in the vicinity of the background movement amount. Next, the movement vector having the largest number among the movement vectors of the classified main subject component 411 is set as a main subject vector 413. Here, it is possible to classify the main subject and the background from the histogram of the movement vector without using the shake information detected by the angular velocity sensor 105. In this case, for example, assuming that there is a main subject in the vicinity of the center in each image by panning shooting by the user, for example, the vector size is close to zero, and the moving vector is present in a continuous area in the image. The corresponding area is considered the main subject area. However, depending on the size of the search area of the movement vector, there is also an erroneous detection of the movement vector. Therefore, the main subject and the background can be classified more accurately by referring to the shake information from the sensor as in the above embodiment.

  As described above, in step S402, the CPU 103 determines that the search block that has detected the movement vector corresponding to the main subject area has a movement vector used in previous image alignment. A weight corresponding to the number of times is added.

  In step S403, the CPU 103 specifies a movement vector corresponding to the main subject area. In the present embodiment, the movement amount of the background between the target images is estimated based on the signal from the angular velocity sensor 105, and the movement vector having the largest number when the movement vector corresponding to the background is excluded is the main subject. The movement vector corresponding to is estimated. However, the number to be counted and compared between the movement vectors is a number based on the weight added to each region in S402. In this embodiment, the angular velocity of the imaging apparatus 100 is detected by the angular velocity sensor 105. However, the present invention is not limited to this, and an acceleration sensor, a sensor for measuring a position, or the like may be used.

  Further, when counting the number of movement vectors in S401, the processing may be performed simultaneously in consideration of the weight of the search block in S402 (for example, counting the weight).

  In step S <b> 404, the CPU 103 performs image alignment by shift movement in the memory of the primary storage device 104 using the movement vector of the main subject. Here, since it is assumed that the shooting is performed with panning as a method for determining the movement vector of the main subject, the background movement vectors are classified by the above method. As described above, when the background movement vector cannot be classified by the signal of the angular velocity sensor 105, the second most movement vector is determined, or whether an area having the movement vector exists with a certain amount of clumps. Detect and estimate the movement vector of the main subject. When it is assumed that shooting is performed with the imaging apparatus 100 fixed using a tripod, the largest number of movement vectors on the image can be simply determined as the movement vector of the main subject. Further, the movement vector may be handled separately in the H direction and the V direction, or may be handled in combination.

  In step S <b> 405, the CPU 103 adds a weight for use in the next search to the search block in which the estimated motion vector corresponding to the main subject is calculated.

  Finally, in S406, it is determined whether there is a remaining image to be aligned. That is, as long as there is an image to be aligned, the process returns to S401, the next image is aligned, and when the alignment of all the images is completed, the process of S212 in FIG.

  Details of the method of estimating the movement vector of the main subject area in the alignment process of FIG. 4A will be described with reference to FIGS. 5A and 5B.

  In FIG. 5A, images 501 to 504 are taken continuously, and the alignment in the region 505 is taken as an example. An area 505 is enlarged and a search block is arranged 506. A movement vector such as 306 in FIG. 3 is detected for each search block.

  FIG. 5B shows the weight corresponding to each search block set as 506, and is temporarily stored in the RAM in association with the position of the search block. Reference numeral 507 denotes an initial value. In this embodiment, the initial value is 0. Reference numeral 508 denotes detection results of the movement vectors of the images 501 and 502, and 512 and 515 denote detection results of the movement vectors of the images 502 and 503 and detection results of the images 503 and 504, respectively. The weight distributions used when calculating the movement vector of the main subject from the movement vectors of the detection results 508, 512, and 515 are 509, 513, and 516, respectively. A pattern 510 a in the corresponding weight distribution 509 indicates a region that is substantially the same as the most numerous movement vectors in the detection result 508. The pattern 511a indicates a region in the detection result 508 that is substantially the same vector as the second largest number of movement vectors. Furthermore, the patterns 510b and 511b in the weight distribution 513 corresponding to the detection result 512 are almost the same vectors as the movement vector having the largest number and the movement vector having the second largest number in the movement vector detection result 512, respectively. It shows a certain area. In addition, the patterns 510c and 511c in the weight distribution 516 corresponding to the detection result 515 are substantially the same vectors as the movement vector having the largest number and the movement vector having the second largest number among the movement vector detection results 515, respectively. It shows a certain area.

  In the detection result 509, since the weight of each search block in the weight distribution 509 has an initial value of 0, the movement vector having the largest number of movement vectors may be used as the movement vector of the main subject. Accordingly, when the number of movement vectors of the pattern 510a and the pattern 511a is compared in the weight distribution 509, the pattern 510a has five patterns 510a and four patterns 511b, and the pattern 510a is larger. For this reason, the movement vector of the pattern 510a is used as a movement vector of the main subject area in the subsequent alignment processing.

  Next, in the detection result 512, 1 is added as a weight to the search block having the movement vector used in the alignment of the image 501 and the image 502 in the weight distribution 513. In this example, the pattern 510b has a total of six, with four movement vectors and two search block weights. Therefore, alignment is performed using the movement vector of the pattern 510b. By the way, when comparing the totals based on the weight, if there are two or more types of movement vectors that are the most promising candidates, give a specific gravity according to either the number of movement vectors or the weight of the search block. It shall be. Or it shall follow a detection order.

  Finally, in the detection result 515, in the weight distribution 516, 1 is added to the search block having the movement vector used for the alignment of the image 503 and the image 504 after taking over the current weight. The pattern 510c has a total of eight, including four movement vectors and four search block weights. In addition, the pattern 511c has a total of 7, with four movement vectors and three search block weights. Therefore, alignment is performed using the movement vector of the pattern 510c as a movement vector corresponding to the main subject area.

  As described above, in the present embodiment, alignment is performed with priority given to evaluation values from the same region as the subject region used for alignment in the past. By doing so, image misalignment can also be suppressed for images such as 501 and 502, 502 and 503, and 503 and 504 that have a relationship such as 501 and 504 that do not directly detect a movement vector. In addition, the movement vectors in the same region are not simply used, but weights are given priority for handling. That is, even when an erroneous detection of a movement vector occurs, the influence of the erroneous detection can be suppressed by using the weight accumulated with the movement vector of another area.

  In FIG. 5, the block weight is described as 1. However, the present invention is not limited to this, and the weight coefficient can be freely adjusted. For example, the coefficient may be set between 0 and 1, and the weight may be obtained by multiplying the number of times used for alignment.

  As described above, in the present embodiment, in the alignment of three or more images, the area corresponding to the movement vector of the main subject detected between the images is weighted to calculate the movement vector of the main subject between the next images. Used for. By doing in this way, the position can be aligned while suppressing the influence at the time of erroneous detection of the movement vector. In addition, an appropriate panning-like composite image can be generated using this alignment method.

(Second Embodiment)
In the first embodiment, when sequentially aligning a plurality of images with a main subject, a region corresponding to the movement vector of the main subject detected between the images is weighted to improve the alignment accuracy. Processing was shown. Further, in the present embodiment, alignment is performed in an area in the main subject area that is not desired to be shaken by using auxiliary information such as a signal from the angular velocity sensor 105 and subject detection results such as face detection and human body detection by known image analysis. To be done. Since the configuration of the imaging apparatus according to the second embodiment is the same as that of the first embodiment shown in FIG.

  FIG. 6 is a flowchart relating to processing of the CPU 103 relating to shooting and recording in the panning shooting mode.

  Steps similar to those in FIG. 2 are denoted by the same step numbers and description thereof is omitted. In step S601, a specific area that is particularly important among the main subject area and the main subject area is determined using the angular velocity detected from the signal of the angular velocity sensor 105 and a known subject detection result. Align with the area as a reference.

  FIG. 7 is a flowchart showing details of main subject detection and alignment processing using auxiliary information performed in S601 of FIG. Note that steps that perform the same processing as in FIG.

  In the present embodiment, in step S701, the CPU 103 weights the search block based on the region that is important in the subject determined from the angular velocity of the imaging device 100 detected by the angular velocity sensor 105 and the subject detection result described above. Set. This will be described in more detail with reference to FIG.

  Here, it is assumed that the panning direction can be determined from the angular velocity of the imaging apparatus 100, and the moving direction 901 of the main subject can be determined relatively from the movement vector for each region. At this time, the weight of the weight distribution 902 is set so that the top of the main subject is given priority as in the left colored portion 903. Note that the weight may be set so that priority is given to a part of the head of the main subject, for example, the leading edge of the main subject, as in the color painting unit 904.

  When a face is detected by subject detection means such as a face detection circuit incorporated in the image processing unit 106, an area 905 that will be important in the detected subject (area), for example, an eye in the face area The weight of the search block is set like the weight distribution 906 centering on the area of. For example, the weight of the search block may be set like the weight distribution 907 by combining the concepts of the weight distribution 902 and the weight distribution 906. As another method for determining an important specific area, the main subject area corresponding to the focus position may be set as the specific area based on the information of the focus area in the AF operation.

  As described above, in the present embodiment, alignment is performed with priority given to evaluation values from the same region as the subject region used for alignment in the past. By doing so, image misalignment can also be suppressed for images such as the image 501 and the image 504 that are not related to the direct movement vector detection. In particular, in the present embodiment, a specific region that is particularly desired as a reference is determined based on the motion information of the imaging apparatus or the subject detection information. Also, the movement vectors in the same region are not simply used, but are weighted and handled preferentially. That is, even when an erroneous detection of a movement vector occurs, the influence of the erroneous detection can be suppressed by using the weight accumulated with the movement vector of another area.

  Further, in the present embodiment, by adding a weight to the search block, it is possible to perform an alignment that mainly suppresses blurring of a part of the main subject. Further, by using shooting conditions such as the angular velocity of the imaging apparatus 100 and subject recognition such as face detection and human body detection, it is possible to suppress blur at a characteristic part of the main subject. It is possible to provide an image that is inconspicuous.

  Images of the combined images generated in the above embodiments are shown in FIGS. An image 904 is an image diagram of an image generated by combining in step S213 based on the images 901, 902, and 903. The moving object, the main subject vehicle, is clearly visible, and the background trees are reflected. In this method, the background flow can be changed and controlled according to the shooting interval (frame rate), the number of combined images, the number of taps of blurring processing, and the like.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  In addition, the present invention is not limited to devices such as digital cameras, but includes built-in or external connection of imaging devices such as mobile phones, personal computers (laptop type, desktop type, tablet type, etc.), game machines, etc. It can be applied to any device. Therefore, the “imaging device” in this specification is intended to include any electronic device having an imaging function.

DESCRIPTION OF SYMBOLS 100 Image pick-up device 101 Optical system 102 Image pick-up element 103 Central processing unit (CPU)
104 Primary storage device 105 Image processing device 106 Recording medium 107 Secondary storage device 108 Display unit 109 Operation unit

Claims (12)

Calculating means for calculating a movement vector for each divided region in at least three or more images picked up by the image pickup means;
Alignment means for aligning a part of the plurality of images based on the movement vector calculated by the calculation means;
Counting means for counting the number of movement vectors used in the alignment means for each of the divided areas;
The image processing apparatus characterized in that the alignment unit specifies a movement vector used for alignment between images from the movement vector calculated by the calculation unit, using the number of times counted by the counting unit. The alignment means includes
The movement vector used for the alignment between the images is specified by adding a weight according to the number of times counted by the counting unit to the movement vector for each divided area. Image processing device. The alignment means includes
From the movement vector calculated by the calculating means, a movement vector corresponding to a specific area which is a main subject area and is used as a reference for alignment is specified and used for alignment between images.
A motion vector corresponding to a background in the plurality of images is estimated from a motion vector for each of the divided regions based on motion information of an imaging unit that captures the plurality of images between the plurality of images. The image processing apparatus according to claim 1 or 2. The alignment means includes
From the movement vector calculated by the calculating means, a movement vector corresponding to a specific area which is a main subject area and is used as a reference for alignment is specified and used for alignment between images.
Based on the movement information of the imaging means that captures the plurality of images between the plurality of images and the movement vector, the moving direction of the main subject is determined, and the moving direction of the main subject is determined for each divided region. 4. The image processing apparatus according to claim 1, wherein a movement vector to be used for alignment between the images is specified by assigning a corresponding weight. 5.   5. The image according to claim 3, wherein movement information of an imaging unit that captures the plurality of images between the plurality of images is at least one of angular velocity, acceleration, and displacement of the imaging unit. Processing equipment. Subject detection means for detecting a specific subject in the plurality of images;
6. The alignment unit according to claim 1, wherein the alignment unit specifies a movement vector of an area corresponding to the specific subject detected by the subject detection unit as a movement vector used for alignment between the images. The image processing apparatus according to any one of the above. The alignment means includes
The image processing apparatus according to claim 1, wherein a movement vector of an area corresponding to an in-focus position in the plurality of images is specified as a movement vector used for alignment between the images. .   8. The image processing apparatus according to claim 1, further comprising a combining unit that performs a blurring process on a background area of the plurality of images and combines the plurality of images that have been aligned in the main subject region by the alignment unit. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 8, wherein the synthesizing unit performs a blurring process on a background area of the plurality of images with a blurring amount corresponding to a movement amount of a background between images. A calculation step of calculating a movement vector for each divided region in at least three or more images captured by the imaging means;
An alignment step of aligning the plurality of images based on a movement vector calculated by the calculation means;
A counting step for counting the number of movement vectors used in the alignment means for each of the divided areas;
The image processing method characterized in that, in the alignment step, a movement vector to be used for alignment between images is specified from the movement vector calculated by the calculation means using the number of times counted by the counting means.   A computer-executable program in which the procedure of the image processing method according to claim 10 is described.   A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 10.

Images