JP2011176777A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a high-quality image without causing blur or artifact in image synthesis. <P>SOLUTION: The image processing apparatus includes: a registeration means for registering a plurality of images; and an image synthesis means for synthesizing the plurality of registered images to generate a composite image. The registeration means includes: a region separating means for separating each reference image into a plurality of regions by partitioning the image to be registered at a predetermined boundary line when the reference image is any one of the plurality of images and the other images are images to be registered; and a region changing means for changing the region of each image to be registered by moving the boundary line so that the regions may be different between the images to be registered. The image to be registered is registered on the reference image for each of the regions. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method for performing alignment processing and composition processing of a plurality of images taken in time series.

  When an image capturing apparatus such as a digital camera or a digital video camera captures an image, if the image capturing apparatus moves due to camera shake, there is a problem that the captured image is blurred and unclear. For this reason, the imaging apparatus captures a plurality of temporally continuous images, detects a motion between the images by taking a correlation between the plurality of images, and based on the motion vector, A so-called electronic camera shake correction process is known in which image degradation due to camera shake is improved by matching and matching corresponding points between images. As an example of such a camera shake correction process, Patent Document 1 discloses that a plurality of images are acquired with a short exposure time so that camera shake does not occur, and the relative positions of the images are set so as to cancel the locus of camera shake. A technique for obtaining a blur-free image with proper exposure by synthesizing while shifting is disclosed.

  In Patent Document 2, the most probable motion vector (global motion vector) is specified and aligned from a plurality of motion vectors obtained by performing pattern matching from a plurality of regions between images. Technology is disclosed. Furthermore, Patent Document 3 discloses a technique for performing alignment by applying a motion obtained for each region without specifying a global motion vector for each corresponding region.

JP-A-9-261526 JP 2008-005084 A JP-A-8-110939

  However, according to the technique disclosed in Patent Document 2 described above, high-accuracy blur correction can be performed in consideration of the rotational movement blur component in addition to the translational blur component. Since correction is performed by applying a motion vector, for example, when camera shake having a degree of freedom other than parallel movement and rotational movement occurs, or when there is a movement of a subject, correct alignment cannot be performed. Therefore, when the aligned images are synthesized, there is a problem that blur occurs in a region having a motion that does not match the global motion vector on the image, and the resolution is significantly impaired.

  According to the technique disclosed in Patent Document 3, since an image is divided into regions and alignment is performed using motion vectors obtained for each divided region, the image quality in each divided region is greatly improved. When the motion vectors of adjacent divided regions are slightly different, there is a problem that block noise due to region boundary mismatching is noticeable and the image quality of the entire image is significantly reduced.

  The present invention has been made to solve the above problem, and an object of the present invention is to provide an image processing apparatus capable of generating a high-quality image free from blurring and unpleasant artifacts during synthesis.

In order to solve the above problems, the present invention employs the following means.
The present invention is an image processing apparatus comprising: an alignment unit that performs alignment between a plurality of images; and an image synthesis unit that combines the plurality of aligned images to generate a combined image, An alignment unit that divides the reference image into a plurality of regions by dividing the reference image by a predetermined boundary line when any one of the plurality of images is a standard image and another image is a reference image. Division means, and area changing means for changing the area of each reference image by moving the boundary line so that the areas differ between the reference images, and for each area, the reference image An image processing apparatus is provided that performs alignment of the reference image to be aligned.

  In the above-described image processing apparatus, it is preferable that the area dividing unit makes the area a rectangular area.

  Furthermore, in the above-described image processing apparatus, it is preferable that the area changing unit changes the area by translating the boundary line.

  The present invention is an image processing method for performing alignment between a plurality of images and generating a composite image by combining the plurality of aligned images, and at the time of the alignment, any one of the plurality of images When one of the reference images is a reference image and the other image is a reference image, the reference image is divided into a plurality of regions by dividing the reference image by a predetermined boundary line, and the regions differ between the reference images. Moving the boundary line to change the region of each reference image, and aligning the reference image to be aligned with the reference image for each region. An image processing method is provided.

  Thus, according to the present invention, it is possible to generate a high-quality image that does not cause blurring or unpleasant artifacts during synthesis.

1 is a block diagram illustrating a schematic configuration of a digital camera when an image processing apparatus according to an embodiment of the present invention is applied. It is explanatory drawing which shows an example when the image is expand | deployed to the frame memory in the image processing apparatus concerning one Embodiment of this invention. It is explanatory drawing which shows an example when the image is expand | deployed to the frame memory in the image processing apparatus concerning one Embodiment of this invention. It is a block diagram which shows schematic structure of the image position alignment part of the image processing apparatus concerning one Embodiment of this invention. FIG. 5 is an explanatory diagram illustrating a procedure for area division with respect to a reference image in the image processing apparatus according to the embodiment of the present invention. It is explanatory drawing which shows the mode of block matching with the reference | standard image and reference image by which area | region division | segmentation were performed in the image processing apparatus concerning one Embodiment of this invention. It is a block diagram which shows schematic structure of the image synthetic | combination part of the image processing apparatus concerning one Embodiment of this invention. In the image processing apparatus according to an embodiment of the present invention, it is an explanatory diagram showing the position of the boundary line of a region when a reference image and a reference image that is divided into blocks and subjected to block matching are sequentially synthesized. It is explanatory drawing which shows a mode that the reference | standard image and reference image by which area | region division | segmentation and block matching were carried out are each synthesize | combined in the image processing apparatus concerning one Embodiment of this invention. It is a flowchart which shows the process of the image processing concerning the image processing apparatus concerning one Embodiment of this invention.

An embodiment of an image processing apparatus according to the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram illustrating a schematic configuration of a digital camera including an image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment can be applied to, for example, a digital camera as shown in FIG. That is, in the digital camera, after performing predetermined image processing on a plurality of images that are continuous in time series by the imaging device, the image processing apparatus according to the present embodiment performs alignment processing between the plurality of images, These multiple images are combined to generate a combined image.

  As shown in FIG. 1, the digital camera described in this embodiment includes an imaging unit 100, an A / D conversion unit 101, an image processing unit 102, a frame memory 103, an image alignment unit 104, an image composition unit 105, a selector 106, An image display unit 107 and an image compression unit 108 are provided.

  The imaging unit 100 includes an optical lens unit and an imaging element. The optical lens unit forms an image of light emitted or reflected from the subject on the image sensor, and may be configured by combining a diaphragm, an optical low-pass filter, a mechanical shutter, and the like as necessary. The image sensor converts an image formed on a plurality of photoelectric conversion elements arranged on a two-dimensional plane into an electric signal, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor). An image sensor or the like can be used. If a predetermined color filter array is installed on the image sensor, different color information can be acquired for each photoelectric conversion element, and a color image can be formed by the image processing unit 102 described later.

  An A / D (Analog / Digital) conversion unit 101 converts an analog image signal converted into a voltage signal by the imaging unit 100 into a digital image signal. The image processing unit 102 processes the image signal supplied from the A / D conversion unit 102 to generate one piece of optimized image data. Here, for example, noise suppression processing, edge enhancement processing, scaling processing, and the like are included. When the color filter array as described above is applied, a multi-channel color signal can be obtained by demosaicing processing. In the case of handling a color image, white balance adjustment processing, YC separation processing, and the like are required in addition to the demosaicing processing described above. A plurality of images are acquired and acquired by repeating image capturing by the imaging unit 100, A / D conversion of the image signal by the A / D conversion unit 101, and predetermined image processing by the image processing unit 102 a plurality of times. The processed image is stored in the frame memory 103.

  The frame memory 103 stores image data that has been subjected to predetermined processing by the image processing unit 102 as described above. Specifically, image data is written in the frame memory 103 as shown in FIG. In the example of FIG. 2, if the number of acquired images is n, one standard image and n−1 reference images are developed on the frame memory. In the present embodiment, it is assumed that an image acquired immediately after the user presses the shutter button is a standard image, an image acquired thereafter is a reference image, and the reference image is aligned with the standard image. The standard image and the reference image can be arbitrarily determined. In other words, the reference image serving as the alignment reference may be any of a plurality of acquired images, and may be designated by the user.

  The frame memory 103 also stores composite image data generated by combining the base image and the reference image as will be described later. When there are a plurality of reference images, image composition processing in the image composition unit 105 described later is repeated, and the resultant composite image is stored in the composite image storage area. That is, when the reference image is the first reference image, a composite image obtained by combining the base image and the reference image is stored. When the reference image is the second or later reference image, the first image is stored. The combined image obtained by combining the combined image and the reference image is sequentially stored using the combined image combined with the reference image as a standard image. Therefore, the composite image data is written in the frame memory 103 as shown in FIG. As the frame memory 103, a dynamic random access memory (DRAM) or a synchronous DRAM (SDRAM) can be used. However, the frame memory 103 is not limited to this as long as it can temporarily store information serving as image data.

  When the image alignment unit 104 determines any one of a plurality of images stored in the frame memory 103 as a reference image and an image other than the reference image as a reference image, the image alignment unit 104 The reference image is compared with the reference image, and the reference image is aligned with the reference image. For this reason, the alignment unit 104 includes a region division setting unit 110, a region division change unit 111, a motion estimation unit 112, and a motion information holding unit 113, as shown in FIG.

  When performing alignment, first, the area division setting unit 110 reads one reference image from a plurality of images stored in the frame memory 103, and divides the reference image by a predetermined boundary line to thereby define a plurality of reference images. Divide into areas. Here, the shape of the area to be divided is not limited, but rectangular block division is desirable from the viewpoint of circuit scale reduction and speedup. In the following, the present embodiment will be described on the assumption that block division is performed.

The area division change unit 111 reads out other reference images stored in the frame memory 103, and sets the areas of the other reference images so that the areas differ between the reference images. In other words, with reference to the boundary line in the reference image area divided by the area division setting unit 110, the boundary line is set in parallel by moving the boundary line between the reference images. Change the area so that each area is different. Specifically, for example, assuming that the horizontal block size is Bx, the vertical block size is By, and the number of used images is n, the horizontal movement amount Sx (i) and the vertical shift with respect to the i-th reference image The moving amount Sy (i) in the direction is expressed by the following equation.
Here, int () is a function that returns the integer form of the number in parentheses.

  For example, when the number of used images is four, that is, when there are four images stored in the frame memory 103, first, as shown in FIG. = 1) the reference image region is divided, and then the region division changing unit 111 moves the boundary line of the first reference image region to move the second (i = 2) reference image region. Set. In addition, for the third reference image (i = 3), the first and second images are determined by setting the boundary line positions different from the boundary line positions of the first and second regions. An area different from the reference image area is set. The area can be changed by changing the size of the area, for example, by changing the pitch of the boundary line.

  The motion estimation unit 112 compares the reference image divided by the region dividing unit 110 and the reference image whose region has been changed by the region division changing unit 111 with the standard image, and each reference image has been compared with the standard image for each region. Estimate the movement of As a motion estimation method, a block matching method can be applied. FIG. 6 conceptually shows this state. The estimated motion vector (Mvx, Mvy) is stored in the motion information holding unit 113 as motion information for each area of each reference image with respect to the base image, and is output as motion information 200 to the image composition unit 105 during image composition processing. . In this embodiment, the motion to be estimated is only the translation component, but a rotation angle may be added thereto, or an affine transformation parameter with a higher degree of freedom may be estimated for each region.

  The image composition unit 105 includes a reference image and any reference image among a plurality of images stored in the frame memory 103, and the movement of the reference image with respect to the reference image supplied from the image registration unit 104 in synchronization therewith. Based on the information, the reference image is synthesized with the standard image. When a composite image obtained by combining the base image and the reference image is already stored in the frame memory 103, the reference image is combined with the composite image based on the composite image.

  Therefore, the image composition unit 105 includes a selector 120, a coordinate conversion unit 121, and an adder 122 as shown in FIG. The selector 120 selects which image is read out from the images stored in the frame memory 103. That is, in the image composition unit 105, first, a reference image is read from the frame memory 103. When this reference image is the first reference image, that is, when the first composition is performed, it is synchronized with the reference image. Then, the reference image is read from the frame memory 103, and when the composition is the second or later, the composite image is read instead of the reference image.

  The coordinate conversion unit 121 is supplied with motion information 200 from the motion information holding unit 113 of the image alignment unit 104 in synchronization with the reference image, and the coordinate conversion unit 121 coordinates the reference image based on the motion information. Perform conversion. In this embodiment, since the motion component for alignment is only translated, the reference image is translated based on the motion vector (Mvx, Mvy). That is, the coordinate conversion unit 121 aligns the reference image with the standard image by translating the reference image based on the motion vector. The adder 122 generates a composite image by adding the coordinate-converted and aligned reference image to the standard image or the composite image. The generated composite image is stored in the composite image storage area of the frame memory 103.

  FIG. 8 shows a region boundary on the composite image when the number of used images is four, that is, when four images are stored in the frame memory 103, for example. As shown in FIG. 8, since the area boundaries at the time of alignment are shifted between the reference images, the boundary between the areas becomes less noticeable as the number of combined images increases. Further, FIG. 9 is a conceptual diagram showing how the composite image is updated with respect to the alignment by the block matching method described in FIG.

  Returning to FIG. 1, the selector 106 switches between a composite image by the image alignment unit 104 and the image composition unit 106 and a non-composite image by the image processing unit 102 according to a control signal from a control mechanism (not shown). The image display unit 107 is configured by, for example, an LCD (Liquid Crystal Display) or the like, and allows the user to check the composition at the time of imaging or to reproduce the captured image (the path at the time of reproduction is not shown). . The image compression unit 108 generates a code obtained by compressing captured image data. The JPEG (Joint Photographic Experts Group) method is common today as a compression method for still images. When handling moving images, the MPEG (Moving Picture Experts Group) method or the H.264 method recommended by the ITU-T (International Telecommunication Union Telecommunication Standardization Sector) can be used. The compressed and encoded image data is stored in an external storage device (not shown).

Next, image processing in the image processing apparatus according to the present embodiment configured as described above will be described with reference to the flowchart of FIG.
As shown in FIG. 10, an image is picked up by the image pickup unit 100 in step S100, and converted into a digital signal by the A / D conversion unit 101 in step S101. It is desirable that the image acquired here has a sufficiently high shutter speed so that camera shake does not occur. In step S102, the image processing unit 102 performs predetermined image processing such as noise suppression processing, edge enhancement processing, and scaling processing on the image captured in step S100 and step S101 and A / D converted. In step S 103, the processed image data is stored in the frame memory 103. The above processing is repeated for a required number of predetermined images. By repeating this process, image data is written in the frame memory 103 as shown in FIG.

  In step S104, it is determined whether or not the above processing has been completed for a predetermined required number of sheets. If the process for the required number of sheets has not been completed, the process returns to step S100, and the above process is repeated. If the process for the required number of sheets has been completed, the process proceeds to step S105.

  In the next step S105, the image alignment unit 104 reads the standard image from the frame memory 103, and similarly reads the reference image from the frame memory 103 in step S106. In step S107, the reference image is aligned with the read standard image. As described above, as described above, the image alignment unit 104 divides the first reference image into a plurality of regions, and the other reference images have different regions so that the regions differ between the reference images. Is changed, pattern matching is performed for each area of each reference image, and the motion between the base image and the reference image is estimated.

  In the next step S108, the image composition unit 105 determines whether the reference image is the first reference image or the second and subsequent reference images. If it is the first reference image, the reference image is referred to. After aligning the images, a composite image is generated, and the process proceeds to step S109. In step S109, the generated composite image is newly written in an area for storing the composite image on the frame memory 103. As a result of the determination in step S108, if it is the second and subsequent reference images, the process proceeds to step S110, and the combined image up to the previous frame is read. In step S <b> 111, a reference image is aligned with the read composite image to generate a composite image, and the generated composite image is stored in the composite image area of the frame memory 103. With the above processing, data as shown in FIG. 3 is written in the frame memory 103.

  In step S112, it is determined whether or not the synthesis process for the last reference image has been completed. If the target of the previous synthesis process is not a reference image, the process returns to step S105, and the above-described reference image is described above. Repeat the process. If the result of determination in step S112 is that the target of the composition process is the last reference image, that is, if it is determined that the composition processing for all the acquired images has been completed, the process proceeds to step S113 and the composite image is selected. The final output image is generated by dividing by the number of sheets used, and this routine ends. Note that if the number of acquired images is limited to a power of 2, the division in step S113 can be realized only by a simple bit shift, which is desirable from the viewpoint of circuit scale reduction and speedup.

  Here, FIG. 1 submitted on the same date as the present specification shows an image generated by the conventional method. In other words, the most probable motion vector (global motion vector) is obtained from a plurality of motion vectors obtained by dividing the reference image into a plurality of predetermined regions and performing pattern matching on the divided regions. ) Is specified and is aligned with the reference image. Reference FIG. 1A is an image obtained by performing alignment by such a conventional method, and reference FIG. 1B shows a boundary of divided areas on the image of reference FIG. 1A. Is. In the reference FIG. 1B, a broken line A100 indicates the range of the image shown in the reference FIG. 1A, and a solid line A101 indicates the boundary of the divided areas at the time of alignment. Looking at the image in FIG. 1A, it can be seen that a block artifact is generated at the position of the solid line A101 shown in FIG.

  On the other hand, FIG. 2 shows a composite image generated by the image processing method of this embodiment. The reference information 2 (a) is an image synthesized by alignment by the image processing method of the present invention. Reference FIG. 2 (b) shows the boundary of the divided areas on the image of reference FIG. 2 (a), and the broken line A102 shows the range of the image shown in FIG. 2 (a). Reference FIG. 2A is an example in which five images are combined, and the black solid line A103 in Reference FIG. 2B is a boundary between divided areas when the base image and the first reference image are aligned. Therefore, it is at the same position as A101 in FIG. 1 (b). Further, the red solid line A104, the blue solid line A105, and the green solid line A106 in the reference FIG. 2B are divided areas when the reference image and the second, third, and fourth reference images are aligned, respectively. Indicates the boundary. As shown in FIG. 2 (a), it can be seen that block artifacts due to the mismatch of the region boundaries are greatly reduced compared to the image of FIG. 1 (a), and the image quality is improved.

  Thus, according to the present image processing device, according to the image processing device according to the present embodiment, when aligning the reference image with respect to the standard image, the boundary line between the regions divided between the plurality of reference images Are different from each other, so that the boundary of the region becomes inconspicuous at the time of image composition. Further, noise can be suppressed by synthesizing the aligned images, and a high-quality image can be obtained. Furthermore, when the image processing apparatus according to the present embodiment is applied to electronic camera shake correction, it is possible to perform robust positioning particularly for a moving subject, and to reduce image quality degradation due to block artifacts. .

  The image processing apparatus according to the present invention can be applied not only to an imaging apparatus such as a digital camera and a digital video camera as described above, but also to a general-purpose or dedicated computer and a program that runs on the computer. be able to. Such a computer includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and a program for realizing all or part of the above processing by the CPU or the like. By reading the program recorded on the recorded computer-readable recording medium, developing the program in a ROM, RAM, etc., and executing information processing / calculation processing, the above-described performance evaluation processing of the mechanical device is performed. make it happen.

  Accordingly, each of the processing units described above is realized by the CPU developing a processing program stored in a predetermined ROM in the RAM and executing the expanded program. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, and the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

DESCRIPTION OF SYMBOLS 100 Image pick-up part 101 A / D conversion part 102 Image processing part 103 Frame memory 104 Image registration part 105 Image composition part 106 Selector 107 Image display part 108 Image compression part 110 Area division setting part 111 Area division change part 112 Motion estimation part 113 Motion information holding unit 120 Selector 121 Coordinate conversion unit 122 Adder 200 Motion information

Claims (4)

An image processing apparatus comprising: an alignment unit that performs alignment between a plurality of images; and an image synthesis unit that generates a composite image by combining the plurality of aligned images.
The alignment means includes
A region dividing unit that divides the reference image into a plurality of regions by dividing the reference image by a predetermined boundary line when any one of the plurality of images is a reference image and the other image is a reference image;
An area changing unit that changes the area of each reference image by moving the boundary line so that the areas differ between the reference images; and
An image processing apparatus that performs alignment of the reference image to be aligned with respect to the reference image for each region.   The image processing apparatus according to claim 1, wherein the area dividing unit makes the area a rectangular area.   The image processing apparatus according to claim 1, wherein the area changing unit changes the area by moving the boundary line in parallel. An image processing method for performing alignment between a plurality of images and synthesizing the plurality of aligned images to generate a combined image,
In the alignment,
Dividing any one of the plurality of images into a plurality of regions by dividing the reference image by a predetermined boundary line when the other image is a reference image and the other image is a reference image;
Changing the region of each reference image by moving the boundary line so that the region is different between the reference images, and
An image processing method comprising: aligning the reference image to be aligned with the reference image for each region.

Images