JP4713430B2 - Imaging apparatus and image composition method - Google Patents

Description

  The present invention relates to an imaging apparatus and an image composition method that perform camera shake correction, for example.

  Conventionally, a technique is known in which a plurality of images are continuously photographed with a short exposure time, and an image with less camera shake is obtained by combining the photographed images (for example, Patent Documents). 1). FIG. 7 is a flowchart showing a conventional technique for obtaining an image with little camera shake. As shown in FIG. 7, the imaging apparatus performs color photographing by starting photographing (step ST11), and stores the photographed color photographing data in the image storage unit (step ST12). Then, the imaging apparatus determines whether or not the designated number of images has been obtained (step ST13). If the designated number of images has not been obtained, the processes of steps ST11 to ST13 are repeated. On the other hand, when the designated number of images is obtained in step ST13, the imaging apparatus calculates the positional deviation between the plurality of images stored in the image storage unit and corrects the calculated positional deviation. (Step ST14), and the image is synthesized (step ST15).

  For example, when four color images are taken, the taken four color images are stored in the image storage unit in the form shown in FIG. FIG. 8 shows an image stored in the image storage unit 50 of the imaging apparatus. The image storage unit 50 includes an image 51 including the luminance signal 1 and the color signal 1, an image 52 including the luminance signal 2 and the color signal 2, an image 53 including the luminance signal 3 and the color signal 3, and a luminance signal. 4 and the color signal 4 are stored. Then, the image composition unit of the imaging apparatus synthesizes the images 51 to 54 as shown in FIG.

In addition, when a plurality of images are captured and combined, it is known that higher sensitivity can be achieved as the number of images to be combined increases (for example, Patent Document 2).
Japanese Patent No. 3405404 Japanese Patent No. 3349129

  However, the conventional apparatus has a problem that the capacity of the memory necessary for storing the captured images increases as the number of captured images increases. In the conventional apparatus, if the number of images to be captured is reduced, the S / N ratio of the combined image is reduced, and the exposure time of each image is increased and combined in order to prevent a decrease in the S / N ratio. When the number of images is reduced, there is a problem that the influence of the amount of camera shake for each image increases. That is, the conventional apparatus has a problem that a large amount of memory is required to improve the quality of the visual image, and the cost is increased and the mounting area is increased.

  The present invention has been made in view of the above point, and by deleting the color signal of each image, the quality of the visual image is suppressed while suppressing an increase in memory capacity, an increase in cost, and an increase in mounting area. An object of the present invention is to provide an imaging apparatus and an image composition method capable of improving the above.

  An image pickup apparatus according to the present invention includes an image pickup unit that picks up an image, a color signal deletion unit that deletes a color signal from image data of a part of a plurality of the same images repeatedly picked up by the image pickup unit, Image storage means for sequentially storing image data of the plurality of identical images composed of the part of the number of images from which the signal has been deleted and an image from which the color signal has not been deleted, and the color signal stored in the image storage means being deleted Color signal complementing means for complementing the color signal of the image data of the partial number of images from which the color signal stored in the image storage means is deleted based on the color signal of the image data of the image that has not been processed, and the color An image synthesizing unit that synthesizes the plurality of identical images after the color signal is complemented by the signal complementing unit is employed.

  The image composition method of the present invention includes a step of deleting a color signal from image data of a part of a plurality of the same images repeatedly taken, and the part of the number of images from which the color signal has been deleted. The step of sequentially storing the image data of the plurality of the same images composed of images from which the color signal is not deleted, and the stored color signal based on the color signal of the image data of the image from which the stored color signal is not deleted A step of complementing the color signal of the image data of the partial number of images from which the image is deleted, and a step of synthesizing the plurality of the same images after complementing the color signal.

  According to the present invention, by deleting the color signal of each image, it is possible to improve the quality of the visual image while suppressing an increase in memory capacity, an increase in cost, and an increase in mounting area.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an imaging apparatus 100 according to Embodiment 1 of the present invention. The imaging device 100 can be applied to a communication terminal device such as a mobile phone or a digital camera, for example.

  The timing control unit 102, the luminance / color signal transfer unit 103, and the color signal deletion unit 104 constitute an image data processing unit 110. Further, the positional deviation correction unit 106, the color signal complementing unit 107, and the luminance / color signal combining unit 108 constitute an image combining unit 111. The image data processing unit 110 and the image composition unit 111 constitute an image processing unit 112. The image data processing unit 110 is responsible for deleting the memory usage of the image storage unit 105, and the image composition unit 111 is responsible for synthesizing an image by complementing the deleted color signal.

  The imaging unit 101 includes a two-dimensional CCD, a MOS sensor, or the like, and by imaging using a photographing lens, the captured image is input as an optical signal, and the input optical signal is converted into an electrical signal. Then, the imaging unit 101 outputs the converted electrical signal to the luminance / color signal transfer unit 103 as image data. In addition, the imaging unit 101 repeatedly captures a plurality of the same images within a predetermined time, and outputs the captured image data of the same images to the luminance / color signal transfer unit 103. Here, the image data includes a luminance signal and a color signal, and is a YUV signal or an RGB signal. In the YUV signal, the Y signal is a luminance signal and the UV signal is a color signal. In the RGB signal, since the G signal has a large contribution to the luminance, the G signal is a luminance signal and the RB signal is a color signal. It is.

  The timing control unit 102 stores in advance the number of images from which a color signal is deleted among a plurality of the same images, and deletes the color signal to the luminance / color signal transfer unit 103. When the image data of the image to be input is input to the luminance / color signal transfer unit 103, the output of the image data to the color signal deletion unit 104 is instructed. The image from which the color signal is deleted can be arbitrarily selected, and the timing control unit 102 stores in advance an image from which the arbitrarily selected color signal is deleted.

  The luminance / color signal transfer unit 103 outputs, to the color signal deletion unit 104, image data of an image from which the color signal is input, which is input from the imaging unit 101, in accordance with an instruction from the timing control unit 102. Then, the luminance / color signal transfer unit 103 outputs the image data of the image from which the color signal is deleted, which is input from the color signal deletion unit 104, to the image storage unit 105. In addition, the luminance / color signal transfer unit 103 outputs the image data of the image input from the imaging unit 101 without deleting the color signal to the image storage unit 105.

  The color signal deleting unit 104 deletes the color signal from the image data of the image input from the luminance / color signal transfer unit 103 and outputs the image data after the color signal is deleted to the luminance / color signal transfer unit 103. For example, the color signal deletion unit 104 deletes a color signal, in the case of a YUV signal, a method of deleting all of the UV signal to form a black and white image, a method of deleting only the U signal, a bit of the UV signal A method of reducing the number, a method of thinning out colors spatially and changing YUV = 4: 4: 4 to YUV = 4: 4: 2 or YUV = 4: 2: 0, and a color signal combining these methods It is possible to use a method for differentiating each image to be deleted. Further, the color signal deletion unit 104 can use a method of deleting an RB signal in the case of an RGB signal as a method of deleting a color signal.

  The image storage unit 105 sequentially stores the image data input from the luminance / color signal transfer unit 103. Then, the image storage unit 105 outputs a plurality of stored image data of the same image to the misalignment correction unit 106 when the stored image data of the same image reaches a predetermined number. For example, when the stored image data of the same image reaches four images, the image storage unit 105 outputs the stored four image data of the same image to the misalignment correction unit 106. . As a result, the image storage unit 105 stores a plurality of image data of the same image, but the image data of the plurality of the same image stored is an image data of an image from which the color signal is not deleted and an image from which the color signal is deleted. Both of the image data.

  The misregistration correction unit 106 corrects the misregistration of the image using the luminance signals of the plurality of image data of the same image input from the image storage unit 105. Specifically, the misregistration correction unit 106 determines one image serving as a reference for the amount of deviation, and obtains the amount of deviation of each image with respect to that image, thereby determining the position of the pixel of the reference image and the pixel of each image. Associate. Then, the misregistration correction unit 106 outputs the image data of the image whose misregistration is corrected to the color signal complementing unit 107.

  The color signal complementing unit 107 is based on the color signal of the image from which the color signal has not been deleted when there is an image from which the color signal has been deleted among the plurality of the same images input from the positional deviation correction unit 106. Thus, the color signal of the image from which the color signal has been deleted is complemented. Then, the color signal complementing unit 107 outputs the image data of the image complemented with the color signal to the luminance / color signal synthesis unit 108. Further, the color signal complementing unit 107 outputs the image data of the image from which the color signal has not been deleted to the luminance / color signal combining unit 108 as it is. For example, when there are a plurality of images whose color signals are not deleted, the color signal complementing unit 107 averages the color signals of the images whose color signals are not deleted as a method of complementing the color signals. A method of complementing a color signal for an image from which the color signal has been deleted, a method of complementing a color signal for an image from which the color signal has been deleted using only a part of the color signal of an image from which the color signal has not been deleted, or A method of complementing the same color signal as the color signal of the image from which the color signal has not been deleted as the color signal of the image from which the color signal has been deleted can be used. If the UV signals of two images among the five color images are deleted, the UV signals of the remaining three images can be averaged to complement the image from which the UV signals have been deleted.

  A luminance / color signal combining unit 108 as image combining means performs image combining processing on the image data of a plurality of identical images input from the color signal complementing unit 107 pixel by pixel. At this time, since the luminance signals of all the image data input from the color signal complementing unit 107 to the luminance / color signal synthesizing unit 108 are not deleted, the luminance / color signal synthesizing unit 108 inputs from the color signal complementing unit 107. The luminance signals of all the image data of a plurality of the same images are synthesized. Then, the luminance / color signal synthesis unit 108 outputs the image data subjected to the image synthesis process to the image storage unit 109.

  The image storage unit 109 stores the image data after the image synthesis process input from the luminance / color signal synthesis unit 108.

  Next, the operation of the imaging apparatus 100 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the imaging apparatus 100.

  The imaging apparatus 100 performs color imaging by starting imaging, and the imaging unit 101 obtains a luminance signal and a color signal of the captured image (step ST201).

  Next, the timing control unit 102 does not give any instruction to the luminance / color signal transfer unit 103, and the luminance / color signal transfer unit 103 stores the luminance signal and color signal of the image input from the imaging unit 101 as an image. The image storage unit 105 stores the image data including the input luminance signal and color signal (step ST202).

  Next, the imaging apparatus 100 determines whether or not imaging of a predetermined number of images has been completed (step ST203).

  If imaging of a predetermined number of images has not been completed, the imaging apparatus 100 repeats the processing from step ST201 to step ST203.

  In addition, when imaging of a predetermined number of images is completed, the imaging apparatus 100 continues to perform color imaging, and the imaging unit 101 obtains a luminance signal and a color signal of the captured image (step ST204).

  Next, the timing control unit 102 instructs the luminance / color signal transfer unit 103 to delete the color signal, and the luminance / color signal transfer unit 103 receives the image data input from the imaging unit 101 in accordance with the instruction from the timing control unit 102. To the color signal deletion unit 104, and the color signal deletion unit 104 deletes the color signal from the input luminance signal and color signal (step ST205).

  For example, when the image data is a YUV signal and is in a YUV = 4: 2: 2 format, the YUV signal is transmitted from the luminance / color signal transfer unit 103 to the color signal deletion unit 104 with a data arrangement of “YUYVYUYVYUYV. Therefore, the color signal deletion unit 104 can delete the UV signal by deleting every other data in order from the second data from the top. Then, the color signal deletion unit 104 outputs only the remaining Y signal to the luminance / color signal transfer unit 103.

  Then, the color signal deletion unit 104 outputs the image data from which the color signal has been deleted to the luminance / color signal transfer unit 103, and the luminance / color signal transfer unit 103 outputs the image data from which the color signal has been deleted to the image storage unit 105. Output.

  Next, the image storage unit 105 stores the image data from which the color signal has been deleted (step ST206).

  Next, imaging apparatus 100 determines whether or not imaging of a predetermined number of images has been completed (step ST207).

  If imaging of a predetermined number of images has not been completed, the imaging apparatus 100 repeats the processing from step ST204 to step ST207.

  When the predetermined number of images have been captured, the image storage unit 105 outputs the stored image data.

  FIG. 3 shows an image stored in the image storage unit 105 when it is determined in step ST207 that imaging of a predetermined number of images has been completed. FIG. 3 shows a case where, when four identical images are combined, there is one image data from which the color signal is not deleted and three image data from which the color signal is deleted. The image data 301 from which the color signal is not deleted consists of the luminance signal 1 and the color signal 1. Further, the image data 302 from which the color signal has been deleted consists only of the luminance signal 2, the image data 303 from which the color signal has been deleted consists only of the luminance signal 3, and the image data 304 from which the color signal has been deleted includes only the luminance signal 4. Consists of. Therefore, the image storage unit 105 does not need a storage capacity for storing the color signals of the image data 302 to 304 from which the color signals are deleted, or can be used for storing other data.

  Next, the misregistration correction unit 106 corrects the misregistration of the image using the luminance signal of the plurality of image data of the same image (step ST208).

  Next, the color signal complementing unit 107 complements the color signal of the image from which the color signal has been deleted (step ST209).

  Next, the luminance / color signal combining unit 109 combines a plurality of pieces of image data of the same image (step ST210).

  FIG. 4 is a diagram illustrating a color signal complementing method in the color signal complementing unit 107 and an image synthesizing method in the luminance / color signal synthesizing unit 108. For example, when synthesizing one image from which color signals have not been deleted and three images from which color signals have been deleted, the color signal complementing unit 107 has the same color signals as one color signal 1 remaining. The color signals 1 ′, 1 ″, 1 ′ ″ are complemented as if they exist in the three images. Then, the luminance / color signal synthesis unit 108 synthesizes the luminance signals 1 to 4 of the four images. For example, when synthesizing the color signals by addition, the color signal 1 is multiplied by four. Image synthesis is performed by synthesizing color signals.

  Next, the image storage unit 109 stores the combined image data.

  As described above, according to the first embodiment, the color signal of a part of the image data is deleted, and the deleted color signal is complemented at the time of image composition, thereby increasing the memory capacity and increasing the cost. The visual image quality can be improved while suppressing the increase in the size and the mounting area. Further, according to the first embodiment, the image storage unit 105 is adjusted by adjusting the number of images output from the luminance / color signal transfer unit 103 to the color signal deletion unit 104, that is, the number of images from which the color signal is deleted. It is possible to achieve both the adjustment of the memory amount and the adjustment of the image quality such as the S / N ratio. For example, when the color signal of the image data of one of the four images is deleted, an image with a high S / N ratio can be obtained although the amount of data stored in the image storage unit 105 is large. In addition, when the color signal of the image data of three images out of the four images is deleted, the S / N ratio is lower than when the color signal of the image data of one image is deleted. The amount of data stored in the image storage unit 105 can be greatly deleted. Further, according to the first embodiment, in image synthesis, luminance signals of all captured images can be used for synthesis, so that a reduction in visual image quality can be suppressed. Further, according to the first embodiment, since the positional deviation correction unit 106 can correct the positional deviation between images using only the luminance signal, the deletion of the color signal does not affect the positional deviation correction. Even if the memory amount of the image storage unit 105 is deleted, the image quality can be further improved. In addition, according to the first embodiment, since the image misalignment is corrected, a highly sharp composite image can be obtained without performing imaging while being fixed with a tripod or the like.

(Embodiment 2)
FIG. 5 is a block diagram showing the configuration of the imaging apparatus 500 according to Embodiment 2 of the present invention. The imaging device 500 can be applied to a communication terminal device such as a mobile phone or a digital camera, for example.

  The luminance / color signal transfer unit 103 and the color signal deletion unit 104 constitute an image data processing unit 510. Further, the image data processing unit 510 and the image composition unit 111 constitute an image processing unit 512.

  As shown in FIG. 5, imaging apparatus 500 according to Embodiment 2 adds shutter button operation detection section 501 except for timing control section 102 in imaging apparatus 100 according to Embodiment 1 shown in FIG. 1. . 5, parts having the same configuration as in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The shutter button operation detection unit 501 detects that the shutter button has been operated for the start of imaging by the imaging unit 101 or the end of imaging by the imaging unit 101, and notifies the luminance / color signal transfer unit 103. , That the shutter button has been operated.

  When the luminance / color signal transfer unit 103 receives a notification from the shutter button operation detection unit 501 that the shutter button has been operated, the color signal deletion unit 104 converts the image data of a predetermined number of images input from the imaging unit 101. Output to. Then, the luminance / color signal transfer unit 103 outputs the image data of the image from which the color signal is deleted, which is input from the color signal deletion unit 104, to the image storage unit 105. Also, the luminance / color signal transfer unit 103 receives the image data of a predetermined number of images input from the imaging unit 101 when the shutter button operation detection unit 501 does not receive a notification that the shutter button has been operated. It outputs to 105 as it is.

  Next, the operation of the imaging apparatus 500 will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the imaging apparatus 500.

  Imaging apparatus 500 performs color imaging by starting imaging, and imaging section 101 obtains a luminance signal and a color signal of the captured image (step ST601).

  Next, the shutter button operation detection unit 501 determines whether or not the shutter button has been operated (step ST602).

  If it is determined that the shutter button has been operated, the luminance / color signal transfer unit 103 deletes the color signals of one or a predetermined number of images after determining that the shutter button has been operated (step ST603). Then, image storage section 105 stores image data of an image consisting only of the input luminance signal (step ST604).

  When it is determined that the shutter button is not operated, the luminance / color signal transfer unit 103 outputs the image data input from the imaging unit 101 to the image storage unit 105 as it is, and the image storage unit 105 receives the input luminance. Image data of an image composed of a signal and a color signal is stored (step ST604).

  Next, imaging apparatus 500 determines whether or not imaging of a predetermined number of images has been completed (step ST605).

  If imaging of a predetermined number of images has not been completed, imaging apparatus 500 repeats the processing from step ST601 to step ST605.

  When the predetermined number of images have been captured, the image storage unit 105 outputs the stored image data. In addition, since the operation | movement after step ST605 is the same as FIG. 2, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

  As described above, according to the second embodiment, in addition to the effects of the first embodiment, there is an influence of camera shake immediately after the shutter button is operated, such as immediately after the shutter button is pressed down and immediately after the shutter button is released. Since the color signal of the image captured immediately after detecting the operation of the shutter button is deleted, the color signal can be complemented using the color signal of an image with relatively good image quality, and the image is clearer. An image can be obtained.

  In the first embodiment and the second embodiment, the color signal is complemented after correcting the misregistration. However, the present invention is not limited to this, and the misregistration may be corrected after the color signal is complemented. In the first embodiment and the second embodiment, the memory area of the image storage unit 105 obtained by deleting the color signal may store a newly captured image, or other than image storage. It may be used as a memory area necessary for this processing.

  The imaging device and the image composition method according to the present invention are suitable for performing, for example, camera shake correction.

1 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention. The flowchart which shows operation | movement of the imaging device which concerns on Embodiment 1 of this invention. The figure which shows the image memorize | stored in the image memory | storage part which concerns on Embodiment 1 of this invention. The figure which shows the color signal complementation in the color signal complementation part which concerns on Embodiment 1 of this invention, and the synthetic | combination of the image in a brightness | luminance and color signal synthetic | combination part. FIG. 3 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 2 of the present invention. FIG. 7 is a flowchart showing the operation of the imaging apparatus according to the second embodiment of the present invention. Flow diagram showing the operation of a conventional imaging device The figure which shows the image memorize | stored in the image memory | storage part of the conventional imaging device Figure showing conventional image composition

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image pick-up apparatus 101 Image pick-up part 102 Timing control part 103 Luminance / color signal transfer part 104 Color signal deletion part 105 Image memory | storage part 106 Position shift correction part 107 Color signal complement part 108 Brightness / color signal synthesis part 109 Image preservation | save part

Claims (10)

Imaging means for imaging;
A color signal deleting unit that deletes a color signal from the image data of a part of the same number of images repeatedly captured by the imaging unit;
Image storage means for sequentially storing image data of the plurality of the same images composed of the part of the number of images from which the color signals have been deleted and an image from which the color signals have not been deleted;
The color signal of the image data of the partial number of images from which the color signal stored in the image storage unit is deleted based on the color signal of the image data of the image from which the color signal stored in the image storage unit is not deleted Color signal complementing means for complementing,
Image synthesizing means for synthesizing the plurality of identical images after complementing the color signal by the color signal complementing means;
An imaging apparatus comprising:   The color signal complementing means complements a color signal by averaging color signals of image data of an image from which the color signal has not been deleted to obtain color signals of image data of the partial number of images. The imaging apparatus according to 1.   2. The color signal complementing means complements color signals by making the color signals of the number of images from which the color signals have been deleted the same as the color signals of the images from which the color signals have not been deleted. The imaging device described.   The imaging apparatus according to claim 1, wherein the color signal deleting unit deletes a UV signal of a YUV signal as the color signal.   The imaging apparatus according to claim 1, wherein the color signal deleting unit deletes an RB signal of an RGB signal as the color signal.   6. The imaging according to claim 1, wherein the color signal deleting unit deletes color signals of image data of the partial number of images arbitrarily selected from the plurality of the same images. apparatus.   The imaging apparatus according to claim 1, wherein the color signal deleting unit deletes color signals of image data of the partial number of images captured immediately after operating a shutter button. A misalignment correction unit that corrects misalignment between images captured by the imaging unit based on luminance signals of image data of a plurality of identical images captured by the imaging unit;
The imaging apparatus according to claim 1, wherein the image synthesis unit synthesizes the plurality of the same images whose positional deviation is corrected by the positional deviation correction unit.   The communication terminal device which comprises the imaging device in any one of Claims 1-8. Deleting the color signal from the image data of a part of the plurality of images of the same image repeatedly captured;
Sequentially storing the image data of the plurality of the same image consisting of the part of the number of images from which the color signal has been deleted and the image from which the color signal has not been deleted;
Complementing the color signal of the image data of the part of the number of images from which the stored color signal has been deleted, based on the color signal of the image data of the image from which the stored color signal has not been deleted;
Synthesizing the plurality of identical images after complementing the color signal;
An image composition method comprising:

Images