JP2008270982A - Image processing device and method, and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device, an imaging device, and an image processing method which can minimize operability deterioration without increasing the number of combinations more than necessary in combining a plurality of images and through image quality improvement. <P>SOLUTION: When an SN ratio (Signal to Noise ratio) of an image and brightness are improved by a combination of a plurality of images, whether or not brightness (brightness value) of combined images has reached a predetermined level is determined. By determining whether to continue or end combined image acquisition and combination processing, extra combination processing can be avoided. Thus, both image improvement and operability (reduction of a processing time) can be realized. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method, and more particularly, to an image processing apparatus, an imaging apparatus, and an image processing method that minimize improvement in image quality and deterioration in operability.

  Conventionally, the demand for electronic still cameras and camera-equipped mobile phones has been steadily increasing as portable imaging devices equipped with small camera modules. Although user needs vary, improving image quality during low-light shooting is also an important issue. In addition to increasing the sensitivity of the camera itself, measures for improving the image quality during low-illumination shooting include electrical gain-up, long exposure, and improved sensitivity through image composition.

  Gain-up makes it possible to increase the brightness easily and to keep the shutter speed fast, so that it has good effect of following a subject with large movements and is less likely to cause camera shake. On the other hand, since noise is also amplified at the same time, the amount of increase in noise is more conspicuous when the amount of image signal is small, such as at low illuminance, and as a result, the image quality is often impaired.

  Long exposure is a measure to improve sensitivity by slowing down the shutter speed and performing exposure over several frames. Since the circuit gain can be kept low, it is possible to shoot with an excellent SN (Signal to Noise Ratio) without amplifying the noise. On the other hand, since the shutter speed is slow, there is a drawback in that subject tracking is poor and camera shake is likely to occur.

  Image synthesis is a technique for improving brightness by continuously shooting a plurality of subjects under the same conditions and superimposing and synthesizing these images in subsequent processing. Since the shutter speed can be kept fast and the gain can be lowered, there is an advantage that subject tracking is good and SN is excellent. The SN increases as the number of combined images increases. However, there is a drawback in that it takes time for the image combining processing and the photographing time becomes longer, which gives stress to the user.

There has been proposed a technique for preventing deterioration of image quality due to an inadvertent operation by a user using an image synthesis method (see, for example, Patent Document 1). FIG. 5 shows a configuration diagram of the prior art, and FIG. 6 shows an operation flowchart of the prior art. Such a conventional imaging apparatus includes an imaging start command input unit 501, an imaging unit 502 that performs imaging based on an imaging start command input to the imaging start command input unit 501, and image information obtained by the imaging unit 502. Image processing means 503 for processing the image, and signal generation means 504 for notifying the user of the end of shooting by the shooting means 503 (see FIGS. 5 and 6), and by giving the user an end signal for long exposure shooting. In addition to reducing the burden of holding the camera more than necessary for the user, the image quality is prevented from being deteriorated due to the user's careless operation. Further, even when a plurality of images are combined, the same effect as described above can be obtained by notifying the user of the start and end of shooting.
JP 2006-157342 A

  However, with the above-described technology, the burden of holding the camera more than necessary is reduced by notifying the user of the start and end of the exposure at the time of long exposure or multiple image synthesis, and inadvertent during shooting Prevents image quality degradation due to simple operation. Here, in recent years, the user's requirements for camera performance and functions have become strict and multifaceted. Although user needs vary, generally speaking, it can be said that the image quality is improved and the operability is improved. However, the conventional techniques as described above cannot improve the operability while improving the image quality when combining a plurality of images.

  The present invention has been made in view of such a situation, an image processing apparatus that minimizes deterioration in operability while improving image quality without increasing the number of images to be combined more than necessary when combining a plurality of images. An object is to provide an imaging apparatus and an image processing method.

  According to the first aspect of the present invention, there is provided a luminance measuring means for measuring the luminance level of the image data, a judging means for judging whether the measured luminance level has reached a predetermined luminance level, and the judging means for the image data. The image data acquisition means for acquiring new image data, and a synthesis means for combining the acquired new image data with the image data. The luminance measuring unit measures a luminance level of the synthesized image data synthesized by the synthesizing unit, the judging unit judges whether the luminance level of the synthesized image data has reached the predetermined luminance level, and The image processing apparatus is characterized in that the acquisition of the new image data and the synthesis are repeated until it is determined that the luminance level has been reached.

  According to a second aspect of the present invention, in the apparatus according to the first aspect, the synthesizing unit adjusts the position so that the characteristic portions extracted from the image data before the synthesis overlap, and synthesizes the acquired image data. And

  According to a third aspect of the present invention, there is provided an image pickup apparatus comprising the image processing apparatus according to the first or second aspect, wherein the image data acquisition means acquires by photographing.

  According to a fourth aspect of the present invention, there is provided a luminance measuring means for measuring the luminance level of the image data, a judging means for judging whether the measured luminance has reached a predetermined luminance level, and the judging means for obtaining the image data. If it is determined that the predetermined luminance level has not been reached, an image data acquisition unit that acquires new image data, a determination unit that determines the number of new image data acquired by the image data acquisition unit, and the acquisition An image processing apparatus comprising: a combining unit configured to combine at least one new image data and the image data.

  According to a fifth aspect of the present invention, in the apparatus according to the fourth aspect, the synthesizing unit adjusts the position so that the characteristic portions extracted from the image data before the synthesis overlap, and synthesizes the acquired image data. And

  According to a sixth aspect of the present invention, there is provided an image pickup apparatus comprising the image processing apparatus according to the fourth or fifth aspect, wherein the image data acquisition means acquires by photographing.

A seventh aspect of the present invention is the apparatus according to the sixth aspect, wherein the determining means determines the number of photographing times for acquiring new image data by the following equation.
Predetermined brightness level / brightness level of image data taken first = α
Number of shots = (minimum integer value greater than or equal to α) −1

  The invention according to claim 8 is a step of measuring a luminance level of image data, a step of determining whether the measured luminance level has reached a predetermined luminance level, and the image data has reached the predetermined luminance level. If not, acquiring new image data; synthesizing the acquired new image data with the image data; measuring a luminance level of the synthesized image data; Determining whether the brightness level of the composite image data has reached the predetermined brightness level, obtaining the new image data until determining that the brightness level has reached the predetermined brightness level, and performing the synthesis This is an image processing method characterized by repetition.

  According to the present invention, it is possible to provide an image processing apparatus, an imaging apparatus, and an image processing method that do not increase the number of images to be combined more than necessary when combining a plurality of images, and suppress deterioration in operability to a minimum while improving image quality. Can do.

  Embodiments of the present invention will be described below in detail with reference to the drawings. The embodiments described below are preferred embodiments of the present invention, and thus various technically preferable limitations are given. However, the scope of the present invention is particularly limited in the following description. As long as there is no description which limits, it is not restricted to these aspects.

  First, the configuration of the imaging apparatus according to the embodiment of the present invention will be described. FIG. 1 is a block configuration diagram of an imaging apparatus according to the present embodiment. The camera module 1 includes an optical lens 2, an image sensor 3, an AD (Analog to Digital) converter 4, an image processing DSP (Digital Signal Processor) 5, an I / F (interface) circuit 6, and a power source / V driver circuit 7. And a microcomputer 8. The camera module 1 is connected to the CPU 10 and the memory 11 via the device internal bus 9. An LCD (Liquid Crystal Display) 12 is connected to the device internal bus 9 as an image display means.

  An AD converter (Analog to Digital) 4 converts an optical lens 2, an image sensor 3, and an analog signal output from the image sensor 3 into a digital signal.

  The image processing DSP 5 performs predetermined image processing such as color interpolation, color correction, and image quality adjustment on the digital signal output from the AD converter 4 to generate and output a digital video signal, as well as a horizontal / vertical synchronization signal, Outputs the CLK (Clock) signal.

  The I / F circuit 6 exchanges the output signal of the image processing DSP 5 and the control signal from the CPU 10 via the device internal bus 9.

  The power source / V driver circuit 7 drives the image sensor 3.

  The microcomputer 8 controls the image processing DPS 5 and the power / V driver circuit 7 on the basis of a control signal coming through the I / F circuit 6.

  Next, operation processing of the imaging apparatus according to the present embodiment will be described with reference to the drawings. FIG. 2 is a flowchart showing a series of shooting operation processes including a process of determining the necessity of improving the luminance value (image brightness) in the imaging apparatus according to the present embodiment.

  After the apparatus is turned on (step S1), the camera is started (step S2), and camera initialization (Initialize) is executed (step S3). In this initialization, initial values such as camera operation clock, shooting angle of view, and image quality settings are set.

  When the initialization operation is completed, shooting with the camera becomes possible (step S4). The video signal obtained from the image sensor 3 is input to the image processing DSP 5 and subjected to predetermined image processing (step S5). The image processing DSP 5 is provided with various registers (Registers) for image processing and image quality adjustment, and image quality adjustment can be performed by changing the register values. The image processing DSP 5 outputs a video signal that has been subjected to predetermined image processing.

  The video signal photographed by the camera module 1 is once taken into the memory 11 via the device internal bus 9 and previewed on the LCD 12 (step S6).

  Next, the image processing DSP 5 analyzes the image data stored in the memory 11 in step S6, specifically, determines the luminance value (step S7). The determination is performed by comparing the luminance signal level of the captured image data with a preset reference value (Y_ref). The luminance signal level of the image data can be obtained, for example, by calculating the average value (Y_ave) of the luminance signals of the obtained image data.

  Next, the necessity for luminance value improvement is determined based on the result of the luminance value determination. If Y_ave ≧ Y_ref, it is determined that the luminance value does not need to be improved, assuming that the luminance value has reached a predetermined level. When the predetermined luminance level is reached and it is determined that the necessity for improving the luminance value is unnecessary, the next image necessary for the image composition processing for improving the luminance value is not required (step S8 / No), and image recording is performed. The process proceeds to processing (step S9). When the image recording process is completed, a notification that the image recording process has ended is sent (step S10), and the series of operations is ended after the image is stored.

  On the other hand, when the luminance level Y_ave <Y_ref, it is determined that the luminance value needs to be improved. If the necessity of improving the luminance value is recognized (Y_ave <Y_ref), the next image is captured (step S8 / Yes). The CPU 10 continuously instructs the camera to acquire the next image and also instructs the image processing DSP 5 to perform image composition. With this command, the camera takes the next image with the same shutter speed and gain as the previous time (step S4). Such second image data is referred to as image data B.

  The image processing DSP 5 combines the previous image data (referred to as image data A) and the image data B (step S5).

  Here, a method for synthesizing a plurality of images will be described. In the case of a completely stationary subject, there is no problem even if it is synthesized as it is without any processing. However, when an actual shooting scene is taken into consideration, it can be considered that the apparatus side and the subject side move slightly. In such a case, if a process of simply combining images is performed, the images are combined in a shifted state, resulting in a blurred image, a so-called ghost image, and the image quality is significantly degraded. Therefore, a synthesis method that prevents image quality deterioration is required. FIG. 3 is a diagram for explaining a feature point extraction method which is an example of a method for combining a plurality of images.

  In the feature point extraction method, portions that are features of the subject are extracted from the composition source image (image data A and image data B) (see the circles in FIG. 3), and the position adjustment is performed so that these feature points overlap. Go and synthesize.

  As a result, even when a plurality of images in which the positions of subjects in the image are shifted are combined in order to improve the luminance value, it is possible to obtain a high-quality image while preventing image quality deterioration.

  The luminance value of the image synthesized in step S5 is determined again in steps S7 and S8. Thereafter, image capturing and composition are repeated until it is determined in step S8 that the luminance value does not need to be improved (Y_ave ≧ Y_ref).

  According to the above embodiment, the brightness level of the captured image data is analyzed, and image capturing and synthesis are repeated until the brightness value reaches a predetermined level. Therefore, the optimum brightness value (brightness) under any shooting conditions. ) Can be obtained, and more than necessary shooting and composition processing is not required, so that the processing time can be shortened and shooting with optimum image quality can be performed while maintaining operability. . Thus, it is possible to achieve both improvement in image quality and improvement in operability (processing time reduction).

  Next, another embodiment of the present invention will be described with reference to FIG. The difference between the above embodiment and the present embodiment is that a composite number calculation process (step S25 in FIG. 4) is added after the luminance value determination process (step S8 in FIG. 2 and step S22 in FIG. 4). .

  After the apparatus is turned on (step S15), the user activates the camera (step S16), and camera initialization is executed (step S17). When the initialization (step S17) operation is completed, shooting with the camera becomes possible (step S18), and the video signal obtained from the image sensor 3 is input to the image processing DSP 5 and subjected to predetermined image processing to obtain the video signal. Is output (step S19).

  The image processing DSP 5 is equipped with various registers for image processing and image quality adjustment, and image quality adjustment is possible by changing the register values. Next, the video signal is once taken into the memory 11 via the internal bus 9 and previewed on the LCD 12 (step S20).

  Next, in step S21, the image data stored in the memory 11 in step S20 is analyzed. The analysis is performed by the image processing DSP 5. For the image data stored in the memory 11, the image processing DSP 5 measures the luminance signal level and compares it with a preset luminance level reference value Y_ref. The measurement of the luminance level can be obtained by, for example, calculating the average value Y_ave of the luminance signals of the entire image obtained.

  As a result of the luminance level (Y_ave) measurement in step S21, it is determined whether or not the luminance value needs to be improved (step S22). If the luminance level Y_ave <Y_ref, it is determined that the luminance value needs to be improved (step S22 / Yes). If Y_ave ≧ Y_ref, it is determined that the luminance value has reached a predetermined level and the luminance value needs not be improved. (Step S22 / No).

  If it is determined in step S22 that there is no need to improve the luminance value (Y_ave ≧ Y_ref) (step S22 / No), image recording processing of image data is performed (step S23), and notification of completion of image processing is performed ( Step S24), after the image is stored, the series of operations is terminated.

  On the other hand, when the necessity of the luminance value improvement is recognized in step S22 (Y_ave <Y_ref) (step S22 / Yes), the next image is captured (re-photographed) in order to synthesize a plurality of images.

  Here, the composite number calculation (step S25), which is a feature of the present embodiment, will be described. The ratio Y_ref / Y_ave = α of the luminance level reference value Y_ref and Y_ave is calculated, and the number of acquired images β for obtaining the reference luminance level is calculated. β is the smallest integer value −1 equal to or greater than α. Based on the composite number value, the image processing DSP 5 instructs the camera to acquire the next plurality of images. In response to this command, the camera captures an image with the same shutter speed and gain as the previous time (step S18).

  For example, when α = 3.5, the number of acquired images β is 4-1 = 3. The plurality of images photographed based on the value of β are then synthesized by the image processing DSP 5 (step S19). Here, in the flowchart shown in FIG. 4, it is checked again whether the luminance level has reached the reference value through the determination processing in steps S21 and S22, but the determination processing in steps S21 and S22 is skipped. You can also think about doing it.

  According to the above-described embodiment, the number of composites required for improving the brightness value is calculated based on the analysis of the image data captured for the first time, and the necessary number of images are continuously captured, so that the processing time can be shortened. Further, when the luminance value determination process for the combined image data is not required, the processing time can be further shortened.

  It is obvious that the present invention can be applied to all imaging devices such as an electronic still camera and a camera-equipped mobile phone having a configuration using the camera module 1 shown in FIG.

  A program for the CPU to execute the processing shown in the flowchart constitutes a program according to the present invention. As a recording medium for recording the program, a semiconductor storage unit, an optical and / or magnetic storage unit, or the like can be used. By using such a program and a recording medium in a system having a configuration different from that of each of the above-described embodiments and causing the CPU to execute the program, substantially the same effect as the present invention can be obtained.

  Although the present invention has been specifically described above based on the preferred embodiments, it is needless to say that the present invention is not limited to the above-described ones and can be variously modified without departing from the gist thereof.

It is a block block diagram of the imaging device which concerns on embodiment of this invention. It is a flowchart which shows a series of imaging | photography operation processes including the process which judges the necessity for the brightness value improvement in the imaging device which concerns on embodiment of this invention. It is a figure for demonstrating an example of the synthetic | combination method of the several image which concerns on embodiment of this invention. It is a flowchart which shows a series of imaging | photography operation processes including the process which judges the necessity for the brightness value improvement in the imaging device which concerns on embodiment of this invention. It is a block block diagram of a prior art. It is a flowchart which shows the operation processing of a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera module 2 Optical lens 3 Image pick-up element 4 AD converter 5 Image processing DSP
6 I / F circuit 7 Power supply / V driver circuit 8 Microcomputer 9 Device internal bus 10 CPU
11 Memory 12 LCD

Claims (8)

A luminance measuring means for measuring the luminance level of the image data;
Determining means for determining whether the measured luminance level has reached a predetermined luminance level;
If the determination means determines that the image data has not reached the predetermined luminance level, image data acquisition means for acquiring new image data;
Synthesizing means for synthesizing the acquired new image data with the image data,
The luminance measuring unit measures a luminance level of the synthesized image data synthesized by the synthesizing unit,
The determination means determines whether the brightness level of the composite image data has reached the predetermined brightness level, and repeats the acquisition of the new image data and the combining until it is determined that the brightness level has reached the predetermined brightness level. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the synthesizing unit adjusts the position so that the feature portions extracted from the image data before the synthesis overlap, and synthesizes the acquired image data. The image processing apparatus according to claim 1 or 2,
The image pickup apparatus according to claim 1, wherein the image data acquisition unit acquires the image data by shooting. A luminance measuring means for measuring the luminance level of the image data;
Determining means for determining whether the measured luminance has reached a predetermined luminance level;
If the determination means determines that the image data has not reached the predetermined luminance level, image data acquisition means for acquiring new image data;
Determining means for determining the number of new image data acquired by the image data acquiring means;
An image processing apparatus, comprising: a combining unit that combines the acquired at least one new image data and the image data.   5. The image processing apparatus according to claim 4, wherein the synthesizing unit synthesizes the acquired image data by adjusting a position so that feature portions extracted from the image data before synthesis overlap. An image processing apparatus according to claim 4 or 5,
The image pickup apparatus according to claim 1, wherein the image data acquisition unit acquires the image data by shooting. The imaging apparatus according to claim 6, wherein the determining unit determines the number of times of photographing for acquiring new image data by the following formula.
Predetermined brightness level / brightness level of image data taken first = α
Number of shots = (minimum integer value greater than or equal to α) −1 Measuring the brightness level of the image data;
Determining whether the measured luminance level has reached a predetermined luminance level;
If it is determined that the image data has not reached the predetermined brightness level, obtaining new image data;
Combining the acquired new image data with the image data;
Measuring a luminance level of the synthesized composite image data;
Determining whether the brightness level of the composite image data has reached the predetermined brightness level,
An image processing method characterized by repeating the acquisition of the new image data and the synthesis until it is determined that the predetermined luminance level has been reached.

Images