JP2023091493A - Image processing apparatus and image processing method - Google Patents

Abstract

To provide an image processing apparatus and an image processing method that can perform image composition in which a pixel value other than that in an effective pixel area can be appropriately used.SOLUTION: An image processing apparatus aligns and composes pieces of image data of a plurality of frames. The pieces of image data include first data that is data of pixels in an effective pixel area of an image pickup device and is composed of signals corresponding to the sequence of predetermined color components, and second data that is data of pixels other than the pixels in the effective pixel area. The image processing apparatus aligns the second data based on the amount of alignment used for aligning the first data. The image processing apparatus corrects data of a first composite image obtained by composing the aligned first data based on data of a second composite image obtained by composing the aligned second data.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing device and an image processing method, and more particularly to an image synthesizing technique.

There is known a technique for suppressing image blur by synthesizing a plurality of n images that are continuously shot with an exposure time that is 1/n of the required exposure time (Patent Document 1).

JP-A-2007-243775

When merging a plurality of images, it is necessary to align the images. However, if the images are simply aligned, the pixels outside the effective pixel area will be synthesized with their positions shifted. Therefore, a problem arises when it is desired to use the values of pixels outside the effective pixel area. Patent document 1 does not recognize such a problem.

The present invention has been made in view of such problems in the prior art, and one aspect of the present invention provides an image processing apparatus and an image processing method capable of synthesizing an image in which pixel values outside an effective pixel region are appropriately used. offer.

The above object is to provide an image having an acquisition means for acquiring image data of a plurality of frames, a registration means for aligning the image data of the plurality of frames, and a synthesizing means for synthesizing the aligned image data of the plurality of frames. In the processing device, the image data includes first data of pixels in an effective pixel area of an image sensor, which is composed of signals corresponding to an arrangement of predetermined color components, and data of pixels outside the effective pixel area. The second data is pixel data, and the alignment means aligns the image data of the plurality of frames based on the alignment amount used to align the first data contained in the image data of the plurality of frames. The included second data is aligned, and the synthesizing means synthesizes first synthesized image data obtained by synthesizing the aligned first data and second synthesized image data obtained by synthesizing the aligned second data. and generating image data, the image processing device further comprising correcting means for correcting the data of the first synthesized image based on the data of the second synthesized image. be done.

According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of performing image synthesis in which pixel values outside the effective pixel area are appropriately used.

1 is a block diagram showing a functional configuration example of an imaging device as an image processing device according to an embodiment; FIG. Schematic diagram of the pixel area of the image sensor Schematic diagram of synthesis processing of RAW data in the first embodiment Flowchart relating to operation of composite shooting mode in the first embodiment A diagram for explaining the background of the second embodiment. Diagram for explaining the outline of the second embodiment Flowchart relating to operations in composite shooting mode in the second embodiment

The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. In addition, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

In addition, below, the form which implements this invention by imaging devices, such as a digital camera, is demonstrated. However, the imaging function is not essential to the present invention, and can be implemented in any electronic device capable of handling image data. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smart phones, game consoles, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented in other electronic devices.

FIG. 1 is a block diagram showing an example functional configuration of an imaging device 10 as an example of an image processing device according to an embodiment of the present invention.

The control unit 113 has one or more processors capable of executing programs, a RAM, and a ROM. The control unit 113 can load the program stored in the ROM into the RAM and execute it by the processor. By executing a program, the control unit 113 controls the components of the imaging apparatus 10 including the functional blocks shown in FIG. 1 and/or the operations of external devices communicably connected. The functions of the imaging device 10 can be implemented by the control unit 113 executing a program. It should be noted that functions that can be realized by the control unit 113 executing a program may be implemented using hardware (for example, ASIC, FPGA, etc.). The ROM is electrically rewritable, for example, and stores programs executable by the processor of the control unit 113, setting values of the imaging device 10, GUI data, and the like. The control unit 113 can have the form of a system-on-chip (SoC) or a system-on-package (SiP), for example.

A photographing lens 100 has a plurality of lenses and a diaphragm, and generates an optical image of a subject. The imaging element 102 is, for example, a CMOS image sensor, and has a plurality of photoelectric conversion units (pixels) arranged two-dimensionally.

FIG. 2 is a diagram schematically showing the pixel arrangement of the image sensor 102. As shown in FIG. The image sensor 102 has an effective pixel area 200 and an optical black (OB) area 201 outside the effective pixel area as areas in which pixels are arranged. The effective pixel area 200 is an area used (exposed) for photographing a subject image. On the other hand, an OB area 201 is an area that is not used (unexposed) for photographing a subject image. Pixels in the OB region 201 may be provided with, for example, a light shielding film to prevent light from entering. A further OB region 202 may be provided above or below the effective pixel region 200 . The OB area 202 can be used not only for black level calculation but also for line offset correction for each vertical pixel line.

The pixels arranged in the OB region 201 have the same structure as the pixels arranged in the effective pixel region 200 except for the light shielding film. When the pixels in the effective pixel region 200 are provided with color filters, the pixels in the OB region 201 are also provided with color filters in the same manner as the pixels in the effective pixel region 200 .

The effective pixel area 200 and the OB area 201 are the areas of the image sensor 102, but for the sake of convenience in this specification, the same expressions are used for the image obtained by the image sensor 102 as well. For example, an image or an effective pixel area means an image area obtained by the pixels of the effective pixel area 200 of the image sensor 102 in the image of one frame obtained by shooting.

The color filter has a configuration in which unit filters of different colors are arranged, and one unit filter is provided for each pixel. For example, a primary-color Bayer array color filter has a configuration in which four types of unit filters, R (red), G1 (green), G2 (green), and B (blue), are arranged in a repeating unit of 2×2 pixels. . Below, the pixel provided with the unit filter G1 is called a G1 pixel. The same applies to pixels provided with other types of unit filters.

Pixels arranged in the OB region 201 are used, for example, to detect noise components of photoelectric conversion elements (for example, photodiodes) included in the pixels. For example, by subtracting the signal value of the pixel in the OB area 201 from the signal value of the pixel in the effective pixel area 200, the offset due to the noise component can be removed and the black level of the image can be corrected.

A pixel included in the image sensor 102 generates a pixel signal having a value corresponding to the amount of charge generated during the charge accumulation period. Note that in shooting with the mechanical shutter 101 opened and closed, the charge accumulation period corresponds to the exposure period. In photographing in which the mechanical shutter 101 is kept open, the charge accumulation period corresponds to the period from the resetting of the accumulated charges to the elapse of the exposure period. Generally, the former corresponds to still image shooting and the latter corresponds to moving image shooting, but the latter can also correspond to still image shooting.

When one charge accumulation period ends, a pixel signal group (analog image signal) for one frame is read out from the image sensor 102 . One frame of analog image signal includes pixel signals of an effective pixel area 200 and an OB area 201 . The analog image signal is converted into a digital image signal (digital format pixel signal group) by the A/D conversion unit 103 . Note that the A/D conversion unit 103 may be omitted if the image sensor 102 can output a group of digital pixel signals.

At this stage, each pixel signal has only one color component corresponding to the color of the unit filter that the corresponding pixel has. In this specification, a digital image signal composed of pixel signals corresponding to a predetermined color component arrangement (a pixel signal having only one color component corresponding to the color of a unit filter) is referred to as RAW data. call.

RAW data output from the A/D conversion unit 103 or the image sensor 102 is temporarily stored in the memory 114 . The memory 114 is used to temporarily store RAW data, image data processed by the signal processing unit 111, and the like.

The OB integration unit 104 calculates an average pixel value for each type of pixel for the pixel values in the OB area 201 in the RAW data. Here, since the image sensor 102 has a color filter of primary color Bayer array, the OB integration unit 104 calculates the average pixel value of the R, G1, G2, and B pixels in the OB area 201 . This average pixel value is used as the black level.

The OB clamping unit 105 applies OB clamping processing for correcting the black level to the pixel values of the effective pixel region 200 based on the pixel values of the OB region 201 . Specifically, the OB clamping process may be a process of subtracting the average pixel value calculated by the OB integration unit 104 . The OB clamping process can suppress black floating, color shift, and the like in an image obtained from pixel signals in the effective pixel area 200 . Note that the OB clamp processing uses a black level corresponding to the type of pixel. For example, for the pixel values of the R pixels in the effective pixel area 200, the black level obtained from the R pixels in the OB area is subtracted. The OB clamping unit 105 applies OB clamping processing to each of the original images to be combined and to the combined image generated by the combining unit 108 .

A shading correction unit 106 applies shading correction to the pixel values of the effective pixel area 200 . The shading correction corrects luminance reduction according to the pixel position due to the optical characteristics of the imaging lens 100 and the microlenses of the pixels. Therefore, in shading correction, a gain is applied according to the pixel position.

A white balance (WB) processing unit 107 applies white balance adjustment processing to the image after shading correction. The white balance adjustment process is a process of applying a gain corresponding to the pixel type (R, G1, G2, B) to the shading-corrected pixel value.

The misregistration detection unit 109 detects the amount of misregistration for each of the multiple frames of images to be synthesized. The amount of positional deviation may be an absolute amount of deviation where the amount of deviation of the first frame photographed first is set to 0, or may be an amount of relative deviation with respect to the immediately preceding frame. When detecting an absolute shift amount, the image of the first frame is the reference image.

The positional deviation detection unit 109 can detect the amount of deviation as a motion vector of the image of the effective pixel area 200 between frames. The positional deviation detection unit 109 can detect the amount of deviation using any known technique, such as a method using template matching between frames or a method using the output of a gyro sensor or the like included in the imaging device 10 . The positional deviation detection unit 109 outputs the detected positional deviation amount to the positional deviation correction unit 110 or stores it in the memory 114 .

The misregistration correction unit 110 aligns the images of the frames to be synthesized based on the amount of misregistration detected by the misregistration detection unit 109 . For example, the positional deviation correction unit 110 can perform positional alignment by changing the coordinate value of each pixel of the frame to be synthesized according to the amount of positional deviation.

Note that the misregistration correction unit 110 has a mode for aligning only the image of the effective pixel area 200 and a mode for aligning the image of an area including the effective pixel area 200 and the OB area 201 (for example, the entire frame). . In this manner, the positional deviation correction unit 110 can align the data of the effective pixel area 200 (first data) independently of the data of the OB area 201 (second data). For example, the control unit 113 can set the mode of the positional deviation correction unit 110 .

Even if a mode for aligning the entire frame is set, the control unit 113 is set to a mode for aligning only the image of the effective pixel region 200 when the condition that the compositing accuracy of the OB region 201 is lowered by alignment is satisfied. can be changed to For example, when the number of frames in which the OB area 201 overlaps is equal to or less than a predetermined ratio of the total number of frames N, the control unit 113 can change to a mode in which only the images of the effective pixel area 200 are aligned. The number of frames in which the OB region 201 overlaps can be grasped based on the positional displacement amount of each frame detected by the positional displacement detection unit 109 .

Combining unit 108 combines a plurality of images in one of a plurality of selectable combining modes to generate a combined image. It is not necessary to apply the synthesizing process to the entire frame, and it is sufficient to perform the synthesizing process on the effective pixel area and the OB area in the reference image.

Synthesis unit 108 stores the generated synthetic image in memory 114 . In the present embodiment, the imaging apparatus 10 can select an addition mode, an average addition mode, or a comparatively bright mode as a synthesis mode. Note that these combining modes are examples, and other combining modes may be selectable, or there may be only one combining mode.

An image synthesizing method in each synthesizing mode will be described. Assume here that images of N frames (N is an integer equal to or greater than 2) are synthesized. A pixel constituting an image of each frame has coordinates (x, y) in an xy orthogonal coordinate system, and the luminance value of the pixel at the coordinates (x, y) is I_i (x, y) (i=1 to N). , and let I(x, y) be the luminance value of the pixel at the coordinates (x, y) of the synthesized image. The synthesizing unit 108 calculates the luminance value I(x, y) of each pixel of the synthesized image as follows, according to the synthesis mode.

・Addition mode I(x, y)=I_1(x, y)+I_2(x, y)+...+I_N(x, y)
In addition mode, the synthesizing unit 108 adds luminance values of pixels at the same coordinates in each frame to generate a synthesized image. The addition mode is used, for example, when N frames of images shot with an exposure amount that is 1/N of the proper exposure amount are combined to generate a proper exposure image.

・Averaging mode I(x, y)=(I_1(x, y)+I_2(x, y)+...+I_N(x, y))/N
In the averaging mode, the synthesizing unit 108 divides the brightness value obtained in the same manner as in the addition mode by the number of frames N, thereby generating a synthesized image in which the brightness value of each pixel is the average value of N frames. The averaging mode is used, for example, to reduce noise in images shot with high sensitivity.

・Light mode I(x, y)=max(I_1(x, y), I_2(x, y), . . . , I_N(x, y))
Here, max() is a function that extracts the maximum value of the elements in (). A composite image is obtained which consists of the maximum luminance value of the N pixels with the same coordinates in each frame. The comparatively bright mode is effective, for example, when synthesizing an image of fireworks or a starry sky.

In the addition mode and the averaging mode, the combination unit 108 converts the pixel value of the coordinate in another predetermined frame (for example, the first frame) to the coordinates for which there is no pixel value to be combined due to the change in the coordinates due to the alignment. can be used instead. In addition, the synthesizing unit 108 can perform synthesis using only existing pixel values in the averaging mode or the relatively bright mode (in the averaging mode, the divisor is set to the number of frames used for addition (<N)). change). Note that these are only possible examples, and synthesis may be performed using other methods.

Also, the combining unit 108 prevents the image outside the effective pixel area 200 from being combined with the effective pixel area of the reference image. The synthesizing unit 108 can handle pixel values outside the effective pixel region 200 whose coordinates have been changed to overlap with the effective pixel region of the reference image by alignment in the same manner as when pixel values do not exist.

The signal processing unit 111 applies development processing to the (uncombined) RAW data and data of a combined image generated by the combining unit 108 (combined RAW data). It should be noted that the image area corresponding to the effective pixel area 200 is subjected to the development process. Development processing is a general term for a plurality of image processing such as color interpolation processing and tone correction processing (gamma processing). Color interpolation processing is processing that interpolates values of color components that cannot be obtained at the time of shooting, and is also called demosaicing processing. By applying color interpolation processing, each pixel comes to have a plurality of color components (for example, RGB and YCbCr) necessary for a color image, and is no longer RAW data.

The signal processing unit 111 performs detection processing such as detection of feature regions (for example, face regions and human body regions) and their movements, detection processing such as person recognition processing, synthesis processing, scaling processing, encoding and decoding on the image data after development. Treatment can be applied. The signal processing unit 111 also performs data processing such as header information generation processing, generation of signals and evaluation values used for automatic focus detection (AF), and evaluation value calculation such as calculation of evaluation values used for automatic exposure control (AE). Various image processing can be applied, such as processing. Note that these are examples of image processing to which the signal processing unit 111 can apply, and the image processing to which the signal processing unit 111 applies is not limited.

The recording unit 112 records RAW data, composite RAW data, developed image data, audio data accompanying these data, etc. on a recording medium such as a memory card, according to the shooting mode and recording settings.

The operation unit 115 is a general term for input devices (switches, keys, buttons, dials, touch panels, etc.) for the user to give various instructions to the imaging apparatus 10 .
The display unit 116 is, for example, a touch display, and is used to display live view images, reproduced images, GUI, setting values and information of the imaging device 10, and the like.

The operation of the imaging device 10 in the composite shooting mode will be described with reference to the flowchart of FIG. 4 . The composite photographing mode is a photographing mode for recording a composite image obtained by synthesizing a plurality of frames of images captured over time. In the composite photographing mode, one of the plurality of composite modes described above is set. Note that although still image shooting will be described here, it is also possible to perform generation processing for one frame of moving image shooting in which each frame is a composite image.

In S<b>101 , the control unit 113 receives setting of the number of frames N to be combined from the user through the operation unit 115 . Note that the number of frames N may be set in advance, similar to the setting of the synthesis mode.

In S<b>102 , the control unit 113 detects that the user has input a photographing instruction through the operation unit 115 . The shooting instruction may be, for example, a full-press operation of the shutter button included in the operation unit 115 . Upon detecting the input of the shooting instruction, the control unit 113 executes S103.

In S<b>103 , the control unit 113 executes imaging processing for one frame. The exposure conditions (shutter speed, aperture value, sensitivity) at the time of shooting are obtained by executing AE processing using, for example, a live view image before S102 (for example, when half-pressing the shutter button is detected). It can be an appropriate exposure amount. Note that when the synthesis mode is the addition mode, the control unit 113 sets an exposure condition that is 1/N of the appropriate exposure amount based on the number of frames N set in S101.

The control unit 113 stores RAW data obtained by shooting in the memory 114 . The control unit 113 skips S104 and executes S105 when photographing the first frame, and executes S104 when photographing the second and subsequent frames.

In S<b>104 , the positional deviation detection unit 109 detects the deviation amount of the image (RAW data) obtained in the most recent shooting. Here, for all images from the second frame onwards, the amount of deviation from the first frame image (reference image) is detected. Further, the displacement amount is detected using template matching using a part of the reference image as a template.

Here, in order to increase the detection accuracy of the amount of displacement, the positional displacement detection unit 109 generates a plurality of templates from the reference image and detects the amount of positional displacement for each template. Then, the positional deviation detection unit 109 determines the final positional deviation amount based on the plurality of detected positional deviation amounts. The positional deviation detection unit 109 can determine, for example, the most frequent positional deviation amount or the average value of the positional deviation amounts as the final positional deviation amount, but other methods may be used.

In S105, the OB integration unit 104 calculates the black level for each type of pixel using the pixel values of the OB region 201 in the image (RAW data) obtained by the most recent imaging.

In S106, the OB clamp unit 105 uses the black level calculated in S105 to apply OB clamp processing to the pixels in the effective pixel area of the image (RAW data) obtained in the most recent shooting.

In S107, the positional deviation correction unit 110 uses the positional deviation amount detected in S104 to align the image (RAW data) obtained in the latest photographing with the reference image. This alignment processing will be described with reference to FIG.

FIG. 3 schematically shows a configuration for aligning a second frame image (RAW data) 302 with respect to a first frame image (RAW data) 301, which is a reference image, and synthesizing the images. An image 301 has an effective pixel area 301a and an OB area 301b. Similarly, image 302 has an effective pixel area 302a and an OB area 302b.

Here, it is assumed that the mode of the positional deviation correction unit 110 is set to a mode in which only the images in the effective pixel area are aligned. Therefore, in the image 302' after alignment, only the effective pixel area 302a' has moved, and the OB area 302b has not moved. As a result, an image 302' after alignment has a region 302c where no pixels exist. When the mode of the positional deviation correction unit 110 is set to the mode of aligning the entire frame, the coordinate information is internally changed, but the appearance of the image 302 does not change before and after the alignment.

In S108, the synthesizing unit 108 synthesizes the image aligned in S107 (image 302' in FIG. 3) with the reference image or the synthesized image (synthesized RAW data) up to the previous frame. The synthesizing unit 108 performs the synthesizing process as described above according to the set mode.

FIG. 3 shows an example of compositing processing in the averaging mode or the comparatively bright mode in which a pixel-free region 302c generated by alignment is excluded from compositing targets. As a result, the effective pixel area 310a of the synthesized image 310 includes an area 310c where the image of the second frame is not synthesized. As for the OB area 310 of the synthesized image 310, the OB area 301b of the first frame and the OB area 302b of the second frame are synthesized by a method according to the mode.

Note that if the pixel signal is affected by noise in the negative direction, the pixel value may become less than 0 when the black level is reduced by the OB clamping process in S106. To prevent this, the composite image may be given a predetermined positive offset before being stored in memory 114 . If an offset is given to the composite image, the same amount of offset as that given to the composite image is given to the composite image and then the OB clamping process of S105 is applied. After applying the OB clamping process, the offset is removed from both images before applying the compositing process at S108.

In addition, in the flowchart of FIG. 4, synthesis is sequentially performed from the second frame on the reference image, but it is also possible to generate synthesized images from the second frame to the N-th frame and finally synthesize them with the reference image. In this case, synthesis from the 2nd frame to the Nth frame may be performed only in areas where all the frames overlap. In the effective pixel area of the reference image, the method according to the above-described synthesis mode is applied assuming that there is no pixel value to be synthesized for the portion where the synthesized image from the 2nd frame to the Nth frame is not generated. can be done.

In S109, the control unit 113 determines whether or not the shooting of the set number of frames N has been completed. The control unit 113 executes S110 if it is determined that the photographing has been completed, and executes the photographing of the next frame in S103 if it is not determined. Note that the control unit 113 is assumed to continuously perform shooting while the shooting instruction is continuously input, for example. When the input of the shooting instruction stops before the shooting of the set number of frames N is completed, for example, the control unit 113 may discard the processing results performed so far and return to the shooting standby state.

In S<b>110 , the OB integrator 104 calculates the black level based on the pixel values of the OB area of the synthesized image stored in the memory 114 . Calculation of the black level is the same as in S105, except that the pixel values in the OB area are pixel values after the compositing process.

In S111, the OB integrator 104 calculates a value (for example, a variance value) indicating variation in pixel values in the OB area of the synthesized image. The variance value may be calculated for each pixel type (color), or one variance value may be calculated for all pixels. The OB integrator 104 may execute the processes of S110 and S111 in parallel.

In S112, the OB clamp unit 105 uses the black level based on the composite image calculated in S110 and applies OB clamp processing to pixel values within the effective pixel area of the composite image. Note that the reason why the OB clamping process is applied after composition even though OB clamping is applied to the image before composition in S106 is to reduce the black floating due to composition.

For example, when synthesizing a plurality of frames of images shot with high sensitivity in the comparatively bright mode, the images of each frame contain a lot of noise. Lighter mode compositing selects the highest luminance value at each coordinate. Therefore, there is a high possibility that a luminance value that is higher than the original luminance value due to noise will be selected for each coordinate. Therefore, by applying the OB clamping process to the composite image using the black level based on the pixels of the OB region 201 (composite OB region) composited in the comparatively bright mode similarly to the effective pixel region 200, the black floating can be suppressed. Here, the composition in the comparatively bright mode has been described as a typical example of black floating due to composition. However, since black floating due to composition also occurs in other composition modes, the black level based on the pixels in the composite OB area is used for the composite image. OB clamping is applied.

In S113, the WB processing unit 107 applies white balance adjustment processing to the image (RAW data) of the effective pixel area of the combined image that has undergone OB clamp processing.

In S114, the signal processing unit 111 applies development processing to the RAW data to which the white balance adjustment processing has been applied. The signal processing unit 111 changes parameters of noise reduction processing applied in the process of development processing according to the variance value calculated in S111. Specifically, the signal processing unit 111 changes the parameters of the noise reduction processing so that stronger noise reduction processing is applied when the variance value is greater than a predetermined threshold value. This is because the amount of noise is greater when the variance value is greater than the threshold value than otherwise. Note that if there are three or more levels of noise reduction processing, two or more thresholds may be used to adjust the strength of the noise reduction processing. The signal processing unit 111 generates an image data file storing the image data from the image data after the development processing.

In S115, the recording unit 112 records the image data file generated in S114 in a recording medium such as a memory card or an external device. In addition to or instead of the developed image data, composite RAW data of the effective pixel area may be recorded before the OB clamping process is applied in S112. Further, information necessary for the external device to execute the processes of S112 to S114 (the black level calculated in S110 and the dispersion amount calculated in S111) may be recorded in the combined RAW data in addition to the combined RAW data. In addition, the RAW data of N frames of the composition source may be recorded so that the processes of S104 to S114 can be executed by an external device.

When the recording by the recording unit 112 ends, the control unit 113 ends the operation shown in FIG. 4 and returns to, for example, the shooting standby state.

As described above, according to the present embodiment, when combining RAW data of a plurality of frames, alignment of RAW data can be applied separately to data in the effective pixel area and data in the OB area. . By making it possible to align the data of the effective pixel area independently, it is possible to prevent the data of the OB area from being synthesized with the data of the effective pixel area, thereby suppressing deterioration in the quality of the synthesized image. In addition, since it is possible to prevent the OB area from being shifted between frames to be synthesized, a highly accurate black level can be calculated from the OB area after synthesis, and black floating in an image caused by synthesis can be effectively suppressed. can.

● (Second embodiment)
Next, a second embodiment of the invention will be described. This embodiment relates to the case of performing line offset correction in addition to OB clamp processing. The characteristics of a large number of circuit elements formed in an imaging device vary due to manufacturing errors and the like. For example, a difference in pixel signal offset caused by variations in the characteristics of amplifiers provided for each pixel line can be visually recognized as a striped pattern in an image.

Line offset correction is a process of correcting such a difference in pixel signal offset that may occur in units of horizontal pixel lines or vertical pixel lines. A difference in offset amount for each pixel line can be detected in pixels in the OB area. Therefore, line offset correction can be applied to pixel signals read out from the effective pixel area based on the difference in offset amount for each pixel line detected in the OB area.

However, when the amount of alignment is different between the effective pixel area and the OB area, the positions of the pixel lines requiring line offset correction do not match between the OB area and the effective pixel area after alignment. In particular, when synthesizing images of a plurality of frames, the amount of alignment between the OB area and the effective pixel area may differ for each frame. Therefore, the amount of offset for each pixel line detected in the OB area after synthesis processing does not match the amount of offset of pixel signals in the effective pixel area after synthesis, and line offset correction cannot be performed correctly.

An example in which correct line offset correction cannot be performed will be described with reference to FIG. Although line offset correction for horizontal pixel lines is shown here, line offset correction for vertical pixel lines is similar. When performing line offset correction for vertical pixel lines, the OB area 202 shown in FIG. 2 is used.

FIG. 5 shows an image (RAW data) 302 of a second frame obtained by separately applying the amount of alignment between the OB area and the effective pixel area to an image (RAW data) 301 of the first frame, which is a reference image. It schematically shows a configuration in which they are put together and combined. An image 301 has an effective pixel area 301a and an OB area 301b. Similarly, image 302 has an effective pixel area 302a and an OB area 302b.

Also here, it is assumed that the mode of the positional deviation correction unit 110 is set to the mode for aligning only the images in the effective pixel area. Therefore, in the image 302' after alignment, only the effective pixel area 302a' has moved, and the OB area 302b has not moved. As a result, an image 302' after alignment has a region 302c where no pixels exist.

In an image (reference image) 301 of the first frame, two horizontal pixel lines whose signal offsets are larger (or smaller) than the signal offsets of other horizontal pixel lines by a certain amount or more have bright (or dark) striped patterns 501 . forming Similarly, in the image 302 of the second frame, two horizontal pixel lines also form a bright (or dark) striped pattern 502 .

Since the position of the horizontal pixel line in the image pickup device is fixed, streaky patterns are always formed at the same position in the captured image. Therefore, before aligning the effective pixel region 302a of the second frame image with the effective pixel region 301a of the first frame image, streak patterns are formed at the same positions in the first frame image and the second frame image. be done.

However, by aligning the effective pixel region 302 with the effective pixel region 301 of the first frame image, the streaky pattern in the image of the second frame can be reduced in number in the effective pixel region 302′ of the image after alignment. position has changed.

As a result, in the effective pixel region 310a of the composite image 301 of the image of the first frame and the image of the second frame, in addition to the streak pattern 501 of the effective pixel region 301, the streak pattern 502 of the effective pixel region 302a′ is displayed. 'It is included.

On the other hand, focusing on the OB area, the OB areas 301b and 302b of the images of the first and second frames are combined without being aligned. Therefore, the number and positions of streak patterns in the OB region 310b in the synthesized image 310 are the same as before synthesis.

Here, line offset correction will be described. Line offset correction includes detection of the presence or absence and position of a streak pattern, and determination of an offset correction amount. Line offset correction can be performed by the signal processor 111, for example.

The presence or absence and position of the streak pattern is detected when the absolute value of the difference between the representative signal level and the pixel signal line extending in the vertical direction, which is formed by averaging the pixel values of the OB region 310b in the horizontal direction, is equal to or greater than the threshold. It may be the detection of pixels with signal levels and their locations. The representative signal level may be the average or median signal level. Also, the threshold can be determined experimentally in advance.

If there is a pixel having a signal level whose absolute value of the difference from the representative signal level is equal to or greater than the threshold, the position corresponds to the position of the pixel line to be corrected, and the difference between the signal level and the representative signal level is offset. It can be used as a correction amount. Note that the pixel line to be corrected and its position may be detected and the offset correction amount may be determined by a method different from that described here.

For the composite image 310, consider applying line offset correction to the effective pixel region 310a based on the streak pattern detected in the OB region 310b. In this case, in the effective pixel area 511a of the corrected image 511, among the images to be synthesized, a streak pattern of the image (here, the image of the second frame) whose vertical alignment amount of the effective pixel area is not 0 is displayed. 502' remains.

In this embodiment, the above problem is solved by generating a composite image for line offset correction. Specifically, the OB area of each frame image to be synthesized is aligned with the OB area of the reference image in accordance with the amount of alignment of the effective pixel area, and synthesized to obtain a synthesized image for line offset correction. Generate.

Therefore, in this embodiment,
・Synthetic image obtained by aligning and synthesizing effective pixel areas (first synthetic image)
・Synthetic image for line offset correction (second synthetic image) obtained by aligning and synthesizing the extracted OB areas
・Synthetic image for OB clamping (black level adjustment) obtained by synthesizing without aligning the OB areas (third synthetic image)
is generated.

FIG. 6 shows an example of using a synthesized image for line offset correction in the same synthesizing process as in FIG. As shown in FIG. 6, the synthesizing unit 108 extracts an OB area 302b of a second frame image 302 to be synthesized with a first frame image (reference image) 301, and vertically adjusts the alignment amount of the effective pixel area 302a. A direction component is applied to generate the aligned image 601 . Then, the synthesizing unit 108 synthesizes the aligned OB area image 601 with the OB area 301b extracted from the reference image to generate a synthesized image 602 for line offset correction.

When synthesizing three or more frames, the synthesizing unit 108 extracts the OB area for each frame image other than the reference image, applies the vertical component of the alignment amount of the effective pixel area, and performs alignment. The OB area extracted from the image is synthesized sequentially. In this manner, the synthesizing unit 108 can generate a synthesized image for line offset correction.

When aligning the OB area, only the vertical component of the alignment amount of the effective pixel area is applied because the target of the line offset correction is the horizontal pixel line. When the target of line offset correction is a vertical pixel line, only the horizontal component of the alignment amount of the effective pixel area is applied to the OB area 202 of the frame image to align the OB area.

Note that the composite image for line offset correction uses the OB regions extracted from each frame, and therefore does not affect the synthesis of the OB regions of each frame. The synthesizing unit 108 performs synthesizing for line offset correction separately from synthesizing the OB area used for OB clamping and synthesizing the effective pixel area.

In the synthesized image 602 for line offset correction, a streak pattern exists at the same position as the aligned and synthesized effective pixel region 310a. Therefore, by applying line offset correction to the combined effective pixel region 310a using the combined image 602 for line offset correction, the streak pattern can be corrected appropriately. As shown in FIG. 6, no stripe pattern exists in the effective pixel area 511a of the image 511' after the line offset correction.

Note that when applying line offset correction using the synthetic image 602 for line offset correction, the signal processing unit 111 selects an area common to all of the OB areas used to generate the synthetic image 602 in the synthetic image 602. may be used. In this case, the signal processing unit 111 does not use an area, such as the area indicated by 602a in FIG. 6, where only the OB areas of some frames are synthesized, for line offset correction.

FIG. 7 is a flow chart obtained by adding a process related to line offset correction described in this embodiment to the operation in the composite shooting mode of the imaging apparatus 10 described in the first embodiment. S701 to S708 are the same as S101 to S108 in FIG. 4, S711 to S714 are the same as S109 to S712 in FIG. 4, and S716 to S718 are the same as S113 to S115 in FIG.

In S<b>709 , the control unit 113 instructs the signal processing unit 111 to extract the OB area from the most recently captured frame image and supply it to the misregistration correction unit 110 . Further, when executing S709 first, the control unit 113 instructs the signal processing unit 111 to also extract the OB region of the reference image and supply it to the synthesizing unit 108 . The signal processing unit 111 extracts the OB area from the frame image according to the instruction from the control unit 113 and supplies it to the misregistration correction unit 110 .

In addition, the control unit 113 instructs the misregistration correction unit 110 to apply the registration processing to the OB region using the registration amount applied to the effective pixel region in the registration processing in S707, and to supply the result to the synthesizing unit 108. instruct. The misregistration correction unit 110 applies alignment processing to the OB area supplied from the signal processing unit 111 and supplies the result to the synthesizing unit 108 . As described above, in the alignment process, the position deviation correction unit 110 applies only the vertical component of the alignment amount when the OB region 201 is extracted, and when the OB region 202 is extracted, the alignment Applies only the horizontal component of the amount. The misregistration correction unit 110 supplies the OB area to which the alignment process has been applied to the synthesizing unit 108 .

In S710, the control unit 113 instructs the composition unit 108 to generate a composite image for line offset correction. The synthesizing unit 108 generates the OB area after alignment supplied from the misregistration correcting unit 110 most recently and further synthesizes it with the synthesized image stored in the memory 114 . When S710 is executed first, the synthesizing unit 108 synthesizes the aligned OB area supplied from the misregistration correcting unit 110 with the OB area of the reference image supplied from the signal processing unit 111 . Synthesis unit 108 stores the synthesized image of the OB area in memory 114 .

Here, a case has been described in which alignment processing and synthesis processing for generating a synthesized image for offset correction are executed after the synthesis processing of the OB area 301 and the effective pixel area 302 of the frame image for recording is completed. . However, the alignment processing and composition processing for generating the composite image of the OB area for offset correction may also be executed in S707 and S708.

After the OB clamping process of S714 is completed, the control unit 113 instructs the signal processing unit 111 to perform line offset correction in S715. The signal processing unit 111 reads the composite image for offset correction stored in the memory 114 and detects the presence or absence and position of the streak pattern. If no streak pattern is detected, the signal processing unit 111 notifies the control unit 113 of the detection result (not detected) and does not apply line offset correction. In this case, the control unit 113 executes S716 and instructs the WB processing unit 107 to execute white balance adjustment processing.

On the other hand, when a streak pattern is detected, the signal processing unit 111 detects its position and determines an offset correction value for each detected streak pattern. Then, the signal processing unit 111 applies line offset correction to the OB clamped synthesized image stored in the memory 114 . Since subsequent processing is the same as that of the first embodiment, description thereof is omitted.

In this embodiment, an image synthesized by aligning the OB areas is generated for line offset correction. Then, using the synthesized image of the OB area generated for line offset correction, line offset correction is applied to the synthesized image of the effective pixel area. As a result, in addition to the effect of the first embodiment, it is possible to appropriately correct a streaky pattern caused by variations in circuit characteristics of the imaging device.

(Other embodiments)
In the above-described embodiment, the case where a composite image is generated at the time of photographing by the imaging device has been described. However, as described above, the shooting function is not essential in the present invention. Therefore, instead of shooting in S103 (S703), the present invention can also be implemented in an image processing apparatus that acquires one frame of RAW data that has been shot and recorded over time and executes the processes after S104 (S704). is. In this case, the determination processing in S109 (S709) determines whether or not acquisition of the set number of frames has been completed.

Also, the line offset correction described in the second embodiment does not have to be executed at the time of photographing. For example, by recording the composite image of the OB area for line offset correction in association with the composite image of the effective pixel area, the line offset correction can be performed by a device different from the recording device at an arbitrary timing. be able to.

Alternatively, by recording the OB area as RAW data, it is possible to generate a composite image and perform line offset correction with an apparatus different from the recording apparatus at arbitrary timing.

Also, in the above-described embodiments, one or more of the configurations described as functional blocks separate from the image sensor 102 may be mounted on the image sensor 102 . The A/D conversion unit 103, the memory 114, the positional deviation detection unit 109, the positional deviation correction unit 110, and the synthesizing unit 108 can be mounted on the imaging device 102 by using the stacked imaging device as the imaging device 102. FIG.

In this case, the composite image of the effective pixel area can be generated by the imaging device 102 . A composite image of the OB area for line offset correction can also be generated in the imaging device 102 . The image pickup device 102 may output the composite image data of the effective pixel area and the composite image data of the OB area for line offset correction as separate image data, or combine them into one image data and output them. You may If the line offset correction is also possible within the imaging device, the imaging device does not need to output data of the synthesized image of the OB area for line offset correction.

Also, in the second embodiment, the composite image of the OB area for line offset correction is always generated, but it may be generated only when it is determined that line offset correction is necessary. The necessity of line offset correction can be determined, for example, by comparing shooting conditions such as ISO sensitivity and shutter speed at the time of shooting, and environmental conditions such as sensor temperature and environmental temperature, with predetermined threshold values.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The present invention is not limited to the content of the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 10... Imaging device, 102... Image sensor, 104... OB integrating part, 105... OB clamping part, 108... Synthesizing part, 109... Positional deviation detection part, 110... Positional deviation correction|amendment part, 113... Control part

Claims (11)

an acquisition means for acquiring image data of multiple frames;
alignment means for aligning the image data of the plurality of frames;
and synthesizing means for synthesizing the aligned image data of the plurality of frames, wherein
The image data includes first data of pixels in the effective pixel area of the imaging device, which is composed of signals corresponding to an arrangement of predetermined color components, and data of pixels outside the effective pixel area. includes second data,
The alignment means aligns the second data included in the plurality of frames of image data based on an alignment amount used for aligning the first data included in the plurality of frames of image data. match,
The synthesizing means generates data of a first synthesized image obtained by synthesizing the aligned first data and data of a second synthesized image obtained by synthesizing the second data aligned. death,
The image processing apparatus, further comprising correction means for correcting the data of the first synthesized image based on the data of the second synthesized image. 2. The correcting means corrects streak patterns generated in units of pixel lines in the first composite image due to an imaging device used to generate the image data of the plurality of frames. The image processing device according to . When the correction means corrects a streak pattern generated in a horizontal pixel line, the alignment means horizontally adjusts the amount of alignment used for aligning the first data included in the image data of the plurality of frames. 3. The image processing apparatus according to claim 2, wherein the second data is aligned using a vertical component out of the directional component and the vertical component. When the correction means corrects a streak pattern generated in a vertical pixel line, the alignment means horizontally adjusts the amount of alignment used for aligning the first data included in the image data of the plurality of frames. 4. The image processing apparatus according to claim 2, wherein the second data is aligned using a horizontal component out of the directional component and the vertical component. 5. The image processing apparatus according to claim 1, further comprising recording means for recording the data of the first composite image and the data of the second composite image in association with each other. . 6. The image processing according to any one of claims 1 to 5, wherein said synthesizing means further generates data of a third synthesized image obtained by synthesizing said second data not aligned. Device. 7. The image processing apparatus according to claim 6, wherein the data of the third synthesized image is used for adjusting the black level of the data of the first synthesized image. 8. The image processing apparatus according to any one of claims 1 to 7, wherein said second data is data of non-exposed pixels. an imaging device;
an image processing device according to any one of claims 1 to 8;
has
An imaging apparatus, wherein the image data of the plurality of frames are captured over time using the imaging device. An image processing method executed by an image processing device,
obtaining multiple frames of image data;
aligning the frames of image data;
combining the aligned frames of image data;
The image data includes first data of pixels in the effective pixel area of the imaging device, which is composed of signals corresponding to an arrangement of predetermined color components, and data of pixels outside the effective pixel area. includes second data,
Aligning the second data included in the plurality of frames of image data based on an alignment amount used for aligning the first data included in the plurality of frames of image data. aligning;
The synthesizing includes data of a first synthesized image obtained by synthesizing the aligned first data and data of a second synthesized image obtained by synthesizing the aligned second data. including generating
The image processing method, further comprising correcting the data of the first synthesized image based on the data of the second synthesized image. A program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 8.

Images