JP5822606B2 - Image processing apparatus and image processing apparatus control method - Google Patents

Description

  The present invention relates to an image processing apparatus capable of combining a plurality of images obtained by a plurality of photographing operations using a plurality of combining methods, and a control method for the image processing apparatus.

In recent years, digital cameras generally use a solid-state imaging device that can convert received light signals into electrical signals and store them, and shoot and record still images and moving images.
However, when this solid-state imaging device is exposed for a long time, dark current more than signal charge may be accumulated due to dark current generated by heat or the like. For this reason, when long-time exposure is performed using a solid-state imaging device, the S / N ratio is lowered and the image quality is lowered.

  In addition, since the amount of charge that can be accumulated inside the solid-state imaging device is very small, the accumulation of a large amount of dark current due to long-time exposure also narrows the dynamic range of the camera, leading to degradation of image quality.

Therefore, in order to realize long exposure using a solid-state imaging device, for example, in Patent Document 1, a technique corresponding to long exposure without causing deterioration in image quality by reading and accumulating imaging data obtained by multiple exposures. Is published.

JP-A-5-236422

  However, in Patent Document 1, depending on the shooting conditions, the amount of charge per solid-state image sensor increases, and adding only a plurality of pieces of image data may result in overexposure.

  In view of the above problems, the present invention can select whether to add and synthesize a plurality of pieces of image data or selectively output and synthesize the image data, and to perform the synthesis with a single circuit. It is an object of the present invention to provide a control method for an apparatus or an image processing apparatus.

In order to solve the above-described problems, an image processing apparatus according to the present invention includes an acquisition unit that acquires first and second image data and the second image data from the first image data. The first and second image data are compared by using the output from the addition / subtraction means and the addition / subtraction means for adding or subtracting the image data, and according to the comparison result Comparing means for selecting and outputting any one of the data; and first selecting means for selecting and outputting any one of the output of the comparing means and the output of the addition / subtraction means according to the photographing mode; , it has a, said comparison means includes an exclusive means for outputting an exclusive OR between values of a register that indicates the sign of the output from the subtraction means, the first based on the output from the exclusive means And any one of the second image data A second selection means for outputting a, characterized by chromatic and control means for controlling in response to the shooting mode the value of the register.

According to the control method of the image processing apparatus of the present invention, as described in claim 9, the acquisition unit acquires the first and second image data, and the addition / subtraction unit includes the first image data. And an addition / subtraction step for adding or subtracting the second image data from, and a comparison means, using the output from the addition / subtraction means, to compare the first and second image data, and according to a comparison result, A comparison step of selecting and outputting any one of the first and second image data, and a selection unit that outputs either one of the output of the comparison unit and the output of the addition / subtraction unit according to the shooting mode. It possesses a selection step of selecting and outputting, wherein the comparison means further includes an exclusive step for outputting an exclusive OR between values of a register that indicates the sign of the output from the subtraction means, the exclusion step In A second selection step of outputting one of said first and second image data on the basis of the force, to have a, and a control step of controlling according to the shooting mode the value of the register Features.

  According to the present invention, in multiple exposure photography, it is possible to select image composition based on pixel value comparison and multiple composition based on pixel addition according to the photographing mode, and a circuit that achieves such composition is small-scale. Can be obtained at

Block diagram of imaging device Block diagram of digital signal processor Block diagram of the image composition unit Addition comparison block diagram Image diagram of compositing process in comparative light mode Image diagram of compositing process in comparative dark mode Image diagram of composition processing in addition and addition average mode Illustration of pixel value

(First embodiment)
Hereinafter, specific examples to which the present invention is applied will be described in detail with reference to the drawings.

  As words used for explanation in the present embodiment, imaging data refers to digital data after AD conversion of sensor output. Development data refers to data obtained by synchronizing Bayer images and converting RGB to YCrCb based on imaging data, and compressed image data refers to data obtained by compressing development data by, for example, JPEG compression processing.

  FIG. 1 is a block diagram illustrating an overall configuration of an imaging apparatus that is an example of an image processing apparatus according to the first embodiment. The image processing apparatus to which the present invention can be applied is not limited to this, but may be a PC or a portable terminal as long as it can achieve a plurality of image composition methods as described later.

  A subject image formed by the optical system (100) is converted into an electrical signal for each pixel by the image sensor (101). The electrical signal output from the image sensor (101) is converted into a digital signal by the A / D converter (102). The imaging data acquired in this way is converted into development data by the digital signal processing unit (104).

  Further, the operation unit (103) is equipped with a shooting mode changeover switch and a shutter button, and the CPU (107) determines the setting state of the operation unit (103), thereby comparing the shooting mode (for example, in multiple exposure). (Bright, comparative dark, multiple addition, etc.).

  The CPU (107) switches the parameters of the digital signal processing unit (104) depending on the shooting mode.

  The imaging data or the development data output from the digital signal processing unit (104) is temporarily recorded in the memory (110) via the DATA bus (108) and the memory control unit (109).

  In the data compression / decompression unit (111), for example, JPEG compression processing is performed, and the development data recorded in the memory (110) as compressed image data is compressed and output. The compressed image data is recorded on the recording medium (113) via the DATA bus (108) and the recording control unit (112).

  The display control unit (114) performs predetermined processing such as resizing and gamma correction on the development data input via the DATA bus (108) and causes the display unit 115 to display the data.

  Although not shown, each unit including the memory control unit (109), the recording control unit (112), and the display control unit (114) operates according to a control signal from the CPU (107).

  FIG. 2 is a block diagram showing an internal configuration of the digital signal processing unit (104) in FIG.

  The imaging data input from the A / D conversion unit (102) is corrected by the sensor correction unit (203) to correct chromatic aberration generated in the optical unit (100) and gain variation of the element of the imaging element (101), and to detect defective pixels. Discrimination and correction are performed. The imaging data output from the sensor correction unit (203) can be recorded in the memory (110) via the DATA bus (108). The imaging data output from the sensor correction unit (203) is input to the image composition unit (201). The image composition unit (201) reads the image data recorded in the memory (110) via the DATA bus (108), and performs composition processing for each pixel with the image data input from the sensor correction unit (203).

  The multiplexed imaging data is subjected to development processing in the development processing unit (202) to generate development data. Here, the development processing refers to generating development data after RGB to YCrCb conversion by performing, for example, a Bayer image synchronization processing (interpolation processing), gamma correction processing, color space conversion processing, and the like. The development data generated by the development processing unit (202) is recorded in the memory (110) via the DATA bus (108).

In this embodiment, each block of the digital signal processing unit is realized by hardware, but the configuration other than the addition comparison unit (302) may be realized as a software module.
Here, the flow of shooting processing in the multiple exposure shooting mode will be described. The CPU (107) captures an image through the timing generator (106) and the sensor drive control unit (105) according to a control signal in accordance with photographing conditions such as exposure and focus set by the user or automatically determined by scene discrimination or the like. The element (101) is driven. The CPU (107) drives the optical system (100) via the lens drive control unit (116). The electrical signal from the image sensor is then input to the digital signal processor (104) as image data via the A / D converter (102).

  In the digital signal processing unit (104), the sensor correction unit (203) performs correction of chromatic aberration generated in the optical unit (100) and gain variation of the element of the image sensor (101), discrimination of defective pixels, correction processing, and the like. In the multiple-exposure shooting mode, the image data shot on the first image is temporarily recorded in the memory (110) via the DATA bus (108).

  The imaging data photographed after the second image is processed in the same manner as the first image, and when it is output from the sensor correction unit (203), it is input to the image composition unit (201) as the image 2 in FIG. Imaging data that has been captured and combined up to the previous time and recorded in the memory 110 is input as an image 1 to the image combining unit (201), and the image combining unit (201) performs image combining processing according to a combining method according to each mode. Is done. The composite image data output from the image composition unit (201) is developed by the development processing unit 202, and the developed development data is compressed by the data compression / decompression unit (111), via the recording control unit (112). Recording is performed on the recording medium (113). Alternatively, the development data is displayed on the display unit 115 via the display control unit (114).

  Next, details of the image composition processing in the image composition unit (201) will be described.

  FIG. 3 is a block diagram showing an internal configuration of the image composition unit (201) in FIG. The image composition unit (201) has two input ports, and inputs image 1 (first image data) and image 2 (second image data) to the image composition unit (300). The first gain adjustment unit (300) and the second gain adjustment unit (301) multiply the input image 1 and image 2, respectively. Specifically, in the averaging process, for example, when image 1 is a composite image up to the third image and image 2 is the fourth image, image 1: image 2 = 3: 1. Gain is multiplied. The addition comparison unit (302) performs pixel-by-pixel combination processing such as comparison and addition on the two input images, and outputs a combined image to the third gain adjustment unit (303). In the third gain adjusting unit (303), for example, a gain of ¼ is applied to the synthesized image obtained by synthesizing the third and fourth synthesized images from the addition comparing unit (302) as described above. Thus, it is possible to perform the addition averaging process described later.

FIG. 4 is a diagram showing the internal configuration of the addition comparison unit (302) in FIG.
Based on the parameters from the CPU (107) shown in FIG. 1, the addition comparison unit (302) can perform synthesis processing such as addition, addition average, comparatively bright, and comparatively dark. In the present embodiment, multiple exposure shooting is performed. The mode includes an addition mode, an addition average mode, a comparatively bright mode, and a comparatively dark mode.

I_i (x, y) (i = 1 to N, x, y represents the coordinates in the screen), and the pixel values of the image after the synthesis of these N images. Let I (x, y). At this time, the addition mode is
I (x, y) = I_1 (x, y) + I_2 (x, y) +... + I_N (x, y)
The sum of the pixel values of N images for each pixel is combined image data. The averaging mode is
I (x, y) = (I_1 (x, y) + I_2 (x, y) +... + I_N (x, y)) / N
The result of the average processing of the pixel values of N images for each pixel is composite image data. The comparative light mode is
I (x, y) = max (I_1 (x, y), I_2 (x, y),..., I_N (x, y))
The maximum value of the pixel values of N images selected for each pixel is the composite image data. The comparative dark mode is
I (x, y) = min (I_1 (x, y), I_2 (x, y),..., I_N (x, y))
The minimum value of the pixel values of N images selected for each pixel is composite image data.

In the present embodiment, N composite images are generated by repeating two composite processes by the addition comparison unit (302). That is, in addition mode,
I2 (x, y) = I_1 (x, y) + I_2 (x, y),
I3 (x, y) = I2 (x, y) + I_3 (x, y),
...
I (x, y) = IN (x, y) = IN-1 (x, y) + I_N (x, y)
It becomes. In addition, in addition average mode,
I2 (x, y) = (I_1 (x, y) + I_2 (x, y)) / 2
I3 (x, y) = I2 (x, y) / 3 + I_3 (x, y) / 3,
...
I (x, y) = IN (x, y) = IN-1 (x, y) / N + I_N (x, y) / N
It becomes. In comparative light mode,
I2 (x, y) = max (I_1 (x, y), I_2 (x, y)),
I3 (x, y) = max (I2 (x, y), I_3 (x, y)),
...
I (x, y) = IN (x, y) = max (IN−1 (x, y), I_N (x, y)),
It becomes. In comparative dark mode,
I2 (x, y) = min (I_1 (x, y), I_2 (x, y)),
I3 (x, y) = min (I2 (x, y), I_3 (x, y)),
...
I (x, y) = IN (x, y) = min (IN−1 (x, y), I_N (x, y))
It becomes.

  The addition / subtraction circuit (400) is a circuit for adding or subtracting the input 1 and the input 2, and outputs an MSB (Most Significant Bit) which is a sign of the subtraction result to EXOR (401). EXOR (401) is a logic circuit that outputs an exclusive OR with respect to the input of MSB and CMP_SEL from the addition / subtraction circuit (400). The selector 1 (402) (second selection means) and the selector 2 (403) (first selection means) selectively output one of the inputs. CMP_SEL and SUB_SEL are values output from the registers, and are values set by control signals from the CPU (107) in accordance with each mode.

  Here, the operation of each unit in the addition comparison unit (302) in each mode will be described in detail.

  In the addition mode and the addition average mode, the gain applied to each image data in the first and third gain adjustment units (300, 301, 303) before and after is different, and the processing in the addition comparison unit (302) does not change. . An addition process is performed on the image 1 input from the input 1 and the image 2 input from the input 2 in the addition / subtraction circuit (400). Since “1” is set in SUB_SEL, the addition result of image 1 and image 2 is output for each pixel as a result of the synthesis process from selector 2 (403) regardless of the setting of CMP_SEL.

  In the comparatively bright mode, the addition / subtraction circuit (400) performs subtraction processing on the image 1 input from the input 1 and the image 2 input from the input 2. Here, for example, the addition / subtraction circuit (400) adds the values of the input 1 and the input 2 as they are with an adder at the time of addition. Addition and subtraction are realized by a single adder. Thereafter, the MSB is input to EXOR (401) from the subtraction result of the addition / subtraction circuit (400). In the comparatively bright mode, CMP_SEL is set to '0', so that the selector 1 (402) is the smaller value of the input 1 and the input 2 based on the MSB in the subtraction result of the addition / subtraction circuit (400). Select to output. Since SUB_SEL is set to “0”, the output from the selector 1 (402) from the selector 2 (403), that is, the value of the image 1 and the image 2 is compared for each pixel and not smaller. A value is output for each pixel as a result of the synthesis process.

  In the comparative dark mode, the addition / subtraction circuit (400) performs subtraction processing on the image 1 input from the input 1 and the image 2 input from the input 2. Thereafter, the MSB is input to EXOR (401) from the subtraction result of the addition / subtraction circuit (400). In the comparative dark mode, since CMP_SEL is set to “1”, the result of inverting the MSB is input to the selector 1 (402), and the selector 1 (402) Select the value that does not exist and output it. Since SUB_SEL is set to “0”, the output from the selector 1 (402) from the selector 2 (403), that is, the value obtained by comparing the values of the image 1 and the image 2 for each pixel is not large. A value is output for each pixel as a result of the synthesis process.

  The composition of image 1 and image 2 in comparatively bright will be described with reference to FIG.

  5A is an image diagram of the image 1 input to the image composition unit 201, and FIG. 5B is an image diagram of the image 2. FIG.

  The pixel shown in FIG. 5 takes a value of 8 bits (0 to 255), and the white pixel as shown in FIG. 8 has a pixel value X (for example, 8 bit data) whose pixel value is 200 ≦ X. have. Further, it is assumed that the hatched pixels have a pixel value of 100 ≦ X <200, and the dot pattern pixels have a pixel value of X <100.

Each image is input from the upper left (A1) pixel and sequentially input in the horizontal direction.
That is, pixel data up to N8 is input in the order of A1, B1, C1... M1, N1, A2, B2, C2.

For example, in FIG. 4, a case where the K1 pixel (outlined pixel) in FIG. 5A is input to the input 1 and the K1 pixel (dotted pattern pixel) in FIG.
In this case, the two input pixels are compared by the addition / subtraction circuit (400) of FIG. 4, EXOR (401), and selector 1 (402), and the larger value, that is, K1 of FIG. Pixels are output from the image composition unit (300).

  FIG. 5C illustrates an image of a pixel comparison result at K1.

  FIG. 5D is an image diagram in which the values in the F1 pixel of the image 1 and the image 2 are similarly compared. In this case, since the F1 pixel of the image 2 has a large value, the pixel of the image 2 is selected and output as a result of the comparison.

  FIG. 5E shows an image composition result in comparatively bright when images as shown in FIG. 5A and FIG.

  Thus, for example, in the first shooting, lighting is performed from the lower right of the subject, and in the second shooting, shooting is performed from the upper left of the subject. Then, an image with higher gradation is obtained by lighting in both the lower right and the upper left by the image composition based on comparatively bright light according to the present embodiment. If this is a simple addition process, the overexposure will be overexposed. In the addition average process, the dynamic range will be reduced by gain adjustment or the like.

  The composite content of image 1 and image 2 in comparative darkness will be described with reference to FIG.

  6A is an image diagram of the image 1 input to the image composition unit 201, and FIG. 6B is an image diagram of the image 2. FIG.

Each image is input from the upper left (A1) pixel and sequentially input in the horizontal direction.
That is, pixel data up to N8 is input in the order of A1, B1, C1... M1, N1, A2, B2, C2.

  For example, in FIG. 4, it is assumed that the E2 pixel of FIG. 6A is input to the input 1 and the E2 pixel of FIG. 6B is input to the input 2, and the E2 pixel of the input 1 has a smaller value.

  In this case, the two input pixels are compared by the addition / subtraction circuit (400) of FIG. 4, EXOR (401), and selector 1 (402), and the smaller value, that is, E2 of FIG. Pixels are output from the image composition unit (300).

  FIG. 6C illustrates an image of a pixel comparison result in E2.

  FIG. 6D is an image diagram in which the values at the E8 pixel of image 1 and image 2 are similarly compared. In this case, since the E8 pixel of the image 2 has a small value, the pixel of the image 2 is selected and output as a result of the comparison.

  FIG. 6E shows an image composition result in comparative dark when images such as FIG. 6A and FIG. 6B are obtained by a plurality of photographings, in an image diagram.

  As described above, for example, when an overexposed pixel is generated in the first shooting, the second shooting is performed under a lighting shooting condition in which the overexposure is eliminated. By performing image synthesis using comparative darkness on these, it is possible to obtain a good image of overexposure. At this time, in a simple addition process, an overexposed pixel remains as it is, and in an addition average process, the dynamic range is also reduced by gain adjustment.

  The composition contents of image 1 and image 2 in the addition composition will be described with reference to FIG.

  7A is an image diagram of the image 1 input to the image composition unit 201, and FIG. 7B is an image diagram of the image 2. FIG.

Each image is input from the upper left (A1) pixel and sequentially input in the horizontal direction.
That is, pixel data up to N8 is input in the order of A1, B1, C1... M1, N1, A2, B2, C2.

  For example, in FIG. 4, it is assumed that the A1 pixel in FIG. 7A is input to the input 1 and the A1 pixel in FIG.

  In this case, the two input pixels are added by the addition / subtraction circuit (400) in FIG. 4, and the addition result is output from the image composition unit (300) as the pixel of A1. FIG. 7C illustrates an image of the pixel addition result in A1.

  FIG. 7 (d) shows an image composition result obtained by adding the image 1 obtained by a plurality of times of photographing, that is, FIG. 7A and the image 2, that is, FIG. 7B. Become.

  Further, the image output from the image composition unit (300) is output by the image synthesis unit (201) as an averaging process by halving the image output from the image composition unit (300) in the gain adjustment unit (301) shown in FIG. Is also possible.

  As described above, according to the present embodiment, in multiple exposure shooting, it is possible to select image composition by pixel value comparison and multiple composition by pixel addition according to the shooting mode, and achieve such composition. A circuit can be obtained on a small scale.

  In the present embodiment, the image data input to the image composition unit (300) as an image composition target image is output image data from the sensor correction unit (203). However, the present invention is not limited to this. For example, image data in a state in which the image data of the Bayer array shown in FIG. Alternatively, development processing such as gamma correction and color space conversion may be performed to convert the data into, for example, YCrCb format, that is, image data after development processing may be combined. In this case, in FIG. 2, a processing block that performs a part (including synchronization processing) or all of the processing processed by the development processing unit (202) is a front stage of the image composition unit (201) and is a sensor correction unit ( 203).

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

DESCRIPTION OF SYMBOLS 100 Optical system 101 Image pick-up element 102 A / D conversion part 103 Operation part 104 Digital signal processing part 105 Sensor drive control part 106 Timing generator 107 CPU
DESCRIPTION OF SYMBOLS 108 DATA bus 109 Memory control part 110 Memory 111 Data compression / decompression part 112 Recording control part 113 Recording medium 201 Sensor correction part 202 Image composition part 203 Development processing part 300 Image composition part 301 Gain adjustment part 400 Addition / subtraction circuit 401 EXOR circuit 402 Selector 1
403 Selector 2

Claims (11)

Acquisition means for acquiring first and second image data;
Addition / subtraction means for adding or subtracting the second image data from the first image data;
Compare the first and second image data using the output from the addition and subtraction means,
A comparing means for selecting and outputting one of the first and second image data according to a comparison result;
Have a, first selection means for selecting and outputting one of the output of said adder means and an output of the comparing means in accordance with the imaging mode,
The comparing means outputs an exclusive OR of a value indicating a sign of an output from the adding and subtracting means and a register value;
Second selection means for outputting one of the first and second image data based on the output from the exclusion means;
The image processing apparatus characterized by chromatic and control means for controlling in response to the shooting mode the value of the register.   2. The image processing according to claim 1, wherein the comparison unit compares the pixel values of the first and second image data for each pixel and outputs a value that is not smaller than the pixel value. apparatus.   The said comparison means compares the pixel value of the said 1st and said 2nd image data for every pixel, and outputs the value which is not the said pixel value large. Image processing device. When the photographing mode is addition or addition average mode, the addition / subtraction means adds the first and second image data, the first selection means selects and outputs the output of the addition / subtraction means,
When the photographing mode is a comparatively bright or dark mode, the addition / subtraction means subtracts the first and second image data, and the second selection means selects and outputs the output of the comparison means. The image processing apparatus according to any one of claims 1 to 3.   5. The image processing apparatus according to claim 1, wherein each of the first and second image data is image data including pixel values in a Bayer array.   5. The image data according to claim 1, wherein the first image data and the second image data are image data obtained by performing interpolation processing on image data each composed of pixel values in a Bayer array. 6. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the first and second image data are image data after development processing. Imaging means for capturing a subject image and outputting the first and second image data;
An image processing apparatus according to any one of claims 1 to 7,
An image pickup apparatus comprising: a recording unit that performs a predetermined process on an output from the first selection unit and records the output as a composite image data of the first and second image data on a recording medium. An acquisition step in which the acquisition means acquires the first and second image data;
An addition / subtraction step, wherein an addition / subtraction means adds or subtracts the second image data from the first image data;
The comparison means compares the first and second image data using the output from the addition / subtraction means, and selects one of the first and second image data according to the comparison result. A comparison step to output
Selection means, have a, a selection step of selecting and outputting one of the outputs of said subtracting means of said comparison means in accordance with the imaging mode,
The comparing means further outputs an exclusive OR of a value indicating a sign of the output from the adding and subtracting means and a register value;
A second selection step of outputting any one of the first and second image data based on the output in the exclusion step;
Method of controlling an image processing apparatus characterized by chromatic and a control step of controlling the value of the register according to the photographing mode.   A computer-executable program in which a procedure of a control method for an image processing apparatus according to claim 9 is described.   A computer-readable storage medium storing a program for causing a computer to execute each step of the control method of the image processing apparatus according to claim 9.

Images