JP2013009105A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of generating a visually natural wide dynamic range image.SOLUTION: A pixel value conversion unit 51 converts pixel values of image signals obtained by imaging an identical subject a plurality of times with mutually different exposure amounts on the basis of pixel value conversion coefficients Coa and Cob. A composition weighting coefficient generation unit 52 generates composition weighting coefficients. A smoothing processing unit 56 smooths short-time exposure image data Da1 and long-time exposure image data Db2 within a screen. A difference generation unit 53 generates a difference between the smoothed data pieces. A weighting composition unit 54 combines the short-time exposure image data Da1 and the long-time exposure image data Db2 using the composition weighting coefficients. A control unit 11 evaluates the difference, and updates the pixel value conversion coefficients so that the difference becomes smaller.

Description

  The present invention relates to an image processing apparatus and an image processing method for generating a composite image in which a dynamic range is expanded by combining images of a plurality of frames having different exposure amounts.

  While human vision has a very wide dynamic range, an imaging device such as a video camera has a narrow dynamic range. For this reason, when a main subject such as a person is photographed at the same time as the background, the main subject may be photographed in a dark state, or the background may be photographed with a white background and may be photographed in a different state from the appearance. . Therefore, a technique is known in which the same subject is photographed a plurality of times under different photographing conditions such as different exposure amounts, and the dynamic range of the image is expanded by combining a plurality of frames of images.

  When a wide dynamic range image is generated by combining a plurality of frames of images, it is necessary to match the pixel values of the images to be combined based on the exposure amount ratio at the time of shooting. However, in actual photographing, it is difficult to photograph with a set exposure amount ratio, and pixel values become discontinuous at a boundary portion where images when a plurality of frames are combined are switched. For example, Patent Documents 1 and 2 describe techniques for reducing discontinuity of pixel values at the boundary.

JP 2002-223350 A JP 2010-273001 A

  In Patent Documents 1 and 2, when combining a plurality of images, the pixel values of the images to be combined are matched using an image histogram. If processing with a large calculation amount such as a histogram is used to perform highly accurate calculation, the circuit scale increases. An increase in circuit scale causes an increase in cost. When synthesizing moving images in a video camera that shoots moving images, the state of the subject changes every moment, so if you try to quickly respond to the change of the subject and match the pixel value at the boundary, A high-speed arithmetic element is required. If processing with a large amount of computation is to be performed at high speed, it is difficult to avoid an increase in circuit scale and cost.

  In view of such a problem, the present invention can make the pixel values at the boundary when combining images of a plurality of frames continuous without increasing the circuit scale and the amount of calculation so much, and is visually natural. An object of the present invention is to provide an image processing apparatus and an image processing method capable of generating a wide dynamic range image.

  In order to solve the above-described problems of the prior art, the present invention provides the first plurality of image signals obtained by inputting the first plurality of image signals obtained by photographing the same subject a plurality of times with different exposure amounts. A pixel value of at least one of the first plurality of image signals is converted based on a pixel value conversion coefficient so as to match a difference in pixel values in the plurality of one image signal; A pixel value conversion unit (51) that outputs a plurality of image signals, and a combination weighting coefficient that is used when combining the second plurality of image signals with a region including pixels having a predetermined pixel value as a boundary portion. A synthesis weighting coefficient generation unit (52), a smoothing processing unit (56) for smoothing each of the second plurality of image signals within a screen and outputting them as a third plurality of image signals, and the third plurality of image signals The difference of the image signal of A difference generation unit (53) for generating, a weighting synthesis unit (54) for generating a synthesized image signal by weighting and synthesizing the second plurality of image signals using the synthesis weighting coefficient, and the difference generation And a parameter updating unit (11) that evaluates the difference generated by the unit and updates the pixel value conversion coefficient so that the difference is reduced.

  In the above configuration, it is preferable that the parameter updating unit updates the synthesis weighting coefficient in conjunction with updating the pixel value conversion coefficient.

  In the above configuration, the boundary portion is a region including pixels having a plurality of pixel values, and when the largest pixel value of the boundary portion is an upper limit value and the smallest pixel value of the boundary portion is a lower limit value, The synthesis weighting coefficient generation unit is configured to combine the first and second image signals in the second plurality of image signals as a synthesis weighting coefficient in a portion larger than the upper limit value, in the first and second image signals. Of the first and second image signals in a portion smaller than the lower limit value, and between the upper limit value and the lower limit value, the first and second image signals. It is preferable to generate a composite weighting coefficient that cross-fades and mixes the image signals.

  In the above-described configuration, the parameter update unit updates the upper limit value and the lower limit value so that the interval between the upper limit value and the lower limit value becomes narrow, and updates the pixel value conversion coefficient. Preferably generates a composite weighting coefficient based on the updated upper limit value and the lower limit value.

  In the above-described configuration, the smoothing processing unit has a degree of smoothing for an image signal with a small amount of exposure among the plurality of second image signals larger than a degree of smoothing for an image signal with a large amount of exposure. It is preferable to do.

  The parameter update unit may evaluate a difference between the third plurality of image signals in a partial region in the screen.

  In order to solve the above-described problems of the conventional technology, the present invention generates the first plurality of image signals by photographing the same subject a plurality of times with different exposure amounts (S1), and the exposure amount. Based on the pixel value conversion coefficient, the pixel value of at least one of the first plurality of image signals is determined based on the pixel value conversion coefficient so as to match the difference in pixel values in the first plurality of image signals. Convert to generate a second plurality of image signals (S6), and generate a combination weighting coefficient used when combining the second plurality of image signals with a region including pixels having a predetermined pixel value as a boundary portion. Then, each of the second plurality of image signals is smoothed in a screen to generate a third plurality of image signals, a difference between the third plurality of image signals is generated, and the second plurality of image signals are generated. A plurality of image signals are used with the composite weighting coefficient. A composite image signal is generated by weighting and synthesis (S8), the difference is evaluated, and the pixel value conversion coefficient is updated so that the difference becomes small (S504). To do.

  According to the image processing apparatus and the image processing method of the present invention, the pixel values at the boundary when combining the images of a plurality of frames can be made continuous without increasing the circuit scale or the amount of calculation so much. It is possible to generate a natural wide dynamic range image.

It is a block diagram which shows 1st, 2nd, 4th embodiment. 6 is a diagram for explaining an operation of a pixel value conversion unit 51. FIG. It is a figure for demonstrating operation | movement of the multiple exposure image synthetic | combination part 501,503,506,507. It is a figure for demonstrating the weighting synthetic | combination operation | movement in the weighting synthetic | combination part. It is a flowchart which shows the whole operation | movement of each embodiment. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₁ according to the first embodiment. It is a flowchart which shows the specific process of step S7 in FIG. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₂ according to the second embodiment. It is a block diagram which shows 3rd Embodiment. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₃ according to the third embodiment. It is a block diagram which shows 4th Embodiment. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₄ according to the fourth embodiment. It is a block diagram which shows 5th Embodiment. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₅ according to the fifth embodiment. It is a block diagram which shows 6th Embodiment. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₆ according to the sixth embodiment. It is a block diagram which shows 7th Embodiment. A step S5 in FIG. 5 is a flowchart illustrating a specific process of step S5 ₇ according to the seventh embodiment.

  Hereinafter, embodiments of an image processing apparatus and an image processing method of the present invention will be described with reference to the accompanying drawings. Each embodiment shows an example in which an image processing apparatus is mounted on a video camera.

<First Embodiment>
In FIG. 1, a video camera 101 includes an imaging unit 1, an input signal processing unit 2, a multiple exposure image output unit 3, a buffer memory 4, a multiple exposure image synthesis unit 501, a dynamic range compression unit 6, a buffer memory 7, and a compression / decompression process. A unit 8, an output signal processing unit 9, a display unit 10, a control unit 11, a RAM 12, a ROM 13, a recording / reproducing unit 14, and an operation switch 15 are provided. The multiple exposure image synthesis unit 501 includes a pixel value conversion unit 51, a synthesis weighting coefficient generation unit 52, a difference generation unit 53, and a weighting synthesis unit 54. The pixel value conversion unit 51 includes pixel value conversion units 51a and 51b.

  The imaging unit 1 includes an optical system such as a lens and a diaphragm, an imaging device such as a CCD and a CMOS, an exposure and focus control mechanism, and the like. The imaging unit 1 takes out optical information input to the imaging device via the lens as an electrical signal at a predetermined timing, and supplies the electrical signal to the input signal processing unit 2. The imaging unit 1 generates a dynamic range image wider than the dynamic range of the imaging device based on the control by the control unit 11, so that the same subject is shot multiple times under different imaging conditions, so-called multiple exposure. Do. In the first embodiment, shooting is performed with different exposure amounts as different shooting conditions.

  In order to vary the exposure amount, the exposure time may be varied by adjusting the shutter speed, the aperture value may be adjusted, or the two may be combined. In the first embodiment, two images, that is, an image obtained by short-time exposure with a higher shutter speed and an image obtained by long-time exposure at a lower shutter speed are taken and combined. Note that it is preferable to shoot while controlling exposure so that the shutter speed is fixed in accordance with the frame rate when shooting with long exposure and the high luminance portion of the short exposure image is not significantly saturated.

  The imaging unit 1 supplies the electrical signal of the short exposure image and the electrical signal of the long exposure image to the input signal processing unit 2 in either order. The input signal processing unit 2 performs A / D conversion on the input electrical signal, converts it into RGB three primary color signals, performs various signal processing such as white balance adjustment, gain adjustment, gamma processing, etc. Y and color difference signals Cb and Cr are converted. The luminance signal Y and color difference signals Cb and Cr of the short exposure image and the luminance signal Y and color difference signals Cb and Cr of the long exposure image are sequentially input to the multiple exposure image output unit 3. When the RGB signal is converted into the luminance signal Y and the color difference signals Cb and Cr, the amount of data is reduced, so that the required memory capacity can be reduced. At this time, the data amount can be further reduced by limiting the bands of the color difference signals Cb and Cr and performing time-axis multiplexing.

  The multiple exposure image output unit 3 delays the previously input luminance signal Y and color difference signals Cb and Cr by one vertical scanning period using the buffer memory 4. Then, the multiple exposure image output unit 3 includes the luminance signal Y and the color difference signals Cb and Cr (hereinafter referred to as the short exposure image data Da) of the short exposure image, and the luminance signal Y and the color difference signal Cb of the long exposure image. , Cr (hereinafter referred to as long-time exposure image data Db) are supplied to the multiple exposure image composition unit 501 at the same timing. At this time, it is preferable to correct the screen blur of the short-exposure image data Da and the long-exposure image data Db by controlling the read start address of the data from the buffer memory 4 using the motion vector.

  The short exposure image data Da is input to the pixel value converter 51a, and the long exposure image data Db is input to the pixel value converter 51b. The pixel value conversion coefficient Coa is set by the control unit 11 in the pixel value conversion unit 51a, and the pixel value conversion coefficient Cob is set by the control unit 11 in the pixel value conversion unit 51b. Pixel value conversion coefficients Coa and Cob are one of the parameters used for image composition. Hereinafter, the conversion coefficients are abbreviated as Coa and Cob.

  As will be described later, the conversion coefficients Coa and Cob are changed from the temporarily set conversion coefficients to sequentially corrected conversion coefficients. One of the conversion coefficients Coa and Cob may be a fixed conversion coefficient, and the other may be changed to a sequentially corrected conversion coefficient. In the first embodiment, the conversion coefficient Coa for the short-time exposure image data Da is fixed, and the conversion coefficient Cob for the long-time exposure image data Db is sequentially changed.

  It is assumed that the short exposure image data Da and the long exposure image data Db are, for example, 10-bit digital data. Based on the conversion coefficient Coa, the pixel value conversion unit 51a multiplies the input 10-bit short-time exposure image data Da by 64 and converts it into 16-bit digital data. In the first embodiment, the exposure is controlled so that the high-intensity portion of the short-time exposure image is not significantly saturated, so that the pixel value of the short-time exposure image data Da is normalized to the full dynamic range after image synthesis. A fixed gain value is set as the conversion coefficient Coa. FIG. 2A shows a characteristic a1 of the short-time exposure image data Da1 after conversion to 16 bits. The horizontal axis represents the brightness of the subject, and the vertical axis represents the pixel value (luminance value).

  The pixel value converter 51b first converts the input 10-bit long exposure image data Db into 16-bit digital data. FIG. 2B shows the characteristic b1 of the long-time exposure image data Db1 after conversion to 16 bits. As shown in FIG. 2B, the characteristic b1 of the long exposure image data Db1 is a characteristic that the pixel value is saturated at a predetermined brightness or more. Next, the pixel value conversion unit 51b matches the inclination of the non-saturated region of the characteristic b1 with the inclination of the characteristic a1 and matches the pixel values of the characteristic b1 with the short-time exposure image data Da (Da1). The pixel value is converted in accordance with the exposure amount ratio with the long-time exposure image data Db (Db1) to obtain long-time exposure image data Db2 having the characteristic b2. If the exposure amount of the long exposure image data Db is n times the exposure amount of the short exposure image data Da, the pixel value of the characteristic b1 may be multiplied by 1 / n.

  The reason why the pixel value of the long exposure image data Db1 is converted according to the exposure amount ratio is to map the long exposure image data Db2 to the region of the short exposure image data Da1 normalized as described above. . The conversion coefficient Cob for the long-time exposure image data Db is a conversion coefficient for converting the pixel value according to the bit number conversion and the exposure amount ratio.

  It is not essential for the pixel value converters 51a and 51b to convert the short-time exposure image data Da and the long-time exposure image data Db from 10 bits to 16 bits. The pixel value conversion units 51a and 51b match each other with the inclination of the characteristic of the pixel value with respect to the brightness of the subject with respect to at least one of the short-time exposure image data Da and the long-time exposure image data Db. The pixel values may be converted in accordance with the exposure amount ratio so as to match.

  In FIGS. 2A and 2B, gamma correction is omitted and the characteristic a1 is indicated by a straight line and the characteristics b1 and b2 are indicated by straight broken lines. In practice, however, the pixel value conversion units 51a and 51b The input short exposure image data Da and long exposure image data Db are subjected to gamma correction. Therefore, the control unit 11 determines the amount of change in pixel value by standard gamma conversion for each of the short-time exposure image data Da and the long-time exposure image data Db. The control unit 11 sets a temporary correction value based on the obtained change amount, and corrects the conversion coefficient Cob by adding the temporary correction value to the conversion coefficient Cob having the exposure amount ratio as a gain.

  The correction of the conversion coefficient Cob considering the gamma correction will be described with reference to FIG. FIG. 3 is an enlarged view of the vicinity of the points Pa and Pb that are the boundary portions of the synthesis in FIGS. 2 (A) and 2 (B). FIG. 3 shows the characteristics a1 and b2 in a state where the gamma correction is performed. The characteristic b2 is a characteristic of the long exposure image data Db2 in which the pixel value is converted by the conversion coefficient Cob in a state where no correction value is added. The control unit 11 sets an upper limit value ThH on the high luminance side and a lower limit value ThL on the low luminance side as a range for providing a boundary portion when the short exposure image data Da1 and the long exposure image data Db2 are combined. The boundary portion is an area including pixels having a plurality of pixel values, where the upper limit value ThH is the maximum pixel value and the lower limit value ThL is the minimum pixel value.

  In FIGS. 2A and 2B, for the sake of convenience, the images are combined at points Pa and Pb. However, the short-exposure image data Da1 and the long-exposure image data Db2 have an upper limit value ThH and a lower limit value ThL. The boundary between them is synthesized as a synthesis area. The upper limit value ThH and the lower limit value ThL are one of parameters used for image composition.

  As an example, as an initial provisional setting, 3/4 is set to the upper limit value ThH and 1/2 is set to the lower limit value ThL for the pixel value in the saturation region of the characteristic b2. Strictly speaking, the pixel value in the saturation region is not a constant value, but may be set as appropriate. The range that the data of the long-time exposure image data Db2 can take can be calculated by the ratio of the conversion coefficients Coa and Cob set in the pixel value conversion units 51a and 51b, and the pixel value of the saturated region can also be calculated. Therefore, the control unit 11 can set the upper limit value ThH and the lower limit value ThL to appropriate pixel values, respectively. As will be described later, the upper limit value ThH and the lower limit value ThL are corrected so as to narrow the synthesis region sequentially.

  In the first embodiment, the conversion coefficient Cob is corrected so that the long-time exposure image data Db2 indicated by the characteristic b2 has a characteristic b20 that is close to the center pixel value of the upper limit value ThH and the lower limit value ThL. Note that although the correction amount to be corrected to the center pixel value differs depending on the brightness of the subject shown on the horizontal axis in FIG. 3, the correction amount may be such that the average value is displaced to the center pixel value.

  In the pixel value conversion units 51a and 51b, one of the short-time exposure image data Da and the long-time exposure image data Db may be offset to convert the pixel value. When the same subject is shot and combined three or more times under different imaging conditions, it may be difficult to match pixel values only by changing the gain. In this case, it is preferable to use a pixel value offset together.

  The multiple exposure image combining unit 501 combines the short-time exposure image data Da1 and the long-time exposure image data Db2 whose pixel values are converted as described above. Conceptually, the multiple exposure image composition unit 501 includes an area ARb from the pixel value 0 to the pixel value of the predetermined point Pb in the characteristic b2 of the long exposure image data Db2 shown in FIG. A region ARa equal to or greater than the pixel value of the point Pa in the characteristic a1 of the short-time exposure image data Da1 shown in A) is synthesized. The area ARb corresponds to the solid line part of the characteristic b20 in FIG. 3, and the area ARa corresponds to the solid line part of the characteristic a1. The multiple exposure image composition unit 501 makes the pixel value at the point Pa and the pixel value at the point Pb coincide as much as possible by an operation to be described later, and makes the pixel values at the boundary portion continuous.

  The short-time exposure image data Da1 and the long-time exposure image data Db2 obtained by converting the pixel values output from the pixel value conversion units 51a and 51b are input to the synthesis weighting coefficient generation unit 52, the difference generation unit 53, and the weighting synthesis unit 54. Is done. The synthesis weighting coefficient generation unit 52 generates a synthesis weighting coefficient used when synthesizing the short exposure image data Da1 and the long exposure image data Db2. The synthetic weighting coefficient Wa for the short-time exposure image data Da1 and the synthetic weighting coefficient Wb for the long-exposure image data Db2 are set as the synthetic weighting coefficient Wa. The combination weighting coefficients Wa and Wb are one of parameters used for image combination.

  An upper limit value ThH and a lower limit value ThL are input to the composite weighting coefficient generation unit 52 from the control unit 11. The composite weighting coefficient generation unit 52 generates the composite weighting coefficient Wa and the composite weighting coefficient Wb with reference to one of the short-time exposure image data Da1 and the long-time exposure image data Db2. First, a case where long exposure image data Db2 is used as a reference will be described. The pixel position in the frame in the long-time exposure image data Db2 is (i, j), and the pixel value at each position (i, j) is Yb (i, j). The pixel value here is a luminance value. The combined weighting coefficients Wb and Wa at the position (i, j) of each pixel can be expressed by the following equations (1) and (2).

  When the short-exposure image data Da1 is used as a reference, the position of the pixel in the frame in the short-exposure image data Da1 is similarly (i, j), and the pixel value (luminance at each position (i, j) is set. Value) is assumed to be Ya (i, j). The synthesis weighting coefficient Wa and the synthesis weighting coefficient Wb at the position (i, j) of each pixel can be expressed by the following equations (3) and (4). The reference image signal may be either the long exposure image data Db2 or the short exposure image data Da1. That is, the synthesis weighting factors Wa and Wb may be obtained by the equations (1) and (2), or may be obtained by the equations (3) and (4).

  The synthesis weighting coefficients Wa and Wb obtained as described above are input to the weighting synthesis unit 54.

  The difference generation unit 53 generates a difference value of the luminance signal Y between the input short-time exposure image data Da1 and the long-time exposure image data Db2. The control unit 11 sets a positive threshold value and a negative threshold value for the difference value. The difference generation unit 53 counts the difference values exceeding the positive threshold value and the negative threshold value, and generates a difference count value Ddfc.

  The difference generation operation in the difference generation unit 53 will be described with reference to FIG. The difference generator 53 obtains a difference between the short exposure image data Da1 and the long exposure image data Db2 with reference to either one. At this time, it is preferable to obtain the difference within the range between the upper limit value ThH and the lower limit value ThL input from the control unit 11. When the short-exposure image data Da1 is used as a reference, the difference generation unit 53, as shown in FIG. 3, outputs the pixel value (luminance value) of the point Pa1 corresponding to the lower limit ThL of the short-exposure image data Da1 indicated by the characteristic a1. ) To the pixel value (luminance value) of the point Pa2 corresponding to the upper limit value ThH, the difference for each pixel position in the frame from the long-exposure image data Db2 indicated by the characteristic b20 is obtained.

  The difference generation unit 53 subtracts the pixel value of the point Pb1 of the long-time exposure image data Db2 that is the position of the pixel in the frame corresponding to the point Pa1 from the pixel value of the point Pa1 to obtain a difference value. The above is an example of a case where the pixel value of the long-time exposure image data Db2 at the corresponding pixel position is Pb1. Similarly, the difference generation unit 53 subtracts the pixel value of the point Pb2 of the long-time exposure image data Db2 that is the position of the pixel in the frame corresponding to the point Pa2 from the pixel value of the point Pa2 to obtain a difference value. The above is an example of the case where the pixel value of the long-time exposure image data Db2 at the corresponding pixel position is Pb2. Although not shown in FIG. 3, the difference generation unit 53 also obtains a difference value for each pixel located between the point Pa1 and the point Pa2. The difference generation unit 53 generates a difference count value Ddfc that exceeds the positive threshold value and the negative threshold value based on the difference value thus obtained.

  The difference count value Ddfc is generated in the synthesis region between the upper limit value ThH and the lower limit value ThL. The conversion coefficient is obtained by adding a correction value temporarily set based on the change amount of the pixel value by the standard gamma conversion as described above. This is for determining whether Cob is appropriate. If the difference count value Ddfc is large, it indicates that the short-time exposure image data Da1 indicated by the characteristic a1 and the long-time exposure image data Db2 indicated by the characteristic b20 are misaligned in the composite region. In this case, it is necessary to further correct the conversion coefficient Cob. The difference generation unit 53 generates a difference value only in the combined region between the upper limit value ThH and the lower limit value ThL and obtains the difference count value Ddfc, thereby eliminating errors other than the combined region and calculating the difference in the combined region. It can be determined accurately. In addition, since the difference between unnecessary portions is not obtained, the calculation is simplified and the amount of data can be reduced.

  The difference count value Ddfc generated by the difference generation unit 53 is input to the control unit 11. The control unit 11 uses the difference count value Ddfc obtained as described above as an example of determining whether or not the difference between the short-exposure image data Da1 and the long-exposure image data Db2 is large. If the difference count value Ddfc is large, the short-time exposure image data Da1 and the long-time exposure image data Db2 are shifted.

  The control unit 11 corrects the conversion coefficient Cob supplied to the pixel value conversion unit 51b according to the difference count value Ddfc. As a result of correcting the conversion coefficient Cob, when the count value changes in a decreasing direction, the control unit 11 sets the lower limit value to the upper limit value ThH so that the interval between the upper limit value ThH and the lower limit value ThL is narrowed. Update to bring ThL closer. When the upper limit value ThH and the lower limit value ThL are updated, the synthesis weighting coefficients Wa and Wb generated by the synthesis weighting coefficient generation unit 52 are updated. The control unit 11 operates as a parameter updating unit that updates parameters of the conversion coefficient Cob, the upper limit value ThH and the lower limit value ThL, and the synthesis weighting coefficients Wa and Wb.

  The weighting synthesis unit 54 performs weighting synthesis of the short-time exposure image data Da1 and the long-time exposure image data Db2 supplied from the pixel value conversion units 51a and 51b by using the synthesis weighting coefficients Wa and Wb. FIG. 4 conceptually shows weighting synthesis in the weighting synthesis unit 54. The horizontal axis is the pixel value, and the vertical axis is the mixing ratio. That is, a case is shown in which the synthesis weighting factors Wa and Wb are generated by the equations (1) and (2). The mixing ratio 1 is 100%, which indicates that only one of the short-time exposure image data Da1 and the long-time exposure image data Db2 is used.

  As shown in FIG. 4, the weighting / composing unit 54 sets only the long-time exposure image data Db2 (solid line) from the pixel value 0 to the lower limit value ThL, and the long-time exposure image data Db2 from the lower limit value ThL to the upper limit value ThH. Are mixed in such a manner that the short-time exposure image data Da1 (broken line) is sequentially increased from 0 to 1 while sequentially decreasing from 1 to 0. If the upper limit value ThH is exceeded, the weighting combining unit 54 sets only the short-time exposure image data Da1. It should be noted that the equation used when Yb (i, j) in Equations (1) and (3) is greater than or equal to ThL and less than or equal to ThH is the slope when cross-fading the short-time exposure image data Da1 and the long-time exposure image data Db2 in FIG. Represents.

  Note that the weighting / synthesizing unit 54 weights and synthesizes the short-time exposure image data Da1 and the long-time exposure image data Db2 using the synthesis weighting coefficients Wa and Wb with the luminance signal Y and the color difference signals Cb and Cr, respectively. The combination weighting coefficients Wa and Wb generated based on the luminance signal Y may be used for combining the color difference signals Cb and Cr.

  The control unit 11 corrects the conversion coefficient Cob with the difference count value Ddfc evaluated in the immediately preceding frame as described above, updates the upper limit value ThH and the lower limit value ThL, and updates the composite weighting coefficients Wa and Wb. Repeat for each frame of the composite image. Then, the conversion coefficient Cob is sequentially optimized, and the pixel values of the short-exposure image data Da1 and the long-exposure image data Db2 gradually approach each other in the synthesis region between the upper limit value ThH and the lower limit value ThL. In addition, since the synthesis area is gradually narrowed, the short-time exposure image data Da1 and the long-time exposure image data Db2 are mixed in a very narrow range at the center between the upper limit value ThH and the lower limit value ThL. As a result, the synthesized image has a continuous wide pixel range at the boundary and becomes a natural wide dynamic range image.

Returning to FIG. 1, the wide dynamic range image data D _HDR output from the weighting synthesis unit 54 is input to the dynamic range compression unit 6. The dynamic range compression unit 6 compresses the dynamic range in order to convert the wide dynamic range image data D _HDR into image data that can be handled by a normal image processing circuit. In the first embodiment, the dynamic range compression unit 6 compresses the dynamic range of the wide dynamic range image data D _HDR by gradation correction based on the Retinex theory, and generates compressed wide dynamic range image data D _CHDR .

The buffer memory 7 is a memory that temporarily holds the wide dynamic range image data D _HDR or the compressed wide dynamic range image data D _CHDR . The compression / decompression processing unit 8 compresses the compressed wide dynamic range image data D _CHDR held in the buffer memory 7 by a compression method such as JPEG or MPEG, and generates encoded data. The encoded data is recorded in the recording / reproducing unit 14 by the control unit 11. Further, the compression / decompression processing unit 8 decompresses the encoded data when the control unit 11 reads the encoded data recorded in the recording / reproducing unit 14.

The output signal processing unit 9 _{converts the} compressed wide dynamic range image data D _CHDR into a format that can be displayed on the display unit 10 and inputs the converted data to the display unit 10. Further, the output signal processing unit 9 converts the compressed wide dynamic range image data D _CHDR in accordance with the format and size of the external device so as to be supplied to an external device such as a display device (not shown). As an example, the output signal processing unit 9 converts the compressed wide dynamic range image data D _CHDR into an NTSC signal.

  The control unit 11 controls the entire video camera 101. The control unit 11 can be configured by a CPU. As described above, the control unit 11 generates the conversion coefficients Coa and Cob and supplies them to the pixel value conversion units 51a and 51b, generates the upper limit value ThH and the lower limit value ThL, and generates the combined weighting coefficient generation unit 52 and the difference generation. To the unit 53. The control unit 11 also generates or sets various other parameters necessary for the image composition process. In addition, the control unit 11 generates character data to be superimposed on image data to be displayed on the display unit 10 or image data to be externally output, and supplies the generated character data to the output signal processing unit 9.

  A RAM 12 that is a working memory of the control unit 11 is connected to the control unit 11. The ROM 13 stores a program for operating the control unit 11. The control unit 11 controls the video camera 101 based on a program stored in the ROM 13. The recording / reproducing unit 14 is an arbitrary recording medium such as a hard disk drive or a semiconductor memory, and the recording / reproducing is controlled by the control unit 11. An operation switch 15 is connected to the control unit 11, and an operation signal from the operation switch 15 is input to the control unit 11. The operation switch 15 may be a remote controller.

  The image processing apparatus according to the first embodiment may be configured by hardware or software (computer program). You may comprise both together.

  The image processing operation executed by the video camera 101 will be further described with reference to the flowcharts shown in FIGS. FIG. 5 shows the entire image processing operation executed in the first embodiment and second to seventh embodiments described later. In FIG. 5, the imaging unit 1 performs multiple exposure on the same subject at a set shutter speed in step S1. In step S2, the input signal processing unit 2 captures the multiple exposure image signal. In step S3, the input signal processing unit 2 generates RGB signals and converts them into Y, Cb, and Cr signals. In step S4, the multiple exposure image output unit 3 aligns and outputs the short exposure image data Da and the long exposure image data Db. In step S5, the control unit 11 calculates conversion coefficients Coa and Cob. Specific processing in step S5 will be described later.

  In step S6, the pixel value converters 51a and 51b convert the pixel values of the short-time exposure image data Da and the long-time exposure image data Db based on the conversion coefficients Coa and Cob. In step S7, the synthesis weighting coefficient generation unit 52 calculates synthesis weighting coefficients Wa and Wb for synthesizing the short exposure image data Da1 and the long exposure image data Db2 whose pixel values have been converted. Specific processing in step S7 will be described later. In step S8, the weighting / synthesizing unit 54 synthesizes the short-time exposure image data Da1 and the long-time exposure image data Db2 by weighting them using the synthesis weighting coefficients Wa and Wb.

In step S9, the dynamic range compression unit 6 compresses the dynamic range of the wide dynamic range image data D _HDR . In step S 10, the control unit 11 compresses the compressed wide dynamic range image data D _CHDR and records the compressed wide dynamic range image data D _CHDR in the recording / reproducing unit 14 after encoding, or displays the compressed wide dynamic range image data D _CHDR on the display unit 10. To do. In step S <b> 11, the control unit 11 determines whether or not the image composition instruction has ended. Whether or not to synthesize the multiple-exposed images can be switched by operating the operation switch 15. If it is determined that the image compositing instruction has not ended, the process returns to step S1, and if it is determined that the instruction has ended, the operation of the compositing process ends.

Details of the conversion coefficient Coa and Cob calculation process in step S5 of FIG. 5 will be described with reference to FIG. The step S5 to be executed in the first embodiment is referred to as step S5 _1. In FIG. 6, the control unit 11 temporarily sets a conversion coefficient Coa for the short-time exposure image data Da in step S501. As described above, the conversion coefficient Coa may be a fixed gain value for converting the 10-bit short-time exposure image data Da into 16-bit digital data. Although temporarily set, the conversion coefficient Coa is not updated in the first embodiment. In step S502, the control unit 11 temporarily sets a conversion coefficient Cob for the long-time exposure image data Db. As described above, the conversion coefficient Cob is a coefficient for converting the 10-bit long-exposure image data Db into 16-bit digital data, and further converting the pixel value according to the exposure amount ratio.

  In step S503, the control unit 11 corrects the conversion coefficient Cob by the amount of gain change due to gamma processing. The correction here is provisional correction based on the change amount of the pixel value by the standard gamma conversion for each of the short-time exposure image data Da and the long-time exposure image data Db. In step S504, the control unit 11 additionally corrects the conversion coefficient Cob based on the evaluation result of the difference value Ddf between the short exposure image data Da1 and the long exposure image data Db2 in steps S509 and S511.

  In step S505, the control unit 11 sets a positive threshold value and a negative threshold value for the difference value generated by the difference generation unit 53. In step S506, the control unit 11 takes in the difference count value Ddfc generated by the difference generation unit 53. The controller 11 smoothes the difference count value Ddfc in step S507 in order to exclude a sudden change in the difference value Ddf due to noise or the like. In FIG. 1 and FIG. 6, the difference count value Ddfc is generically named, but here, for convenience, the difference count value Ddfc exceeding the positive threshold is set as the difference count value Ddfc (+) and the difference count exceeding the negative threshold. The value Ddfc is referred to as a difference count value Ddfc (−).

  In step S <b> 508, the control unit 11 determines whether or not the difference count value Ddfc (+) exceeding the positive threshold value is within the prescribed value for evaluation preset in the control unit 11. If it is not determined that the value is within the specified value (NO), that is, if the specified value is exceeded, the control unit 11 sets the conversion coefficient Cob to increase in step S509, and the process is performed in step S504. To migrate. If it is determined that the value is within the specified value (YES), the control unit 11 determines that the difference count value Ddfc (−) exceeding the negative threshold value is within the predetermined value for evaluation set in the control unit 11 in step S510. It is determined whether or not. If it is not determined that the value is within the specified value (NO), that is, if the specified value is exceeded, the control unit 11 sets to reduce the conversion coefficient Cob in step S511, and the process is performed in step S504. To migrate. If it is determined that the value is within the specified value (YES), the control unit 11 maintains the temporarily set conversion coefficient Cob in step S512 and shifts the process to step S504.

  Next, with reference to FIG. 7, the details of the processing for calculating the combination weighting coefficients Wa and Wb in step S7 of FIG. 5 will be described. In FIG. 7, the control unit 11 temporarily sets the upper limit value ThH of the synthesis area described in FIG. 3 in step S <b> 701. In step S702, the control unit 11 temporarily sets the lower limit value ThL of the synthesis area described with reference to FIG. In step S703, the control unit 11 determines whether or not to update the upper limit value ThH and the lower limit value ThL depending on whether or not the difference count value Ddfc described with reference to FIG. If it is determined to be updated (YES), the control unit 11 determines in step S704 whether or not the change of the conversion coefficient Cob is small and stable. If it is not determined to update (NO), the controller 11 shifts the processing to step S707.

  If it is determined in step S704 that the conversion coefficient Cob is stable (YES), the control unit 11 determines the difference count value Ddfc (difference count value Ddfc (+), Ddfc (−)) in step S705. It is determined whether or not the change is small and stable. If it is not determined that the conversion coefficient Cob is stable (NO), the control unit 11 shifts the processing to step S707. If it is determined in step S705 that the difference count value Ddfc is stable (YES), the control unit 11 updates the upper limit value ThH and the lower limit value ThL so as to narrow the synthesis area in step S706. If it is not determined that the difference count value Ddfc is stable (NO), the control unit 11 shifts the processing to step S707.

  Subsequently, in step S707, the control unit 11 determines whether or not the long-time exposure image data Db2 is a reference. If it is determined that the long-exposure image data Db2 is the reference (YES), the control unit 11 generates a composite weighting coefficient Wb for the long-exposure image data Db2 in step S708, and in step S709. Then, a synthesis weighting coefficient Wa for the short-time exposure image data Da1 is generated. If it is not determined that the long-exposure image data Db2 is the reference (NO), the control unit 11 generates a composite weighting coefficient Wa in step S710, and generates a composite weighting coefficient Wb in step S711. To do.

As described above, in the first embodiment, by repeating steps S1 to S7 in FIG. 5, the conversion coefficient Cob and the combined weighting coefficients Wa and Wb converge to a preferable state. As a result, the wide dynamic range image data D _HDR output by the weighting / synthesizing unit 54 in step S8 in FIG. 5 is a visually natural image in which the pixel values at the boundary are continuous. As can be seen from the above description, in the first embodiment, the difference count value Ddfc based on the difference value between the short-time exposure image data Da1 and the long-time exposure image data Db2 is obtained, and the conversion coefficient Cob and the synthesis weighting coefficient are obtained. Since it is sufficient to update until Wa and Wb converge, the circuit scale and the calculation amount may be small. Therefore, there is almost no cost increase, and if any, it is slight.

Second Embodiment
The configuration of the second embodiment is the same as the configuration shown in FIG. The second embodiment is obtained by the step S5 ₂ shown in FIG. 8 step S5 in FIG. 5. In FIG. 8, the same parts as those of FIG. As shown in FIG. 8, step S5 ₂ it is is provided with a step S513 between step S501 and step S502 in step S5 ₁ shown in FIG.

Depending on the exposure amount ratio between the short-time exposure image data Da and the long-time exposure image data Db, the short-time exposure image data Da occupies many areas of the composite image. Then, the area occupied by the long-exposure image data Db with less noise is reduced, and the effect of the long-exposure image data Db is not sufficiently exhibited. Therefore, the control unit 11, in step S5 ₂ according to the second embodiment, at step S513, sets to increase the multiple of the transform coefficient Coa provisionally set to 64 times at step S501. For example, the control unit 11 increases the conversion coefficient Coa so that the short-time exposure image data Da is multiplied by 100 times. At this time, it is preferable to change the knee characteristic of the short-time exposure image data Da in the input signal processing unit 2 so that the high luminance side of the short-time exposure image data Da is not saturated.

  By increasing the conversion coefficient Coa, the amount of change in the pixel value when the short-time exposure image data Da is subjected to gamma processing differs from that when step S514 is not provided. Therefore, the correction value of the conversion coefficient Cob is also set to an appropriate value corresponding to the increase setting of the conversion coefficient Coa.

  According to the second embodiment, the area occupied by the long-time exposure image data Db in the composite image can be increased, and a composite image with less noise can be obtained.

<Third Embodiment>
The configuration and operation of the third embodiment will be described with reference to FIGS. 9 and 10. The third embodiment is obtained by the step S5 ₃ shown in FIG. 10 to step S5 in FIG. 5. 9, the same parts as those in FIG. 1 are denoted by the same reference numerals, and in FIG. 10, the same steps as those in FIG. In FIG. 9, the video camera 103 is provided with a multiple exposure image composition unit 503 in place of the multiple exposure image composition unit 501 in FIG. The multiple exposure image composition unit 503 includes a difference region setting unit 55 that outputs a difference count value Ddfcp in a partial specific region of the difference count value Ddfc in the entire screen (frame) output from the difference generation unit 53. Have. Here again, for convenience, the difference count value Ddfcp exceeding the positive threshold is referred to as the difference count value Ddfcp (+), and the difference count value Ddfcp exceeding the negative threshold is referred to as the difference count value Ddfcp (-).

  The specific area in the screen is, for example, the center of the screen. The user may specify a specific area with the operation switch 15. In addition, a face area detected by a face detection unit (not shown) can be set as a specific area.

  In the third embodiment, as illustrated in FIG. 10, the control unit 11 captures the difference count value Ddfcp from the partial region output from the difference region setting unit 55 in step S5063 after step S505. . In step S5073, the controller 11 smoothes the difference count value Ddfcp. In step S5083, the control unit 11 determines whether or not the positive difference count value Ddfcp (+) is within a specified value preset in the control unit 11. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be increased in step S510. In step S5103, the control unit 11 determines whether or not the negative difference count value Ddfcp (−) is within a specified value preset in the control unit 11. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be decreased in step S511. Since the difference count value Ddfcp from the partial area is captured in step S5063, the specified value used in steps S5083 and S5103 in FIG. 10 must be different from the specified value used in steps S508 and S510 in FIG.

  According to the third embodiment, it is determined based on the difference count value Ddfcp from the partial area whether or not the pixel values of the short-exposure image data Da1 and the long-exposure image data Db2 are close to each other in the synthesis area. Therefore, the amount of data can be reduced, and the time until various parameters for synthesis converge can be shortened. Since the difference count value Ddfcp from a partial area is used instead of the difference count value Ddfc of the entire screen, the accuracy of the continuous pixel value at the boundary is better in the first embodiment. However, the difference in accuracy does not become a problem substantially by setting the central area or the face area that the user pays attention to in the screen as a specific area.

  Also in the third embodiment, step S513 in FIG. 8 of the second embodiment may be provided between step S501 and step S502 in FIG.

<Fourth embodiment>
The configuration and operation of the fourth embodiment will be described with reference to FIGS. 11 and 12. The fourth embodiment is obtained by the step S5 ₄ shown in FIG. 12 to step S5 in FIG. 5. 11, the same parts as those in FIG. 1 are denoted by the same reference numerals, and in FIG. 12, the same steps as those in FIG. In FIG. 11, the video camera 104 includes a multiple exposure image composition unit 504 instead of the multiple exposure image composition unit 501 in FIG. 1. The multiple exposure image composition unit 504 is configured such that the difference generation unit 53 does not output the difference count value Ddfc of the first embodiment but outputs the difference value Ddf itself. In the fourth embodiment, the control unit 11 directly evaluates the representative value based on the difference value Ddf.

  In FIG. 12, the control unit 11 sets a threshold value for a difference average value Ddfavg described later in step S5054. The threshold value set here is to increase or decrease the conversion coefficient Cob when the difference average value Ddfavg is equal to or less than the threshold value so as not to follow the fluctuation of the difference average value Ddfavg. In step S5064, the control unit 11 captures the difference value Ddf. In step S5074, the control unit 11 smoothes and averages the difference value Ddf to generate the difference average value Ddfavg. The control unit 11 sets the difference average value Ddfavg as a representative value for evaluating the difference value Ddf. Instead of averaging the difference value Ddf, a difference addition value obtained by adding the difference value Ddf may be used as the representative value.

  In FIG. 12, the control unit 11 determines whether or not the difference average value Ddfavg is within a positive specified value preset in the control unit 11 in step S5084. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be increased in step S509. In step S5104, the control unit 11 determines whether or not the difference average value Ddfavg is within a specified value preset in the control unit 11. If it is not determined to be within the negative specified value (NO), the control unit 11 sets the conversion coefficient Cob to be decreased in step S511.

  As described above, even if the difference average value Ddfavg obtained from the difference value Ddf itself is evaluated instead of the difference count value Ddfc of the difference value exceeding the threshold value, the same effect as that of the first embodiment can be obtained. . In the fourth embodiment, since the difference count value Ddfc is not used, step S705 in FIG. 7 may be a step for determining whether or not the difference average value Ddfavg is stable.

  Also in the fourth embodiment, step S513 in FIG. 8 of the second embodiment may be provided between step S501 and step S502 in FIG.

<Fifth Embodiment>
The configuration and operation of the fifth embodiment will be described with reference to FIGS. 13 and 14. The fifth embodiment is obtained by the step S5 ₅ shown in FIG. 14 step S5 in FIG. 5. In FIG. 13, the same parts as those in FIG. 1 are denoted by the same reference numerals, and in FIG. 14, the same steps as those in FIG. In FIG. 13, the video camera 105 includes a multiple exposure image composition unit 505 instead of the multiple exposure image composition unit 501 in FIG. 1. In the multiple exposure image composition unit 505, the difference generation unit 53 outputs the difference value Ddf itself, and the difference region setting unit 55 includes a difference value in a partial specific region among the difference values Ddf in the entire screen (frame). It is configured to output Ddfp. The specific area in the screen is the same as in the third embodiment.

  In FIG. 14, the control unit 11 sets a threshold value for a difference average value Ddfpavg, which will be described later, in step S5055. The threshold for the difference average value Ddfpavg is the same as the threshold for the difference average value Ddfavg in the fourth embodiment. In step S5065, the control unit 11 captures a partial difference value Ddfp. In step S5075, the control unit 11 smoothes the partial difference value Ddfp and averages it to generate a difference average value Ddfpavg. In step S5085, the control unit 11 determines whether or not the difference average value Ddfpavg is within a positive specified value preset in the control unit 11. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be increased in step S509. In step S5105, the control unit 11 determines whether or not the difference average value Ddfpavg is within a negative specified value preset in the control unit 11. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be decreased in step S511.

  When evaluating the difference value Ddf itself instead of evaluating the difference count value Ddfc exceeding the threshold value, it is preferable to use the difference average value Ddfpavg from a partial area because the data amount can be reduced.

  Also in the fifth embodiment, step S513 in FIG. 8 of the second embodiment may be provided between step S501 and step S502 in FIG.

<Sixth Embodiment>
The configuration and operation of the sixth embodiment will be described with reference to FIGS. 15 and 16. Sixth embodiment is obtained by the step S5 ₆ shown in FIG. 16 step S5 in FIG. 5. 15, the same parts as those in FIG. 1 are denoted by the same reference numerals, and in FIG. 16, the same steps as those in FIG. In FIG. 16, the video camera 106 is provided with a multiple exposure image composition unit 506 instead of the multiple exposure image composition unit 501 in FIG.

  The multiple exposure image composition unit 506 includes a smoothing processing unit 56. The smoothing processing unit 56 smoothes the short-time exposure image data Db2 output from the smoothing processing unit 56a for smoothing the short-time exposure image data Da1 output from the pixel value conversion unit 51a and the pixel-value conversion unit 51b. And a smoothing processing unit 56b.

  The smoothing processing units 56a and 56b can be configured by a low-pass filter that smoothes the pixel values in the screen. By smoothing the short-exposure image data Da1 and the long-exposure image data Db2 by the smoothing processing units 56a and 56b, the movement and noise of the subject other than the screen blur removed by the multiple-exposure image output unit 3 are exposed for a short time. The influence on the difference value between the image data Da1 and the long-time exposure image data Db2 can be reduced.

  The short-exposure image data Da1 and the long-exposure image data Db2 that have been subjected to the smoothing process and output from the smoothing processing units 56a and 56b are input to the difference generation unit 53. The difference generation unit 53 generates a difference count value Ddfc based on the short-time exposure image data Da1 and the long-time exposure image data Db2 that have been subjected to the smoothing process. Since the low luminance portion of the short exposure image data Da1 has a characteristic that the S / N is not good and the gradation is low, the control unit 11 smoothes the filter for the short exposure image data Da1 in the smoothing processing unit 56a. The smoothing coefficient is preferably larger than the smoothing coefficient of the filter for the long-time exposure image data Db2 in the smoothing processing unit 56b. That is, it is preferable that the degree of smoothing for the short-time exposure image data Da1 is larger than the degree of smoothing for the long-time exposure image data Db2.

  In FIG. 16, the control unit 11 smoothes the short-time exposure image data Da1 and the long-time exposure image data Db2 by the smoothing processing units 56a and 56b in step S514. The difference count value Ddfc captured in step S506 is generated based on the difference value in which the influence of subject movement and noise is reduced by the smoothing by the smoothing processing units 56a and 56b. Steps S505 and subsequent steps following step S514 are the same as in the first embodiment.

  Also in the sixth embodiment, step S513 in FIG. 8 of the second embodiment may be provided between step S501 and step S502 in FIG. Also in the sixth embodiment, similarly to the fourth embodiment, the difference average value Ddfavg generated based on the difference value Ddf instead of the difference count value Ddfc of the difference value exceeding the threshold value may be evaluated. Good.

<Seventh embodiment>
The configuration and operation of the seventh embodiment will be described with reference to FIGS. 17 and 18. The seventh embodiment is obtained by the step S5 ₇ shown in FIG. 18 step S5 in FIG. 5. 17, the same parts as those in FIGS. 9 and 15 are denoted by the same reference numerals, and in FIG. 18, the same steps as those in FIG. In FIG. 17, the video camera 107 is provided with a multiple exposure image composition unit 507 in place of the multiple exposure image composition unit 503 in FIG. The multiple exposure image composition unit 507 includes a smoothing processing unit 56 and a difference area setting unit 55.

  In the seventh embodiment, the difference generation unit 53 generates a difference count value Ddfc based on the short-time exposure image data Da1 and the long-time exposure image data Db2 that have been smoothed by the smoothing processing units 56a and 56b. The difference area setting unit 55 generates the difference count value Ddfcp from the partial area based on the difference count value Ddfc.

  In FIG. 18, the control unit 11 captures the difference count value Ddfcp from the partial area output from the difference area setting unit 55 in step S5067. The controller 11 smoothes the difference count value Ddfcp in step S5077. In step S5087, the control unit 11 determines whether or not the positive difference count value Ddfcp (+) is within a specified value preset in the control unit 11. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be increased in step S510. In step S5107, the control unit 11 determines whether or not the negative difference count value Ddfcp (−) is within a specified value preset in the control unit 11. If it is not determined that the value is within the specified value (NO), the control unit 11 sets the conversion coefficient Cob to be decreased in step S511.

  Also in the seventh embodiment, step S513 in FIG. 8 of the second embodiment may be provided between step S501 and step S502 in FIG. Also in the seventh embodiment, instead of evaluating the difference count value Ddfcp from the partial area, the difference average value Ddfpavg from the partial area may be evaluated as in the fifth embodiment.

  The present invention is not limited to the embodiments described above, and various modifications can be made without departing from the scope of the present invention. Although an example in which an image processing device is mounted on a video camera has been shown as each embodiment, the present invention can be applied to an information processing device such as an image display device or a computer. In each embodiment, the captured images are immediately combined to generate a composite image. However, the images captured with different exposure amounts are stored and stored in a storage medium and stored in the storage medium. The image processing apparatus may synthesize the processed images. In each embodiment, the image is synthesized in the form of the luminance signal Y and the color difference signals Cb and Cr, but it is also possible to synthesize with the RGB signal.

In each embodiment, the synthesis of moving images has been described, but still image synthesis may be used. When two still images taken with different exposure amounts are combined, various parameters are repeatedly updated for the same image, and the parameter update process is repeated until an optimum combined image is obtained. In each embodiment, the compressed wide dynamic range image data D _CHDR is displayed on the display unit 10, or the encoded data based on the compressed wide dynamic range image data D _CHDR is recorded on the recording / reproducing unit 14. Wide dynamic range image data D _HDR may be recorded and displayed as it is. As described above, three or more images having different exposure amounts may be combined.

  Furthermore, in each embodiment, although the control part 11 was operated as a parameter update part, the control part which sets or updates the conversion coefficients Coa and Cob, the control part which sets or updates the upper limit value ThH and the lower limit value ThL, May be provided separately, and a specific configuration method is arbitrary.

  By the way, in each embodiment, the digital signal after the multiple exposure image output unit 3 is referred to as data for the sake of convenience. However, the signal includes both analog signals and digital signals without distinguishing them. In each embodiment, the multiple exposure image output unit 3 and later are digital signal processing, but the use of digital signal processing and analog signal processing is also arbitrary.

DESCRIPTION OF SYMBOLS 1 Imaging part 2 Input signal processing part 3 Multiple exposure image output part 6 Dynamic range compression part 11 Control part (parameter update part)
51, 51a, 51b Pixel value conversion unit 52 Synthesis weighting coefficient generation unit 53 Difference generation unit 54 Weighting synthesis unit 55 Difference region setting unit 56, 56a, 56b Smoothing processing unit 101, 103, 104, 105, 106, 107 Video camera 501 503 504 505 506 507 Multiple exposure image composition unit

Claims (7)

The first plurality of image signals obtained by photographing the same subject a plurality of times with different exposure amounts are input, and the pixel values in the first plurality of image signals due to the different exposure amounts are matched. A pixel value conversion unit that converts a pixel value of at least one of the first plurality of image signals based on a pixel value conversion coefficient and outputs the converted pixel value as a second plurality of image signals;
A synthesis weighting coefficient generation unit that generates a synthesis weighting coefficient used when combining the second plurality of image signals with a region including a pixel having a predetermined pixel value as a boundary part;
A smoothing processing unit for smoothing each of the second plurality of image signals in a screen and outputting the plurality of image signals as a third plurality of image signals;
A difference generator for generating a difference between the third plurality of image signals;
A weighting and synthesizing unit that generates a composite image signal by weighting and combining the second plurality of image signals using the composite weighting coefficient;
A parameter update unit that evaluates the difference generated by the difference generation unit and updates the pixel value conversion coefficient so that the difference is reduced;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the parameter updating unit updates the synthesis weighting coefficient in conjunction with updating the pixel value conversion coefficient. The boundary portion is a region including pixels having a plurality of pixel values, and when the largest pixel value of the boundary portion is an upper limit value and the smallest pixel value of the boundary portion is a lower limit value,
The synthesis weighting coefficient generator generates a synthesis weighting coefficient for synthesizing the first and second image signals in the second plurality of image signals, and a portion of the first and second image signals is larger than the upper limit value. Of the first and second image signals in a portion smaller than the lower limit value, and the first and second image signals between the upper limit value and the lower limit value. 3. The image processing apparatus according to claim 1, wherein a composite weighting coefficient is generated that cross-fades and mixes the two image signals. In addition to updating the pixel value conversion coefficient, the parameter updating unit updates the upper limit value and the lower limit value so that the interval between them is narrowed,
The image processing apparatus according to claim 2, wherein the synthesis weighting coefficient generation unit generates a synthesis weighting coefficient based on the updated upper limit value and the lower limit value. The smoothing processing unit makes the degree of smoothing for an image signal with a small amount of exposure out of the second plurality of image signals larger than the degree of smoothing for an image signal with a large amount of exposure. The image processing apparatus according to claim 3. The image processing apparatus according to claim 1, wherein the parameter update unit evaluates a difference between the third plurality of image signals in a partial region in the screen. Shooting the same subject multiple times with different exposure amounts to generate a first plurality of image signals,
Based on a pixel value conversion coefficient, pixels of at least one image signal among the first plurality of image signals so as to match pixel value differences in the first plurality of image signals due to different exposure amounts. Converting the value to generate a second plurality of image signals;
Generating a combination weighting coefficient used when combining the second plurality of image signals with a region including a pixel having a predetermined pixel value as a boundary portion;
Smoothing each of the second plurality of image signals within a screen to generate a third plurality of image signals;
Generating a difference between the third plurality of image signals;
Generating a composite image signal by weighting and combining the second plurality of image signals using the composite weighting coefficient;
The image processing method characterized by evaluating the difference and updating the pixel value conversion coefficient so that the difference becomes small.

Images