JP3740394B2 - High dynamic range video generation method and apparatus, execution program for the method, and recording medium for the execution program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  TECHNICAL FIELD The present invention relates to a high dynamic range video generation method and a generation apparatus that make a subject with a high dynamic range a video without white spots or blackouts.Application fieldsAnd video industry and measurement industry.
[0002]
[Prior art]
There are various methods for recording an image, but today, an imaging device using a CCD is widely used. However, the CCD has a drawback that the dynamic range of light and dark is smaller than that of a silver salt film or the like. On the other hand, a method and apparatus for generating a high dynamic range video from two adjacent images by alternately repeating short-time imaging and long-time imaging using a CCD with an electronic shutter capable of variably controlling the imaging time has been proposed. Yes. For example, techniques disclosed in Japanese Patent Laid-Open Nos. 63-3067779 and 6-141229 correspond to this.
[0003]
[Problems to be solved by the invention]
However, as indicated by 11 in FIG. 1 (a), both methods have the disadvantage that the frame rate is halved in order to synthesize one frame image from two frames of original images, and a moving image with smooth motion cannot be generated. was there. In Japanese Patent Laid-Open No. 6-141229, a dedicated CCD that operates at a substantially double speed is developed to solve this problem. However, this method has a problem in versatility and cost because a general-purpose CCD cannot be used. In addition, since the double speed operation itself becomes technically difficult when the number of pixels of the CCD increases, there is a problem that it cannot be applied to high resolution images such as HDTV and UDTV.
[0004]
In addition, when shooting moving images rather than still images, the position and shape of the moving subject's high-luminance or low-luminance regions vary with time, but the conventional method combines images captured at different times. Therefore, there is a problem that image quality deterioration occurs due to generation of a false signal in the outline region of the synthesis unit. In Japanese Patent Laid-Open No. 6-141229, the two images are gradually mixed according to the luminance value so that the image quality deterioration is not visually noticeable. However, this is not a fundamental solution.
[0005]
The present invention has been made to solve the above-described problems of the prior art, and an object thereof is to provide a method and apparatus capable of generating a high dynamic range video at the same frame rate as the original image. Furthermore, it is an object of the present invention to provide a method and apparatus capable of generating a high-quality, high dynamic range video even when video switching points and image contents change discontinuously between frames. To do.
[0007]
[Means for Solving the Problems]
  As means for solving the above-mentioned problems, in claim 1 of the present invention,In a method for generating a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device capable of variably controlling the imaging time, longer than two images of long-time imaging and short-time imaging immediately after or immediately before Two different synthesis processes, a process A for synthesizing a high dynamic range image at a time imaging time, and a process B for synthesizing a high dynamic range image at a short imaging time than two images of a short imaging and an immediately following or immediately preceding long imaging. The process A is executed for the image at the long imaging time, the process B is executed for the image at the short imaging time,
  Process AInA process C for detecting an overflow area X in a long-time image, a process D for detecting an area Y that is predicted to overflow if a long-time image is captured in a short-time image, and a contour of two areas Line pattern matching, and in step C, a long-time captured image is converted into a region X and a region X other than that.^-The region Y is extracted from the short-time captured image immediately after or immediately before in the process D, and the luminance value of the pixel is adjusted after the position and shape of the area Y are matched with the position and shape of the area X by the process of pattern matching. After magnification correction, region X^-And combining the image at the long-time imaging time from the two images of the long-time imaging and the short-time imaging immediately after or immediately before.HighA dynamic range video generation method is used.
[0008]
  Further, in claim 2 of the present invention,In a method for generating a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device capable of variably controlling the imaging time, longer than two images of long-time imaging and short-time imaging immediately after or immediately before Two different synthesis processes, a process A for synthesizing a high dynamic range image at a time imaging time, and a process B for synthesizing a high dynamic range image at a short imaging time than two images of a short imaging and an immediately following or immediately preceding long imaging. The process A is executed for the image at the long imaging time, the process B is executed for the image at the short imaging time,
  In process B,A process C for detecting an overflow region X in a long-time image, a process D for detecting a region Y that is predicted to overflow if a long-time image is captured in a short-time image, and an outline of two regions The pattern matching of the region Y, and in the step D, the short-time captured image is converted into the region Y and the other region Y.^-In Step C, a long-time image immediately after or immediately before is divided into a region X and another region X.^-Region X in the process of pattern matching^-The position and shape of the region Y^-Are combined with the region Y after the magnification correction of the luminance value of the pixel after being matched with the position and shape of the image, so that the short-time imaging and the long-time imaging immediately after or immediately before the short-time imaging can be performed at the short-time imaging time. Characterized by compositing imagesHighA dynamic range video generation method is used.
[0009]
  In addition, the present inventionClaim 3In a method for generating a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device capable of variably controlling the imaging time, long-time imaging, immediately preceding short-time imaging, and immediately following short-time imaging A process E for synthesizing a high dynamic range image at a long-time imaging time from three captured images, and a high dynamic range image at a short-time imaging time from three images of a short-time imaging, a long-time imaging immediately before and a long-time imaging immediately after A high dynamic range characterized in that the process E is executed for an image at a long imaging time, and the process F is executed for an image at a short imaging time. Use video generation methods.
[0010]
  In addition, the present inventionClaim 4Then, in the process E, a process C for detecting an overflow area X in a long-time image, and a process D for detecting an area Y that is predicted to overflow if the long-time image is taken for a long time. A process of performing pattern matching of the images of the two areas, and an image captured for a long time by the process C is divided into the area X and the other areas X^-In the process D, the area Y1 is cut out from the immediately preceding short-time image, and the area Y2 is cut out from the short-time image immediately after the process D to perform pattern matching between the two areas. An intermediate image Y3 at a long imaging time is generated from a certain area Y1 and area Y2, a region that overlaps with the area X is cut out from the intermediate image Y3, and after correcting the magnification of the luminance value of the pixel, the area X^-And combining the images at the long-time imaging time from the three images of the long-time imaging, the immediately preceding short-time image, and the immediately following short-time imaging.Claim 3The high dynamic range video generation method described in 1) is used.
[0011]
  In addition, the present inventionClaim 5Then, in the process F, a process C for detecting an overflow area X in a long-time image, and a process D for detecting an area Y that is predicted to overflow if the long-time image is captured for a long time. And pattern matching of images in two areas, and in step D, an image captured for a short time is converted into an area Y and another area Y.^-In Step C, the image captured for the long time immediately before is divided into the region X1 and the other region X1.^-The image of the long-time image immediately after the process C is divided into the region X2 and the other region X2.^-Region X1 which is two regions by the process of performing pattern matching of the images of the two regions.^-And region X2^-Intermediate image X3 at a short imaging time from^-To generate an intermediate image X3^-To region Y^-3 regions, which are short-time imaging, immediately preceding long-time imaging, and immediately following short-time imaging, are combined with region Y after pixel luminance value magnification correction is performed. It is characterized by synthesizing images at a short imaging time fromClaim 3The high dynamic range video generation method described in 1) is used.
[0014]
  In addition, the present inventionClaim 6ThenIn an apparatus that generates a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging apparatus capable of variably controlling the imaging time, it is longer than two images of long-time imaging and short-time imaging immediately after or immediately before Two different types of synthesizer A that synthesizes a high dynamic range image at a time imaging time and a synthesizer B that synthesizes a high dynamic range image at a short imaging time than two images of a short time imaging and a long time imaging immediately after or immediately before. A synthesizer, using a synthesizer A for an image at a long imaging time, and using a synthesizer B for an image at a short imaging time,
  The synthesizer A includes a detector C that detects an overflow area X in a long-time image, and a detector D that detects an area Y that is predicted to overflow if the long-time image is captured for a long time. And a pattern matching unit that performs pattern matching of the contour lines of the two regions, and using the detector C, a long-time captured image is converted into the region X and the other region X.^-After extracting the region Y from the short-time captured image immediately after or immediately before using the detector D, and adjusting the position and shape of the region Y to the position and shape of the region X using the pattern matching device, the pixel After correcting the magnification of the luminance value of the region X,^-And the image at the long-time imaging time is synthesized from the two images of the long-time imaging and the short-time imaging immediately after or immediately before.HighA dynamic range video generation device is used.
[0015]
  In addition, the present inventionClaim 7ThenIn an apparatus that generates a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging apparatus capable of variably controlling the imaging time, it is longer than two images of long-time imaging and short-time imaging immediately after or immediately before Two different types of synthesizer A that synthesizes a high dynamic range image at a time imaging time and a synthesizer B that synthesizes a high dynamic range image at a short imaging time than two images of a short time imaging and a long time imaging immediately after or immediately before. A synthesizer, using a synthesizer A for an image at a long imaging time, and using a synthesizer B for an image at a short imaging time,
  The synthesizer B includes a detector C that detects an overflow area X in a long-time image, and a detector D that detects an area Y that is predicted to overflow if the long-time image is captured for a long time. And a pattern matching unit that performs pattern matching of the contour lines of the two regions, and using the detector D, the short-time captured image is converted into the region Y and the other regions Y.^-And using the detector C, a long-time image immediately after or immediately before the region X and the other regions X^-Region X using the pattern matcher^-The position and shape of the region Y^-Are combined with the region Y after the magnification correction of the luminance value of the pixel after being matched with the position and shape of the image, so that the short-time imaging and the long-time imaging immediately after or immediately before the short-time imaging can be performed at the short-time imaging time. Characterized by combining imagesHighA dynamic range video generation device is used.
[0016]
  In addition, the present inventionClaim 8In a device that generates a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device that can variably control the imaging time, long-time imaging, immediately preceding short-time imaging, and immediately following short-time imaging A synthesizer E that synthesizes a high dynamic range image at a long imaging time from the three captured images, and a high dynamic range image at a short imaging time than the three images of the short imaging, the immediately preceding long imaging, and the immediately following long imaging. The synthesizer F has two different synthesizers. The synthesizer E is used for an image at a long imaging time, and the synthesizer F is used for an image at a short imaging time. A dynamic range video generation device is used.
[0017]
  In addition, the present inventionClaim 9Then, the synthesizer E detects the overflow region X in the long-time image, and detects the region Y that is predicted to overflow if the long-time image is captured for a long time. And a pattern matching unit that performs pattern matching of the images of the two regions. The detector C is used to capture an image captured for a long time using the region X and the other regions X.^-The region Y1 is cut out from the immediately preceding short-time image using the detector D, the region Y2 is cut out from the immediately short-time image using the detector D, and the pattern is extracted from the region Y1 and the region Y2. An intermediate image Y3 at a long imaging time is generated using a matching device, a region that overlaps with the region X is cut out from the intermediate image Y3, and after correcting the magnification of the pixel luminance value, the region X^-And the image at the long-time imaging time from the three images of the long-time imaging, the immediately preceding short-time image, and the immediately following short-time imaging.Claim 8The high dynamic range video generation apparatus described in 1) is used.
[0018]
  In addition, the present inventionClaim 10Then, the synthesizer F detects the overflow area X in the long-time image, and detects the area Y that is predicted to overflow if the long-time image is captured for a long time. And a pattern matching unit that performs pattern matching of the images of the two regions, and using the detector D, an image captured for a short period of time is displayed in the region Y and the other regions Y.^-And using the detector C, the previous long-time image is taken as the region X1 and the other region X1.^-The image of the long-time image immediately after using the detector C is divided into the region X2 and the other region X2.^-Into region X1^-And region X2^-To the intermediate image X3 at a short imaging time using the pattern matching device^-To generate an intermediate image X3^-To region Y^-3 regions, which are short-time imaging, immediately preceding long-time imaging, and immediately following short-time imaging, are combined with region Y after pixel luminance value magnification correction is performed. It is characterized by combining images at a short imaging time fromClaim 8The high dynamic range video generation apparatus described in 1) is used.
[0020]
  In addition, the present inventionClaim 11Then, from claim 1Claim 5An execution program for a high dynamic range video generation method is used, wherein the process in the high dynamic range video generation method according to any one of the above is a program for causing a computer to execute the process.
[0021]
  In addition, the present inventionClaim 12Then, from claim 1Claim 5A high dynamic range comprising: a program for causing a computer to execute the process in the high dynamic range video generation method according to any one of the above, and the program recorded on a recording medium readable by the computer. A recording medium on which an execution program of a video generation method is recorded is used.
[0022]
The method and apparatus of the present invention can effectively generate an original image of one frame by alternately using two different synthesis algorithms as shown in 12 of FIG. 1 (b) and 13 of FIG. 1 (c). A high dynamic range image is provided at the same frame rate by generating a one frame high dynamic range image.
[0023]
  Claims 1, 2Provides a method of synthesizing a high dynamic range image from two consecutive frames of image as shown at 12 in FIG.Claim 3Provides a method for synthesizing a high dynamic range image from three consecutive frames as shown at 13 in FIG.
[0024]
  Claim 3This method is a method of synthesizing a high dynamic range image by predicting and generating an image with higher accuracy.Claims 1, 2Although it is possible to synthesize images with higher quality than the above method, it is necessary to have a continuous three-frame image, so it cannot be applied when the video switching point or the image content changes discontinuously between frames, or There is also a problem that even if applied forcibly, only a low-quality composite image can be obtained.
[0025]
  ThereforeThe aboveDetect discontinuities between frames by examining the similarity of two images,Claim 3And the methodClaims 1, 2A method for performing processing by adaptively switching the methods is provided.
[0026]
  Claims 6, 7Provides a device for synthesizing a high dynamic range image from two consecutive frames as shown at 12 in FIG.Claim 8Provides a device for synthesizing a high dynamic range image from three consecutive frames as shown at 13 in FIG.
[0027]
  Claim 8Is an apparatus that synthesizes a high dynamic range image by predicting and generating an image with higher accuracy.Claims 6, 7Although it is possible to synthesize images with higher quality than the above-mentioned apparatus, since it is necessary to have three consecutive frames of images, it cannot be applied when video switching points or image contents change discontinuously between frames, or There is also a problem that even if applied forcibly, only a low-quality composite image can be obtained.
[0028]
  ThereforeThe aboveDetect discontinuities between frames by examining the similarity of two images,Claim 8And the methodClaims 6, 7An apparatus for performing processing by adaptively switching the method is provided.
[0029]
As described above, the method and apparatus according to the present invention predicts and generates an image that should be obtained at the imaging time by performing moving image compensation and contour fitting using means for performing pattern matching, and then uses the image. Synthesize high dynamic range images. In this way, high-quality, high dynamic range images are also provided for video shooting.
[0030]
  Using FIG.Claims 1, 2The operation of this method will be described. Reference numerals 21 to 23 in FIG. 2 denote a part of video data picked up by alternately changing the imaging time. Reference numerals 21 and 23 denote long-time images, and 22 denotes a short-time image. 211 and 231 are areas where overflow has occurred due to overexposure of the subject, and 212 and 232 are other areas. Reference numeral 221 denotes an area that is predicted to overflow if imaging is performed for a long time, and 222 is an area other than that.
[0031]
  Claims 1, 2According to the above method, in step A, 221 is adjusted to the position and shape of 211 in step A to become 223, 223 is subjected to magnification correction of the luminance value of the pixel in step 26, and then combined with 212 and 24 is combined. Is combined with the position and shape of 222 to become 233, and combined with 221 after being subjected to magnification correction of the luminance value of the pixel at 27, 25 is synthesized.
[0032]
Here, the operation of combining the image at a certain time with the image immediately after that is described. For example, when 25 is combined, 212 is adjusted to the position and shape 222 instead of 232. Thus, it is needless to say that an image at a certain time and an image at the previous time can be synthesized by simply replacing the portion that uses the immediately following image with the previous image.
[0033]
  Claim 1According to the above method, the process A performs the pattern matching of the contour lines of 221 and 211, calculates the position and shape shift amount 28 of 221 with respect to 211, and generates 223 by reversing 28 with respect to 221. Then, after correcting the magnification of the luminance value of the pixel at 26, an operation of combining 24 with 212 is performed.
[0034]
  Claim 2According to the above method, the process B performs the pattern matching of the contour lines of 232 and 222, calculates the position and shape shift amount 29 of 232 with respect to 222, and generates 233 by acting 29 on 232 in reverse. Then, the operation of combining 25 is performed in combination with 221 after correcting the magnification of the luminance value in 27.
[0035]
  Next, using FIG.Claim 3The operation of this method will be described. Reference numerals 31 to 34 in FIG. 3 are a part of video data picked up by alternately changing the imaging time. 31 and 33 are short-time captured images, and 32 and 34 are long-time captured images. Reference numerals 321 and 341 denote areas where overflow has occurred due to overexposure of the subject, and reference numerals 322 and 342 denote other areas. Reference numerals 311 and 331 are areas that are predicted to overflow if imaging is performed for a long time, and 312 and 332 are other areas.
[0036]
  Claim 3According to the above method, the process E generates 323 corresponding to the position and shape of 311 and 331 to 321, and after correcting the magnification of the luminance value of the pixel in 37, 35 is synthesized in combination with 322, and the process F 322 and 342 to 332 corresponding to the position and shape are generated from 322 and 342, and 36 is synthesized by combining with 331 after correcting the magnification of the luminance value in 38.
[0037]
  Next, using FIG.Claim 4The operation of this method will be described. In the figure, the same parts as those in FIG. First, pattern matching between the images 311 and 331 is performed, and the amount of change 41 in the position and shape of the image with respect to 311 331 is calculated. The intermediate images 42 and 43 at the imaging time of 32 are predicted and generated with respect to 311 and 331 from the shift amount, and the intermediate image 44 is generated by combining both images. Next, 323 is generated by cutting out an area located within 44 to 321. 323 performs the function of combining 35 by combining the 322 after correcting the magnification of the luminance value in 37.
[0038]
  Next, using FIG.Claim 5The operation of this method will be described. In the figure, the same parts as those in FIG. First, pattern matching between the images 322 and 342 is performed, and the image position and shape change amount 51 with respect to 322 342 is calculated. Intermediate images 52 and 53 at the imaging time of 33 based on the change amount for 322 and 342 are obtained. Is predicted, and the intermediate image 54 is generated by combining the two images. Next, 333 is generated by cutting out an area located within 54 to 332. An operation of combining 36 is performed by combining 331 after correcting the magnification of the luminance value at 333 and 38.
[0039]
  In the method of claim 4 or claim 5,When image pattern matching is performed, the similarity between the two patterns is also calculated. If the similarity is small or there is no area for pattern matching,Claim 4In this method, the similarity between 311 and 321 and the similarity between 321 and 331 are compared by pattern matching of contour lines, and when the similarity between 321 and 331 is larger,Claim 3Instead of this method, the image immediately after using 32 and 33 is used.Claims 1, 2If the similarity between 311 and 321 is greater,Claim 3Instead of this method, the previous image using 31 and 32 is used.Claims 1, 2The effect of switching to the method is performed.Claim 5In the method of the above, the similarity between 322 and 332 and the similarity between 332 and 342 are compared by contour pattern matching, and when the similarity between 332 and 342 is larger,Claim 3Instead of this method, the image immediately after using 33 and 34 is used.Claims 1, 2If the similarity between 322 and 332 is greater,Claim 3Use the previous image using 32 and 33 instead of the methodClaims 1, 2The effect of switching to the method is performed.
[0040]
  Using FIG.Claims 6, 7The operation of the apparatus will be described. Reference numerals 21 to 23 in FIG. 2 denote a part of video data picked up by alternately changing the imaging time. Reference numerals 21 and 23 denote long-time images, and 22 denotes a short-time image. 211 and 231 are areas where overflow has occurred due to overexposure of the subject, and 212 and 232 are other areas. Reference numeral 221 denotes an area that is predicted to overflow if imaging is performed for a long time, and 222 is an area other than that.
[0041]
  Claims 6, 7, 221 is combined with the position and shape of 211 by the combiner A to become 223, 223 is subjected to magnification correction of the luminance value of the pixel at 26, and then combined with 212 to combine 24 and combiner B Thus, 232 is combined with the position and shape of 222 to become 233, and 25 is combined with 221 after being subjected to magnification correction of the luminance value of the pixel at 27.
[0042]
Here, the operation of combining the image at a certain time with the image immediately after that is described. For example, when 25 is combined, 212 is adjusted to the position and shape 222 instead of 232. In this way, it goes without saying that the image at the time can be synthesized from the image at the time just by replacing the portion using the immediately following image with the immediately preceding image.
[0043]
  Claim 6Synthesizer A performs pattern matching of the contour lines of 221 and 211, calculates the position and shape shift amount 28 of 221 with respect to 211, and operates 223 by reversing 28 with respect to 221. After generating and correcting the magnification of the luminance value of the pixel at 26, an operation of combining 24 with 212 is performed.
[0044]
  Claim 7Synthesizer B performs pattern matching of the contour lines of 232 and 222, calculates a position and shape shift amount 29 of 232 with respect to 222, and causes 233 to act on 232 in reverse. It is generated and combined with 221 after correcting the magnification of the luminance value at 27, the effect of combining 25 is performed.
[0045]
  Next, using FIG.Claim 8The operation of the apparatus will be described. Reference numerals 31 to 34 in FIG. 3 are a part of video data picked up by alternately changing the imaging time. 31 and 33 are short-time captured images, and 32 and 34 are long-time captured images. Reference numerals 321 and 341 denote areas where overflow has occurred due to overexposure of the subject, and reference numerals 322 and 342 denote other areas. Reference numerals 311 and 331 are areas that are predicted to overflow if imaging is performed for a long time, and 312 and 332 are other areas.
[0046]
  Claim 8According to the apparatus, the synthesizer E generates 323 matching the position and shape from 311 and 331 to 321. After correcting the magnification of the luminance value of the pixel at 37, it is combined with 322 and 35 is synthesized. The unit F generates 333 that matches the positions and shapes of 322 and 342 to 332, and synthesizes 36 by combining with 331 after correcting the magnification of the luminance value in 38.
[0047]
  Next, using FIG.Claim 9The operation of the apparatus will be described. In the figure, the same parts as those in FIG. First, pattern matching between the images 311 and 331 is performed by the pattern matching unit, and the image position and shape change amount 41 with respect to 311 331 is calculated. The intermediate images 42 and 43 at the imaging time of 32 are predicted and generated with respect to 311 and 331 from the shift amount, and the intermediate image 44 is generated by combining both images. Next, 323 is generated by cutting out an area located within 44 to 321. 323 performs the function of combining 35 by combining the 322 after correcting the magnification of the luminance value in 37.
[0048]
  Next, using FIG.Claim 10The operation of the apparatus will be described. In the figure, the same parts as those in FIG. First, pattern matching between the images 322 and 342 is performed by a pattern matching unit, and the amount of change 51 in the position and shape of the image with respect to 342 of 322 is calculated. The intermediate images 52 and 53 at the imaging time of 33 are predicted and generated from the amount of shift 322 and 342, and the intermediate image 54 is generated by synthesizing both images. Next, 333 is generated by cutting out an area located within 54 to 332. An operation of combining 36 is performed by combining 331 after correcting the magnification of the luminance value at 333 and 38.
[0049]
  In the device of claim 9 or claim 10,When image pattern matching is performed by a pattern matching unit, the similarity between two patterns is also calculated, and the similarity is evaluated by a similarity evaluator. If the similarity is small or there is no area for pattern matching,Claim 9In the apparatus of, the similarity between 311 and 321 and the similarity between 321 and 331 are compared by contour pattern matching, and if the similarity between 321 and 331 is greater,Claim 8Use the image immediately after using 32 and 33 instead of the deviceClaims 6, 7If the similarity between 311 and 321 is greater,Claim 8Use the previous image using 31 and 32 instead of the deviceClaims 6, 7The operation is switched to the processing operation of the apparatus.Claim 10In the apparatus of, the similarity between 322 and 332 and the similarity between 332 and 342 are compared by contour pattern matching, and when the similarity between 332 and 342 is larger,Claim 8Use the image immediately after using 33 and 34 instead of the deviceClaims 6, 7If the similarity between 322 and 332 is greater,Claim 8Use the previous image using 32 and 33 instead of the deviceClaims 6, 7The operation is switched to the processing operation of the apparatus.
[0050]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0051]
  Of the present inventionClaims 1, 2An embodiment of this method is shown in FIG. In FIG. 6, 61 is an original video data storing process, 62 is a data output switching process, and 63 is a data output switching process.Claim 1An embodiment for realizing step A of the method of FIG.Claim 2An embodiment for realizing the process B of the method No. 65 is an output data storing process.
[0052]
  631 and 632 are region separation processes,Claim 13 is an embodiment of the process C and the process D in the method. 633 and 634 are contour shape extraction processes, 635 is a contour shape pattern matching process, 636 is a region movement / deformation process, 637 is a pixel luminance value magnification correction process, and 638 is an image composition process.
[0053]
  641 and 642 are region separation processes,Claim 23 is an embodiment of the process C and the process D in the method. 643 and 634 are contour line extraction processes, 645 is a contour shape pattern matching process, 646 is a region movement and deformation process, 647 is a pixel luminance value magnification correction process, and 648 is an image composition process.
[0054]
The flow of processing according to this embodiment will be described. In this embodiment, the original image data has a dynamic range of 8 bits (possible pixel brightness values are 0 to 255) and has been subjected to inverse gamma correction. Also, a process for synthesizing the image data at the time from the image at a certain time and the image data immediately after the image will be described, where k is the imaging time ratio between the long imaging time and the short imaging time.
[0055]
In 61, video data obtained by alternately changing the imaging time is stored. If this method can be performed by real-time processing, an output (gamma corrected) process of the imaging time variable camera may be directly performed instead of 61.
[0056]
62 sequentially extracts an image and an image of the immediately following frame from 61, and outputs the long-time captured image to 631 and the short-time captured image to 632 if the two images are in the order of the long-time captured image and the short-time captured image. If the order is the short-time captured image and the long-time captured image, the long-time captured image is output to 641 and the short-time captured image is output to 642.
[0057]
Reference numerals 631 and 632 separately output an area having a luminance value of a pixel equal to or greater than a specified value from an input image and an area having a luminance value of a pixel less than the value. By setting the specified value of 631 to 255, 631 outputs the overflow area X to 633 and the other area X^-(Represents X bar in the drawing) is output to 638. By setting the designated value of 632 to 256 / k, 632 outputs an area Y that is predicted to overflow if imaging is performed for a long time to 634 and 636.
[0058]
633 and 634 extract the outline of the region by a chain coding method (reference document: image analysis handbook pp. 579-580), which is a known method, and output it to 635.
[0059]
635 converts the contour shape output from 633 and 634 into Fourier descriptor (reference: image analysis handbook pp.581-582) expression data, and compares the data to perform pattern matching, thereby performing both contour shapes. Is output as affine transformation matrix data (reference document: latest trends in image processing algorithms, separate volume Plus E, pp. 58-72). Note that there are various known methods for contour line shape pattern matching, and the scope of claims of the present invention is not limited to this embodiment. In addition, since the amount of movement of the contour line shape that occurs between the two frames is very small, there is also a simplified method of outputting the shift amount of the centroid coordinates of the two regions as shift data assuming that the entire region moves uniformly without considering the deformation. The content of the data is not limited to this embodiment.
[0060]
Further, in the present embodiment, the case where there is one pattern matching target area is described, but when a plurality of contour lines are extracted at 633 and 634, the barycentric coordinates of the areas surrounded by the respective contour lines are obtained, It goes without saying that the above processing can be easily extended to a plurality of processes by performing the above-described pattern matching between the objects having the closest barycentric coordinates.
[0061]
Based on the affine transformation matrix data output from 635, 636 outputs the position and shape of region Y to 637 in accordance with the position and shape of region X.
[0062]
637 multiplies the luminance value of the pixel by k and outputs the result to 638. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0063]
638 is the output of 637 and the region X output from 631^-Are combined into one piece of image data and stored in 65. As a result, a high dynamic range image at the long imaging time is synthesized from the two images of the long imaging and the immediately following short imaging.
[0064]
641 and 642 are divided into an area having a luminance value of a pixel greater than or equal to a specified value and an area having a luminance value of a pixel less than the specified value from the input image and output. By setting the specified value of 641 to 255, the region X that has not overflowed 641^-Are output to 643 and 646. By setting the specified value of 642 to 256 / k, the region Y that is predicted to overflow if 642 is imaged for a long time is set to 647, and the other region Y^-(Represents Y bar in the drawing) is output to 644.
[0065]
643 and 644 extract the outline of the region by a chain coding method, which is a known method, and output it to 645.
[0066]
645 converts the contour shape output from 643 and 644 into Fourier descriptor expression data, and compares the data to perform pattern matching, thereby converting the position and shape displacement data of both contour shapes into affine transformation matrix data. Output as.
[0067]
646 is a region X based on the affine transformation matrix data output from 645.^-The position and shape of the region Y^-Is output to 648 in accordance with the position and shape.
[0068]
In step 647, the luminance value of the pixel in the region Y output from step 642 is multiplied by k and output to step 648. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0069]
648 combines the output of 646 and the output of 647 into a single piece of image data, and stores this in 65. As a result, a high dynamic range image at the short time imaging time is synthesized from the two images of the short time imaging and the immediately following long time imaging.
[0070]
  Of the present inventionClaims 3 to 5An embodiment of this method is shown in FIG. In FIG. 7, reference numeral 71 denotes an original video data storing process, 72 denotes a data output switching process, and 73 denotesClaim 4An embodiment for realizing step E of the method of FIG.Claim 5An embodiment for realizing the process F of the above method 75 is an output data storing process.
[0071]
  731, 732, and 733 are region separation processes,Claim 43 is an embodiment of the process C and the process D in the method. Reference numeral 734 denotes an intermediate image prediction generation process, 735 denotes an area cutout process, 736 denotes a pixel luminance value magnification correction process, and 737 denotes an image synthesis process.
[0072]
  741, 742, and 743 are region separation processes,Claim 53 is an embodiment of the process C and the process D in the method. 744 is an intermediate image prediction generation process, 745 is an area cutout process, 746 is a pixel luminance value magnification correction process, and 747 is an image composition process.
[0073]
The flow of processing according to this embodiment will be described. In this embodiment, the original image data has a dynamic range of 8 bits (possible pixel brightness values are 0 to 255) and has been subjected to inverse gamma correction. Also, a process for synthesizing the image data at the time from the image at a certain time and the image data immediately after the image will be described, where k is the imaging time ratio between the long imaging time and the short imaging time.
[0074]
71 stores video data taken by alternately changing the imaging time. If this method can be performed by real-time processing, the output (gamma corrected) process of the imaging time variable camera may be directly used instead of 71.
[0075]
72 sequentially extracts the image and the immediately preceding and immediately following frames from 71, and if the three images are in the order of the short-time captured image, the long-time captured image, and the short-time captured image, the previous short-time captured image is set to 731. The long-time captured image is output to 732 and the short-to-long image immediately after is output to 733. If the three images are in the order of a long-time captured image, a short-time captured image, and a long-time captured image, the immediately previous long-short captured image is set to 741, the short-time captured image is set to 742, and the immediately following long-time captured image is 743.
[0076]
Reference numerals 731, 732, and 733 are divided into an area having a luminance value of a pixel that is equal to or greater than a specified value and an area having a luminance value of a pixel that is less than the specified value from the input image. By setting the designated value of 731 to 256 / k, 731 outputs to region 734 the area Y that is predicted to overflow if imaging has been performed for a long time. By setting the specified value of 732 to 255, 732 sets the overflow area X to 735, and other areas X^-Is output to 737. By setting the designated value of 733 to 256 / k, 733 outputs to region 734 the region Y that is predicted to overflow if imaging has been performed for a long time.
[0077]
  734 predicts an intermediate image at a short imaging time from the image of region Y output from 731 and 733 by a dual prime prediction method (reference: information compression technology for digital broadcasting / Internet technology, pp. 218-219). Generate and output to 735. The dual prime prediction method collectively performs the operations 41, 42, 43, and 44 in FIG.Claim 4The image pattern matching operation described in the above method is performed by BMA (block matching algorithm) in the dual prime prediction method.
[0078]
In addition to BMA, there are various known methods for pattern matching of images, and there are known methods other than the dual prime prediction method for predicting and generating images, and the scope of claims of the present invention is in this embodiment. It is not limited.
[0079]
735 cuts out only the image of the area included in the area X output from 732 out of the intermediate image output from 734 and outputs it to 736.
[0080]
736 multiplies the luminance value of the pixel of the image output from 735 by k and outputs it to 737. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0081]
737 combines the output of 732 and the output of 736 into one piece of image data, and stores this in 75. As a result, a high dynamic range image at the long imaging time is synthesized from the three images of the long imaging and the short imaging immediately before and immediately after.
[0082]
741, 742, and 743 are separated from an input image into an area having a luminance value of a pixel greater than or equal to a specified value and an area having a luminance value of a pixel less than the specified value, and output. By setting the specified value of 741 to 255, 741 is overflow region X^-Is output to 744. By setting the specified value of 742 to 256 / k, if 742 has been imaged for a long time, the region Y that is predicted to overflow is changed to 746, and the other region Y^-Is output to 745. By setting the specified value of 743 to 255, 743 is overflow region X^-Is output to 744.
[0083]
  744 is a region X output from 741 and 743.^-An intermediate image at a short imaging time is predicted and generated from the image of the image by the dual prime prediction method, and is output to 745. The dual prime prediction method collectively performs the operations 51, 52, 53, and 54 in FIG.Claim 5The image pattern matching operation described in the above method is performed by BMA (block matching algorithm) in the dual prime prediction method.
[0084]
Reference numeral 745 denotes a region Y output from 742 of the intermediate image output from 744.^-Only the image of the area included in the image is cut out and output to 747.
[0085]
In step 746, the luminance value of the pixel of the image output from 742 is multiplied by k and output to 747. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0086]
747 combines the output of 745 and the output of 746 into one piece of image data, and stores this in 75. As a result, a high dynamic range image at a short imaging time is synthesized from the three images of the short imaging and the short imaging immediately before and immediately after.
[0087]
  Method of the present inventionOne embodiment is shown in FIG. In FIG. 8, the part showing the process of performing the same operation as FIG.Attached.
[0088]
  In FIG. 8, reference numeral 71 is an original video data storing process, 72 is a data output switching process, 731, 732 and 733 are region separation processes, and 734 is an intermediate image prediction generation process based on the dual prime prediction method. Reference numerals 831 to 835 denote internal processing steps of 734, 831 is a motion vector extraction process based on BMA, 832 is a motion compensation vector generation process, 833 is an intermediate image prediction generation process using the previous frame image, and 834 is a subsequent frame. An intermediate image prediction generation process using images, 835 is an intermediate image synthesis process. 836 is an image pattern similarity determination process, 81 is the pattern matching process of the contour line of the region X of the immediately preceding short-time captured image and the contour line of the region Y of the long-time captured image, and 82 is the region of the immediately subsequent short-time captured image A pattern matching process of an X contour line and a contour line of an area Y of a long-time captured image, 83 a processing operation selection process, and 84 synthesizing a high dynamic range image from the long-time captured image and the immediately preceding short-time captured image. 85 is a process of realizing method 1, and a high dynamic range image is synthesized from a long-time captured image and a short-time captured image immediately after the long-time captured image.Claims 1, 2A process 75 for realizing the above method is an output data storing process 75.
[0089]
  Using this figure, 3 frames are arranged in the order of short-time image, long-time image, and short-time image.The method of the present invention from an imageBy implementingClaim 3Use the image immediately before or afterClaims 1, 2An example of switching to this method will be described.
[0090]
The image data of three consecutive frames from 71 is taken out 72, the immediately preceding short-time captured image is output to 731, the long-time captured image is output to 732, and the immediately subsequent short-time captured image is output to 733.
[0091]
731 outputs to region 831 and 81 a region Y that is predicted to have overflowed if the image was captured for a long time. In step 733, the region Y predicted to have overflowed if the image was captured for a long time is output to 831 and 82.
[0092]
  734 is an intermediate image prediction generation process based on the dual prime prediction method, internal blocks 831 to 835 are known processing processes, and 836 isThe present inventionThis is a new process constituting the method. First, the operation of the conventional 734 will be briefly described.
[0093]
831 performs pattern matching on the image of the two areas by the block pattern matching method, calculates a shift vector for each block, and outputs 832, and MAE (average absolute value) which is a residual of the matching results of the two areas. Error) is output to 836. In 832, a shift amount (compensation vector) for making an image at the imaging time of a long-time captured image is separately calculated for each of the immediately preceding image and the immediately following image, and is output to 833 and 834.
[0094]
  In step 833, the region Y output from 731 is moved and deformed into a partial intermediate image using the compensation vector, and output to 835. 834 uses the compensation vector to move and transform the region Y output from 733 into a partial intermediate image and output it to 835. In step 835, one intermediate image is generated by combining two partial intermediate images. Or later,Claim 3If this method is used, the process proceeds along the flow of 735, 736, and 737 in FIG.
[0095]
  next, The present inventionA case where the operation of the method is performed will be described. 836 examines the MAE for each block output from 831. If the sum of MAEs of all blocks> v with respect to a predetermined threshold value v, it is determined that the similarity between the two regions is low 734. Subsequent processing is stopped, and 81 and 82 are commanded to operate.
[0096]
Note that the similarity can be given not only by MAE but also by a known method such as MSE (mean square error) or a cross-correlation function. In addition, the similarity determination can be determined not only by these error components, but also from the variance of the movement vector for each block generated in 831. However, if the technique can express the similarity of two regions. For example, the method of expressing the similarity and the determination method are not limited to the present embodiment.
[0097]
81 extracts the contour shape of the two regions, calculates the amount of movement deformation by matching the contour pattern, and outputs it to 83. Note that the contour line extraction process and the amount of movement deformation are pattern matching based on the coding chain method and the Fourier descriptor transform method described in the embodiment of FIG.
[0098]
82 extracts the contour shape of the two regions, calculates the amount of movement deformation by matching the contour pattern, and outputs it to 83.
[0099]
By performing similarity determination 83, an image having a higher similarity to the long-time captured image is determined from the immediately preceding short-time captured image and the immediately following short-time captured image. The similarity determination is made with the smaller moving deformation amount. For example, when the amount of movement deformation is given by an affine transformation matrix as in the embodiment described with reference to FIG. 6, the sum of squares of the values of all the elements of the normalized matrix is calculated and used as the similarity. In addition to the sum of squares, it is possible to express the similarity, and when the amount of movement deformation is given by a data structure other than the affine transformation determinant, the expression of the similarity changes correspondingly. However, the present invention is not limited to this embodiment as long as it is an expression expressing the similarity between two contour lines.
[0100]
83, when the similarity calculated from 81 is larger than the similarity calculated from 82, it is determined that the similarity between the long-time captured image and the immediately preceding short-time captured image is larger, and 84 is operated. If the similarity calculated from 82 is larger, it is determined that the similarity between the long-time captured image and the immediately subsequent short-time captured image is larger, and 85 is operated.
[0101]
84 synthesizes a high dynamic range image from the long-time captured image and the previous short-time captured image by the process of the embodiment of FIG.
[0102]
85 synthesizes a high dynamic range image from the long-time captured image and the short-time captured image immediately after by the process of the embodiment of FIG.
[0103]
FIG. 9 shows an explanatory diagram for generating a high dynamic range image from an original image when the method of this embodiment is used. Reference numerals 91 to 96 are original images formed by alternately repeating a short-time captured image and a long-time captured image. Here, 91, 93, and 95 are long-time images, and 92, 94, and 96 are short-time images. It is assumed that the spotlight is illuminated after the frame 94 and the region shape of the bright part in the image has changed significantly.
[0104]
  Using this embodiment, basically, a high dynamic range image of one frame is synthesized from an image of three frames.Claim 3Are used (97, 98, 911, 912). In the process of 99, it is determined that the similarity is low at 92 and 94, and the image having a higher similarity as 93 is determined as 92. A high dynamic range image of one frame is synthesized from the two frames of 93 and the previous 92 frames.Claims 1, 2The method is used. In the processing of 910, it is determined that the similarity is low at 93 and 95, and an image with a higher similarity than 94 is determined to be 95, and one frame of high dynamics of 2 frames of 94 and 95 immediately after is determined. Combining range imagesClaims 1, 2The method is used.
[0105]
  Of the present inventionClaims 6, 7One embodiment of this apparatus is shown in FIG. 101 in FIG. 10 is an original video data storage unit composed of a magnetic recording device or a memory, 102 is a control device using a CPU, 103 isClaim 6An embodiment for realizing the synthesizer A of the apparatus of FIG.Claim 7One embodiment for realizing the synthesizer B of the apparatus shown in FIG. 1 is an output data storage unit 105 constituted by a magnetic recording device or a memory.
[0106]
  1031 and 1032 are area separators,Claim 6It is one Embodiment of the detector C and the detector D in the apparatus of. 1033 and 1034 are contour shape extractors, 1035 is a contour shape pattern matching device, 1036 is a region shifter, 1037 is a pixel luminance value magnification corrector, and 1038 is an image synthesizer.
[0107]
  1041 and 1042 are area separators,Claim 7It is one Embodiment of the detector C and the detector D in the apparatus of. 1043 and 1034 are contour line extractors, 1045 is a contour shape pattern matching unit, 1046 is a region shifter, deformer, 1047 is a pixel luminance value magnification corrector, and 1048 is an image synthesizer.
[0108]
The flow of processing according to this embodiment will be described. In this embodiment, the original image data has a dynamic range of 8 bits (possible pixel brightness values are 0 to 255) and has been subjected to inverse gamma correction. Also, a process for synthesizing the image data at the time from the image at a certain time and the image data immediately after the image will be described, where k is the imaging time ratio between the long imaging time and the short imaging time.
[0109]
Reference numeral 101 stores video data obtained by alternately changing the imaging time. If the apparatus can be operated in real time, 101 may be omitted and the output (gamma corrected) of the imaging time variable camera may be directly connected to 102.
[0110]
102 sequentially extracts an image and an image of the immediately following frame from 101. If the two images are in the order of a long-time captured image and a short-time captured image, the long-time captured image is output to 1031 and the short-time captured image is output to 1032. To do. If the order is the short-time captured image and the long-time captured image, the long-time captured image is output to 1041 and the short-time captured image is output to 1042.
[0111]
1031 and 1032 are divided into an area having a luminance value of a pixel greater than or equal to a specified value from an input image and an area having a luminance value of a pixel less than the specified value and output. By setting the specified value of 1031 to 255, 1031 outputs the overflow area X to 1033 and the other area X^-Is output to 1038. By setting the designated value of 1032 to 256 / k, the region Y predicted to overflow if 1032 is imaged for a long time is output to 1034 and 1036.
[0112]
Reference numerals 1033 and 1034 extract the outline of the region by a chain coding method (reference document: image analysis handbook pp. 579-580), which is a known method, and output it to 1035.
[0113]
1035 converts the contour shapes output from 1033 and 1034 into Fourier descriptor (reference: image analysis handbook pp.581-582) expression data, and compares the data to perform pattern matching to thereby form both contour shapes. Is output as affine transformation matrix data (reference document: latest trends in image processing algorithms, separate volume Plus E, pp. 58-72). There are various well-known devices for contour pattern matching. As long as the contour line can be extracted, the scope of claims of the present invention is not limited to this embodiment. In addition, since the amount of movement and deformation of the contour line shape that occurs between the two frames is very small, the deformation is not taken into account, and a simplified device that outputs the shift amount of the center of gravity coordinates of the two regions as shift data assuming that the entire region moves uniformly (example) : MUSE HiVision encoder), and the scope of the claims of the present invention is not limited to this embodiment.
[0114]
Further, in the present embodiment, the case where there is one pattern matching target region is described, but when a plurality of contour lines are extracted at 1033 and 1034, the barycentric coordinates of the regions surrounded by the respective contour lines are obtained, Needless to say, the above pattern matching is performed between objects having the closest barycentric coordinates, so that the processing can be easily expanded to a plurality of processes.
[0115]
Based on the affine transformation matrix data output from 1035, 1036 outputs the position and shape of the region Y to 1037 in accordance with the position and shape of the region X.
[0116]
1037 multiplies the luminance value of the pixel by k and outputs it to 1038. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0117]
1038 is the output of 1037 and the region X output from 1031^-Are combined into one piece of image data and stored in 105. As a result, a high dynamic range image at the long imaging time is synthesized from the two images of the long imaging and the immediately following short imaging.
[0118]
Reference numerals 1041 and 1042 output the input image separately into an area having a luminance value of a pixel equal to or greater than a specified value and an area having a luminance value of a pixel less than the specified value. By setting the specified value of 1041 to 255, the region X where 1041 has not overflowed^-Are output to 1043 and 1046. By setting the designated value of 1042 to 256 / k, the area Y that is expected to overflow if 1042 is imaged for a long time is set to 1047, and the other area Y^-Is output to 1044.
[0119]
Reference numerals 1043 and 1044 extract the outline of the region by a chain coding method, which is a known method, and output it to 1045.
[0120]
1045 converts the contour shape output from 1043 and 1044 into Fourier descriptor expression data, and compares the data to perform pattern matching, thereby converting the position and shape displacement data of both contour shapes into affine transformation matrix data. Output as.
[0121]
1046 is a region X based on the affine transformation matrix data output from 1045.^-The position and shape of the region Y^-Output to 1048 according to the position and shape.
[0122]
1047 multiplies the luminance value of the pixel in region Y output from 1042 by k, and outputs the result to 1048. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0123]
1048 combines the output of 1046 and the output of 1047 into one piece of image data and stores it in 105. As a result, a high dynamic range image at the short time imaging time is synthesized from the two images of the short time imaging and the immediately following long time imaging.
[0124]
  Of the present inventionClaims 8 to 10One embodiment of this apparatus is shown in FIG. In FIG. 11, reference numeral 111 denotes an original video data storage composed of a magnetic recording device or a memory, 112 denotes a control device using a CPU, and 113 denotesClaim 9An embodiment for realizing the synthesizer E of the apparatus of FIG.Claim 10An embodiment for realizing the synthesizer F of the apparatus shown in FIG. 11 is an output data storage unit 115 including a magnetic recording device or a memory.
[0125]
  Reference numerals 1131, 1132, and 1133 are area separators,Claim 9It is one Embodiment of the detector C and the detector D in the apparatus of. Reference numeral 1134 denotes an intermediate image prediction generator, 1135 denotes a region segmenter, 1136 denotes a pixel luminance value magnification corrector, and 1137 denotes an image synthesizer.
[0126]
  1141, 1142 and 1143 are area separators,Claim 9It is one Embodiment of the detector C and the detector D in the apparatus of. Reference numeral 1144 denotes an intermediate image prediction generator, 1145 denotes a region segmenter, 1146 denotes a pixel luminance value magnification corrector, and 1147 denotes an image synthesizer.
[0127]
The flow of processing according to this embodiment will be described. In this embodiment, the original image data has a dynamic range of 8 bits (possible pixel brightness values are 0 to 255) and has been subjected to inverse gamma correction. Also, a process for synthesizing the image data at the time from the image at a certain time and the image data immediately after the image will be described, where k is the imaging time ratio between the long imaging time and the short imaging time.
[0128]
111 stores video data taken by alternately changing the imaging time. If the apparatus can be operated in real time, 111 may be omitted and the output of the imaging time variable camera (gamma corrected) may be directly connected to 112.
[0129]
112 sequentially extracts an image and an image of the immediately preceding and immediately following frames from 111. If the three images are in the order of a short-time captured image, a long-time captured image, and a short-time captured image, the previous short-time captured image is set to 1131. The long-time captured image is output to 1132, and the short-long image immediately after is output to 1133. Also, if the three images are in the order of a long-time image, a short-time image, and a long-time image, the previous long-short image is 1141, the short-time image is 1142, and the next long-time image is 1143 is output.
[0130]
Reference numerals 1131, 1132, and 1133 output the input image separately into an area having a luminance value of a pixel greater than a specified value and an area having a luminance value of a pixel less than the specified value. By setting the designated value of 1131 to 256 / k, 1131 outputs to Y 1134 an area Y that is predicted to overflow if imaging is performed for a long time. By setting the specified value of 1132 to 255, 1132 changes the overflow area X to 1135 and the other area X^-Is output to 1137. By setting the designated value of 1133 to 256 / k, 1133 outputs, to 1134, a region Y that is predicted to overflow if imaging has been performed for a long time.
[0131]
  Reference numeral 1134 denotes a dual prime prediction computing unit (reference: information compression technology for digital broadcasting / Internet technology, pp. 218-219), which is an intermediate image at a short imaging time from the image of the region Y output from 1131 and 1133. Prediction is generated and output to 1135. The dual prime prediction computing unit collectively performs the operations 41, 42, 43 and 44 of FIG. 4 described in the explanation of the operation. The BMA (Block Matching Algorithm) calculator in the dual prime calculatorClaim 9This corresponds to the image pattern matching device described in the apparatus.
[0132]
In addition to the BMA, there are various known devices for the image pattern matching device, and there are known devices other than the dual-prime prediction computing device for the image prediction generator. The scope of the claims of the present invention is not limited to this embodiment as long as it is an apparatus for performing the above.
[0133]
1135 cuts out only the image of the area included in the area X output from 1132 out of the intermediate image output from 1134 and outputs it to 1136.
[0134]
In 1136, the luminance value of the pixel of the image output from 1135 is multiplied by k and output to 1137. Thereby, the sensitivity difference between the images of two frames having different imaging times is corrected. The luminance values of the pixels that image the same luminance portion are the same.
[0135]
1137 combines the output of 1132 and the output of 1136 into one piece of image data and stores it in 115. As a result, a high dynamic range image at the long imaging time is synthesized from the three images of the long imaging and the short imaging immediately before and immediately after.
[0136]
Reference numerals 1141, 1142, and 1143 output the input image by separating it into an area having a luminance value of pixels greater than or equal to a specified value and an area having a luminance value of pixels less than the specified value. By setting the specified value of 1141 to 255, 1141 is overflow region X^-Is output to 1144. By setting the designated value of 1142 to 256 / k, the region Y that is predicted to overflow if 1142 is imaged for a long time is set to 1146, and the other region Y^-Is output to 1145. By setting the specified value of 1143 to 255, 1143 is overflow region X^-Is output to 1144.
[0137]
  1144 is an area X output from 1141 and 1143^-The intermediate image at the short time imaging time is predicted and generated from the first image by the dual prime prediction calculator, and is output to 1145. The dual prime prediction computing unit collectively performs the operations 51, 52, 53, and 54 in FIG. The BMA (Block Matching Algorithm) calculator in the dual prime calculatorClaim 10This corresponds to the image pattern matching device described in the apparatus.
[0138]
Reference numeral 1145 denotes an area Y output from 1142 among the intermediate images output from 1144.^-Only the image of the area included in is cut out and output to 1147.
[0139]
1146 multiplies the luminance value of the pixel of the image output from 1142 by k, and outputs the result to 1147. As a result, the sensitivity difference between the images of two frames having different imaging times is corrected, and the luminance values of the pixels capturing the same luminance portion are the same.
[0140]
1147 combines the output of 1145 and the output of 1146 into a piece of image data and stores it in 115. As a result, a high dynamic range image at a short imaging time is synthesized from the three images of the short imaging and the short imaging immediately before and immediately after.
[0141]
  Device of the present inventionFIG. 12 shows an embodiment in which three frames of images arranged in the order of a short-time captured image, a long-time captured image, and a short-time captured image are processed. In FIG. 12, parts that are the same as those in FIG.AttachedYes.
[0142]
  In FIG. 12, reference numeral 111 denotes an original video data storage composed of a magnetic recording device or a memory, 112 denotes a control device using a CPU, 1131, 1132, and 1133 denote area separators, and 1134 denotes an intermediate based on the dual prime prediction method. It is an image prediction generator. Reference numerals 1231 to 1235 denote apparatuses for performing the internal processing of 1134, 1231 is a motion vector extractor based on BMA, 1232 is a motion compensation vector generator, 1233 is an intermediate image prediction generator using a previous frame image, and 1234 is An intermediate image prediction generator 1235 using the rear frame image is an intermediate image synthesizer. 1236 is an image pattern similarity determination unit, 121 is a pattern matching unit for the contour line of the region X of the immediately preceding short-time captured image and the contour line of the region Y of the long-time captured image, and 122 is the region of the immediate short-time captured image. X contour line and long-time captured image area Y contour pattern matching unit, 123 is a processing operation selector, and 124 synthesizes a high dynamic range image from the long-time captured image and the previous short-time captured image.Claims 6, 7The device 125 synthesizes a high dynamic range image from a long-time captured image and a short-time captured image immediately thereafter.Claims 6, 7Reference numeral 115 denotes an output data storage unit constituted by a magnetic recording device or a memory.
[0143]
  Using this figure, from the image of 3 frames arranged in the order of short-time captured image, long-time captured image, and short-time captured imageThe present inventionWith the equipment ofClaim 8Use the image immediately before or afterClaims 6, 7An example of switching the processing operation to this apparatus will be described.
[0144]
Three consecutive frames 112 are extracted from 111, and the immediately preceding short-time captured image is output to 1131, the long-time captured image is output to 1132, and the immediately subsequent short-time captured image is output to 1133. 1131 outputs to region 1231 and 121 a region Y that is predicted to have overflowed if the image was captured for a long time. A region 1133 outputs the region Y predicted to have overflowed if imaged for a long time to 1231 and 122.
[0145]
  Reference numeral 1134 denotes an intermediate image prediction generation device based on the dual prime prediction method. Internal blocks 1231 to 1235 are known processing devices.Device of the present inventionIt is a new device that constitutes. First, the operation of the conventional 1134 will be briefly described.
[0146]
Reference numeral 1231 performs pattern matching on the image of the two regions by the BMA method, calculates a shift vector for each block, and outputs MAE (average value absolute value error), which is a residual of the matching result of the two regions, to 1232. 1236. Reference numeral 1232 calculates a shift amount (compensation vector) for making an image at the imaging time of a long-time captured image separately for each of the immediately preceding image and the immediately following image and outputs it to 1233 and 1234.
[0147]
  1233 uses the compensation vector to move and transform the region Y output from 1131 into a partial intermediate image and output it to 1235. 1234 uses the compensation vector to move and transform the region Y output from 1133 into a partial intermediate image and output it to 1235. In step 1235, one intermediate image is generated by combining two partial intermediate images. Or later,Claim 8When the process continues in the apparatus of FIG. 11, the process proceeds along the flow of 1135, 1136, and 1137 in FIG.
[0148]
  next,The present inventionA case where the operation of the apparatus is performed will be described. 1236 examines the MAE for each block output from 1231. If the sum of MAEs of all blocks> v with respect to a predetermined threshold value v, it is determined that the similarity of the two regions is low 1132 The process is stopped, and operations are commanded to 121 and 122.
[0149]
121 extracts the contour line from the region Y output from 1131 and the region X output from 1132 by the contour line extractor by the coding chain method, and calculates the movement deformation amount by the pattern matching device by the Fourier descriptor transform method. And output to 123.
[0150]
122 extracts the contour line from the region Y output from 1133 and the region X output from 1132 by the contour line extractor by the coding chain method, and calculates the movement deformation amount by the pattern matching device by the Fourier descriptor transform method. And output to 123.
[0151]
In step 123, similarity determination is performed using the moving deformation amount, thereby determining an image having a higher similarity to the long-time captured image from the immediately preceding short-time captured image and the immediately following short-time captured image. The similarity determination is made with the smaller moving deformation amount. For example, when the amount of movement deformation is given by an affine transformation matrix as in the embodiment described with reference to FIG. 10, the sum of squares of the values of all elements of the normalized matrix is calculated and used as the similarity. In addition to the sum of squares, it is possible to express similarity. Also, when the amount of movement deformation is given by a data structure other than the affine transformation determinant, the expression of similarity changes correspondingly, but if the expression expresses the similarity of two contour lines, the present invention The scope of the claims is not limited to this embodiment.
[0152]
If the similarity calculated from 121 is greater than the similarity calculated from 122, it is determined that the similarity between the long-time captured image and the immediately preceding short-time captured image is greater and 124 is operated. If the similarity calculated from 122 is larger, it is determined that the similarity between the long-time captured image and the short-time captured image immediately after is larger, and 125 is operated.
[0153]
Reference numeral 124 synthesizes a high dynamic range image from the long-time captured image and the immediately preceding short-time captured image by the apparatus of the embodiment of FIG.
[0154]
125 synthesizes a high dynamic range image from the long-time captured image and the short-time captured image immediately after it by the apparatus of the embodiment of FIG.
[0155]
Note that the present invention can be realized by configuring a part or all of the functions of each unit in the apparatus shown in FIGS. 10 to 12 by a computer program and executing the program using the computer. It is needless to say that the processing procedure of the method according to the processing steps described with reference to FIGS. 6 to 8 can be configured by a computer program, and the program can be executed by the computer. A program for causing the computer to execute the processing procedure is a recording medium readable by the computer, such as FD (floppy disk (registered trademark)), MO, ROM, memory card, CD, DVD, removable disk, etc. To record, store and distribute It is possible. It is also possible to provide the above program through a network such as the Internet or electronic mail.
[0156]
【The invention's effect】
  In the conventional method and apparatus, as shown by 11 in FIG. 1 (a), one frame image is synthesized from a two frame image by a single synthesis algorithm, so that the frame rate is halved. And the equipmentClaims 1, 2And the methodClaims 6, 7In the apparatus of FIG.Claim 3And the methodClaim 8Both devices can generate a high dynamic range image at the same frame rate as the original image, as indicated by 13 in FIG.
[0157]
  In addition, the present inventionClaims 1, 2Methods andClaims 6, 7In this apparatus, as shown in FIG. 2, images of two frames having different imaging times are not synthesized as they are, but are synthesized after predicting and generating an image that should be obtained at the same imaging time. Do not synthesize natural images.
[0158]
  Furthermore,Claims 1, 2Methods andClaims 6, 7The device of can provide an accurate composite image when the movement of the contour line part and the movement in the contour line are the same,Claim 3Methods andClaim 8This apparatus can provide an accurate composite image even when the motion of the contour line portion and the motion within the contour line are different, so that a higher dynamic range image with higher quality can be provided.
[0159]
  On the other hand,Claim 3Methods andClaim 8This device requires three consecutive frames of video, so it cannot be applied when video switching points or image content changes discontinuously between frames, or even if it is forcibly applied. There is also a problem that only a low-quality composite image can be obtained. Therefore,Method and apparatus of the present inventionAnd detecting the discontinuous portion by examining the similarity between the two images,Claim 3And the methodClaims 1, 2Method orClaim 8With equipmentClaims 6, 79 is adaptively switched to perform processing, so that even if there is a discontinuous portion between images as shown in FIG. It is possible to synthesize.
[Brief description of the drawings]
FIG. 1 (1) shows a conventional method, and (2) shows the present invention.Claims 1, 2(3) is the method of the present invention.Claim 3It is the figure explaining the response | compatibility of the original image data at the time of using each method, and synthetic | combination image data.
FIG. 2 of the present inventionClaims 1, 2Methods andClaims 6 and 7It is the figure explaining the effect | action of this apparatus.
FIG. 3 of the present inventionClaim 3Methods andClaim 8It is the figure explaining the effect | action of this apparatus.
FIG. 4 of the present inventionClaim 4Methods andClaim 9It is the figure explaining the effect | action of this apparatus.
FIG. 5 shows the present invention.Claim 5Methods andClaim 10It is the figure explaining the effect | action of this apparatus.
FIG. 6 of the present inventionClaims 1, 2It is the figure which showed one Embodiment of the method of.
[Fig. 7] of the present invention.Claims 3, 4, and 5It is the figure which showed one Embodiment of the method of.
[Fig. 8]Method of the present inventionIt is the figure which showed one Embodiment.
FIG. 9Method and apparatus of the present inventionFIG. 6 is a diagram showing a correspondence relationship between original image data and composite image data in the case of using.
FIG. 10 shows the present invention.Claims 6 and 7It is the figure which showed one Embodiment of the apparatus of.
FIG. 11 shows the present invention.Claims 8, 9, 10It is the figure which showed one Embodiment of the apparatus of.
FIG.Device of the present inventionIt is the figure which showed one Embodiment.

Claims (12)

In a method for generating a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device capable of variably controlling the imaging time,
A process A for synthesizing a high dynamic range image at a long imaging time from two images of a long imaging and a short imaging immediately after or immediately before,
And two different synthesis processes, which are different processes B of synthesizing a high dynamic range image at a short imaging time than two images of short-time imaging and immediately or immediately before long-time imaging
Perform process A for images at the long imaging time, execute process B for images at the short imaging time ,
In process A,
A process C for detecting an overflow region X in an image captured for a long time;
A process D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
Performing a pattern matching of contour lines of two regions,
In Step C, the long-time captured image is separated into the region X and the other region X ⁻
The region Y is extracted from the short-time captured image immediately after or immediately before by the process D,
Position of the area Y by a process of performing pattern matching, the position of the shaped regions X, after the magnification correction of the luminance value of the pixel after conforming to the shape, area X ^- by combining with a long time after an imaging Or, the image at the long imaging time is synthesized from the two short imaging images just before
A method of generating a high dynamic range video. In a method of generating a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device capable of variably controlling the imaging time,
A process A for synthesizing a high dynamic range image at a long imaging time from two images of a long imaging and a short imaging immediately after or immediately before;
And two different synthesis processes, B, which synthesize a high dynamic range image at a short imaging time than two images of short-time imaging and immediately or immediately before long-time imaging,
Perform process A for images at the long imaging time, perform process B for images at the short imaging time,
In process B,
A process C for detecting an overflow region X in an image captured for a long time;
A process D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
Performing a pattern matching of contour lines of two regions,
In Step D, the short-time captured image is separated into the region Y and the other region Y ⁻
In Step C, a long-time captured image immediately after or immediately before is separated into a region X and another region X ⁻ ,
After matching the position and shape of the region X ^{− with} the position and shape of the region Y ⁻ by the process of pattern matching, imaging is performed for a short time by combining with the region Y after the magnification correction of the luminance value of the pixel. And a method of generating a high dynamic range image, comprising synthesizing an image at a short imaging time from two images of long imaging immediately after or immediately before . In a method for generating a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging device capable of variably controlling the imaging time,
A process E for synthesizing a high dynamic range image at a long imaging time from three images of a long imaging, a short imaging immediately before and a short imaging just after,
Two synthesis processes different from each other in the process F of synthesizing a high dynamic range image at a short imaging time from three images of a short imaging, a long imaging immediately before and a long imaging immediately after.
A method of generating a high dynamic range image, wherein the process E is executed for an image at a long imaging time, and the process F is executed for an image at a short imaging time. In process E,
A process C for detecting an overflow region X in a long-time image,
A process D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
A process of performing pattern matching of two image areas,
In Step C, an image captured for a long time is separated into a region X and a region X ⁻ other than that,
In step D, the region Y1 is cut out from the immediately preceding short-time image,
The region Y2 is cut out from the short-time captured image immediately after the process D,
An intermediate image Y3 at a long imaging time is generated from the two areas Y1 and Y2 through the process of pattern matching of the two areas,
Cut out region overlapping the position to a region X from the intermediate image Y3, after performing the magnification correction of the luminance value of the pixel, region X ^- and by synthesizing a long time imaging and the immediately preceding short image and immediately after the short The method for generating a high dynamic range image according to claim 3 , wherein an image at a long imaging time is synthesized from three temporal imaging images. In process F,
A process C for detecting an overflow region X in a long-time image,
A process D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
A process of performing pattern matching of two image areas,
Separated into, ^- the image of the short imaging region Y and the other region Y by a process D
Separated into, ^- other areas X1 long time image of the imaging an area X1 immediately before the process C
Separated into, ^- other areas X2 long time image of the imaging an area X2 immediately after the process C
An intermediate image X3 ⁻ at a short time imaging time is generated from the two regions X1 ⁻ and X2 ⁻ by the process of performing pattern matching of the images of the two regions,
Intermediate image X3 ^- the region Y ^- after cutting out a region overlapping the positionally by synthesizing an area Y after the magnification correction of the luminance values of the pixels, and a long imaging short imaging before and immediately The method for generating a high dynamic range image according to claim 3 , wherein an image at a short imaging time is synthesized from three short imaging images immediately after. In an apparatus that generates a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging apparatus that can variably control the imaging time,
A synthesizer A that synthesizes a high dynamic range image at a long imaging time from two images of long imaging and immediately or immediately before short imaging;
Two synthesizers different from synthesizer B for synthesizing a high dynamic range image at a short time imaging time than two images of short time imaging and immediately after or immediately before long time imaging,
Long using the synthesizer A is an image of the imaging time, an image in a short time imaging time have use the synthesizer B,
Synthesizer A
A detector C for detecting an overflow region X in an image captured for a long time;
A detector D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
A pattern matching unit that performs pattern matching of contour lines of two regions,
The long-time captured image is separated into the region X and the other region X ⁻ using the detector C, the region Y is extracted from the short-time captured image immediately after or immediately before using the detector D, and the pattern matching unit is used. position of the area Y Te, the position of the shaped region X, after the magnification correction of the luminance value of the pixel after conforming to the shape, area X ^- and by synthesizing a long time imaging and immediately or shortly before a brief An apparatus for generating a high dynamic range video, wherein the image at a long imaging time is synthesized from two captured images . In an apparatus that generates a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging apparatus that can variably control the imaging time,
Long and synthesizer A for synthesizing a high dynamic range image in long imaging times than two images of short imaging immediately or just before the imaging,
Two synthesizers different from synthesizer B for synthesizing a high dynamic range image at a short time imaging time than two images of short time imaging and immediately after or immediately before long time imaging,
For the image at the long imaging time, use the synthesizer A, for the image at the short imaging time, use the synthesizer B,
Synthesizer B
A detector C for detecting an overflow region X in an image captured for a long time;
A detector D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
A pattern matching unit that performs pattern matching of contour lines of two regions,
The detector D is used to separate the short-time captured image into the region Y and the other region Y ⁻ , and the detector C is used to separate the long-time captured image immediately after or immediately before into the region X and the other region X ⁻ , By combining the position and shape of the region X ^{− with} the position and shape of the region Y ⁻ using the pattern matching unit and then combining with the region Y after the magnification correction of the luminance value of the pixel, An apparatus for generating a high dynamic range video, wherein an image at a short imaging time is synthesized from two images of imaging and long-time imaging immediately after or immediately before . In an apparatus that generates a high dynamic range image by alternately repeating long-time imaging and short-time imaging using an imaging apparatus that can variably control the imaging time,
A synthesizer E for synthesizing a high dynamic range image at a long imaging time from three images of a long imaging, a short imaging immediately before and a short imaging immediately after;
Two synthesizers different from the synthesizer F for synthesizing a high dynamic range image at a short time imaging time than the three images of the short time imaging, the immediately preceding long time imaging, and the immediately following long time imaging,
An apparatus for generating a high dynamic range image, wherein a synthesizer E is used for an image at a long imaging time, and a synthesizer F is used for an image at a short imaging time. Synthesizer E
A detector C for detecting an overflow region X in an image captured for a long time;
A detector D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
A pattern matching unit that performs pattern matching of images in two regions,
A long-time image is separated into region X and other region X ⁻ using detector C, region Y1 is cut out from the previous short-time image using detector D, and detector D is used. A region Y2 is cut out from a short-time captured image immediately after, an intermediate image Y3 at a long-time imaging time is generated from the region Y1 and the region Y2 using the pattern matching unit, and a region that overlaps the region X from the intermediate image Y3 , And after correcting the magnification of the luminance value of the pixel, by combining with the region X ⁻ , an image at the long-time imaging time is obtained from the three images of the long-time imaging, the immediately preceding short-time image, and the immediately following short-time imaging. The high dynamic range video generation apparatus according to claim 8 , wherein the high dynamic range video generation apparatus is combined. The synthesizer F
A detector C for detecting an overflow region X in an image captured for a long time;
A detector D for detecting a region Y that is predicted to overflow if the image is captured for a long time in an image captured for a short time;
A pattern matching unit that performs pattern matching of images in two regions,
Using the detector D, the short-time image is separated into the region Y and the other region Y ⁻ , and using the detector C, the immediately previous long-time image is separated into the region X1 and the other region X 1 ⁻ , separated into regions X1 ^{- -} detector other regions X2 long time image of the imaging an area X2 immediately using C and region X2 ^- intermediate image in a short time capturing time by using the pattern matching unit from X3 ^- Is extracted from the intermediate image X3 ⁻ and the region Y ⁻ is overlapped with the region Y ⁻ , and then combined with the region Y after the pixel luminance value magnification correction is performed. The high dynamic range video generation device according to claim 8 , wherein the image at the short time imaging time is synthesized from the three images of the long time imaging and the short time imaging immediately thereafter. An execution program for a high dynamic range video generation method, characterized in that a program for causing a computer to execute the process in the high dynamic range video generation method according to any one of claims 1 to 5 is provided. . A program for causing a computer to execute the process in the high dynamic range video generation method according to any one of claims 1 to 5 ,
A recording medium recording an execution program for a high dynamic range video generation method, wherein the program is recorded on a recording medium readable by the computer.

Images