US20120314945A1 - Apparatus and method for image processing - Google Patents
Apparatus and method for image processing Download PDFInfo
- Publication number
- US20120314945A1 US20120314945A1 US13/398,444 US201213398444A US2012314945A1 US 20120314945 A1 US20120314945 A1 US 20120314945A1 US 201213398444 A US201213398444 A US 201213398444A US 2012314945 A1 US2012314945 A1 US 2012314945A1
- Authority
- US
- United States
- Prior art keywords
- tone correction
- image
- map
- ghost
- right image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 230000008569 process Effects 0.000 claims description 17
- 238000009499 grossing Methods 0.000 claims description 12
- 230000002457 bidirectional effect Effects 0.000 claims description 11
- 230000006870 function Effects 0.000 description 38
- 238000002156 mixing Methods 0.000 description 10
- 230000001186 cumulative effect Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 206010047571 Visual impairment Diseases 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000008570 general process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
Definitions
- Example embodiments of the following description relate to an apparatus and method for correcting a tone of a panoramic image, and more specifically, an apparatus and method for performing tone correction before blending a plurality of images to generate a panoramic image.
- a panoramic image may be generated by blending a plurality of images acquired by photographing a subject at several positions or from several angles.
- an artifact may occur at a seam connecting the plurality of images, which may cause a user to sense a difference because each of the plurality of images may have been acquired under differing conditions, such as, different exposure values, different gradations, and different lens shadings.
- a tone correction may need to be performed, among the plurality of images, before blending the plurality of images.
- the example embodiments may include an apparatus for processing an image, the apparatus including a ghost information acquisition unit to acquire, from an overlapping region of a left image and a right image, ghost information based on an optimal threshold and a cost map; a tone correction map generator to generate a tone correction map, based on a global tone correction function generated using the ghost information, and based on a local tone correction level generated using local information of an optimal seam; and a bidirectional corrector to correct the left image and the right image using a final tone correction map generated by applying, to the tone correction map, a weighting based on the left image and a weighting based on the right image.
- a ghost information acquisition unit to acquire, from an overlapping region of a left image and a right image, ghost information based on an optimal threshold and a cost map
- a tone correction map generator to generate a tone correction map, based on a global tone correction function generated using the ghost information, and based on a local tone correction level generated using local information of an optimal seam
- the cost map may be generated by overlapping a region of the left image and the right image, such that a similarity between the left image and the right image is reflected on the cost map.
- the left image and the right image may correspond to images acquired by photographing a stationary subject or a moving subject at a plurality of photographing positions to generate a panoramic image.
- the similarity between the left image and the right image may be based on a level of color similarity and gradient similarity between the corresponding pixels in the overlapping region of the left image and the right image.
- the ghost information acquisition unit may include an optimal seam assumption unit to assume a path having a greatest similarity on the cost map to be an optimal seam of the left image and the right image; and a binarization unit to generate a ghost map by binarizing the cost map, based on the optimal threshold that is estimated using an optimal threshold estimation process.
- the tone correction map generator may include a global calculation unit to calculate a global tone correction function, in which a tone difference, between a pixel of the left image and a pixel of the right image, corresponding to pixels other than a ghost pixel in the overlapping region, is reflected using the ghost information.
- the tone correction map generator may further include a smoothing unit to smooth a value of the calculated global tone correction function through a smoothing filter.
- the tone correction map generator may include a local calculation unit to generate the local information of the optimal seam by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam.
- the tone correction map generator may further include a correction level calculation unit to calculate a local tone correction level by comparing local information and a value of a global tone correction function, of a pixel included in the optimal seam.
- the bidirectional corrector may include a weighting application unit to apply, to the tone correction map, a weighting varying according to a distance between the left image and the overlapping region, and a weighting varying according to a distance between the right image and the overlapping region.
- the example embodiments may also include a method of processing an image, the method including acquiring, from an overlapping region of a left image and a right image, ghost information, based on an optimal threshold and a cost map; generating a tone correction map, based on a global tone correction function generated using the ghost information, and based on a local tone correction level generated using local information of an optimal seam; and correcting the left image and the right image using a final tone correction map generated by applying, to the tone correction map, a weighting based on the left image and a weighting, based on the right image.
- the generating of the cost map may include overlapping a region of the left image and the right image, such that a similarity between the left image and the right image is reflected on the cost map.
- the acquiring may include assuming a path having a greatest similarity on the cost map to be an optimal seam of the left image and the right image, and generating a ghost map by binarizing the cost map based on the optimal threshold that is estimated using an optimal threshold estimation process.
- the generating may include calculating a global tone correction function in which a tone difference between a pixel of the left image and a pixel of the right image, corresponding to pixels other than a ghost pixel in the overlapping region is reflected, using the ghost information.
- the generating may include generating the local information of the optimal seam by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam.
- the generating of the tone correction map may further include calculating a local tone correction level by comparing local information and a value of a global tone correction function, of a pixel included in the optimal seam.
- the correcting may include applying, to the tone correction map, a weighting varying according to a distance between the left image and the overlapping region and a weighting varying according to a distance between the right image and the overlapping region.
- FIG. 1 parts (a) and (b), illustrate a panoramic image, according to a conventional art
- FIG. 2 illustrates an issue occurring due to a moving subject, according to a conventional art
- FIG. 3 illustrates pixels on the same position in an overlapping region of input images, according to a conventional art
- FIG. 4 illustrates an apparatus for processing an image, according to example embodiments
- FIG. 5 illustrates an operation of generating a ghost map, according to example embodiments
- FIG. 6 illustrates an operation of generating a global tone correction function, according to example embodiments
- FIG. 7 illustrates an operation of generating local information of an optimal seam, according to example embodiments
- FIG. 8 illustrates a tone correction map, according to example embodiments
- FIG. 9 illustrates a bidirectional correction of an input left image and an input right image, according to example embodiments.
- FIG. 10 illustrates an image processing method, according to example embodiments.
- FIG. 11 illustrates a panoramic image having a corrected tone, according to example embodiments.
- FIGS. 1A and 1B illustrate an example of a panoramic image, according to a conventional art.
- FIG. 1A illustrates a panoramic image resulting from matching two images without a tone correction.
- FIG. 1B illustrates a panoramic image resulting from estimating an optimal seam and matching two images without a tone correction.
- An overlapping region of two images acquired by photographing the same subject may have different exposure values or different gradation levels for each image, even though both images correspond to the same subject.
- an unnatural line may occur in an intermediate portion of an image, as illustrated in FIGS. 1A and 1B .
- FIG. 2 illustrates an issue occurring due to a moving subject, according to a conventional art.
- a panoramic image may be generated by blending a target image and an object image.
- a portion inside of a box in the target image and a portion inside of a box in the object image correspond to an overlapping portion.
- a tone in the portion inside of the box in the object image may be corrected by referencing the portion inside of the box in the target image.
- a moving subject not visible from the portion inside of the box in the target image may be present in the portion inside of the box in the object image.
- the portion inside of the box in the object image and the portion inside of the box in the target image may contain different content even though the corresponding images are acquired by photographing a subject at the same location.
- a tone correction may not be performed since the moving subject included in the portion inside of the box in the object image is not visible from the target image.
- a tone correction may be performed by referencing the target image, and excluding a portion corresponding to the moving subject.
- a general tone correction process may include generating a look-up table of the object image, related to the target image, through histogram matching, and correcting a tone of the object image, based on the look-up table.
- a general tone correction process may include correcting a region of the object image, in an overlapping region of the object image, and the target image through a histogram used for performing matching, estimating a color conversion matrix using a corrected region of the object image, and correcting a tone of the object image using the estimated color conversion matrix.
- a general tone correction process may include correcting a tone of the object image by calculating a weighting that uses a tone difference between the object image and the target image in a portion of a seam, and a distance from a pixel to be corrected.
- FIG. 3 illustrates pixels on the same position in an overlapping region of input images, according to a conventional art.
- an inaccurate tone correction may be performed due to a matching error of one to two pixels occurring in a boundary of an object, even though an image matching may be performed successfully in an overlapping region of an input image 310 and an input image 320 of FIG. 3 .
- a general tone correction process may have an issue, such as, an image becoming darker or brighter as a panorama may be added since a tone of an object image is corrected for a single target image only.
- FIG. 4 illustrates an apparatus for processing an image, according to example embodiments.
- the apparatus for processing an image may include a ghost information acquisition unit 410 , a tone correction map generator 420 , and a bidirectional corrector 430 .
- the ghost information acquisition unit 410 may generate a cost map on an overlapping region of a left image and a right image.
- the left image and the right image may correspond to input images.
- a similarity between the left image and the right image may be reflected on the cost map.
- an optimal seam estimation process may be used among processes for naturally joining a left image and a right image.
- the optimal seam estimation process may include estimating an optimal seam by connecting similar pixels, based on a level of color similarity and gradient similarity between pixels, corresponding to the left image and the right image in an overlapping region of the left image and the right image.
- the ghost information acquisition unit 410 may acquire ghost information based on an optimal threshold.
- the ghost information may indicate information about a region having a relatively small value of similarity between the left image and the right image.
- the similarity may be determined, based on various elements indicating a character of an image such as a color, saturation, brightness, and a tone of each image.
- the left image and the right image may correspond to images acquired by photographing a stationary subject or a moving subject at a plurality of photographing positions.
- a panoramic image may be generated by putting the left image and the right image together.
- the ghost information acquisition unit 410 may include an optimal seam assumption unit 411 and a binarization unit 413 .
- the optimal seam assumption unit 411 may assume a path having a greatest similarity on the cost map to be an optimal seam of the left image and the right image.
- the binarization unit 413 may binarize the cost map, based on the optimal threshold that may be estimated, using an optimal threshold estimation process.
- a binarization of the cost map may indicate a conversion of the cost map into grayscale.
- the binarization unit 413 may binarize the cost map by comparing similarity of each pixel on the cost map with the optimal threshold, thereby generating a ghost map.
- the binarization unit 413 may binarize the cost map to “1” when the similarity of each pixel on the cost map may be greater than the optimal threshold, and may binarize the cost map to “0” when the similarity of each pixel on the cost map may be less than the optimal threshold.
- a portion binarized to “1” on the ghost map may be expressed in a relatively bright color
- a portion binarized to “0” on the ghost map may be expressed in a relatively dark color.
- the binarization unit 413 may generate the ghost map through binarization, and thus, a complicated arithmetic operation may not be used to rapidly generate the ghost map.
- the ghost map may include information about a pixel associated with a tone correction and ghost information unrelated to the tone correction. For example, a portion binarized to “1” may indicate pixels associated with the tone correction, and a portion binarized to “0” may indicate a ghost portion unrelated to the tone correction.
- the tone correction map generator 420 may generate a global tone correction function based on the ghost information in the overlapping region of the left image and the right image.
- the global tone correction function may reflect all pixels other than a ghost pixel in the overlapping region. For example, when a panoramic image is generated in a horizontal direction using a plurality of images photographed in horizontal directions, a tone difference in a vertical direction may not be an issue while a tone difference in a horizontal direction may be an issue. The tone difference in a vertical direction may be trivial.
- the tone correction map generator 420 may generate the global tone correction function by calculating a cumulative mean of tone differences, of portions other than a ghost pixel, in a vertical direction on the overlapping region of the left image and the right image.
- the global tone correction function may reflect a tone difference between the left image and the right image in an entire overlapping region.
- the tone correction map generator 420 may generate local information of the optimal seam.
- the local information may be generated by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel included in the optimal seam.
- the local information may indicate information about a tone difference between pixels included in the optimal seam.
- the tone difference may refer to a tone difference between a pixel of the left image and a pixel of the right image, corresponding to pixels included in the optimal seam.
- the tone correction map generator 420 may generate a tone correction map based on a global tone correction function and a local tone correction level.
- the local tone correction level may be determined by comparing the local information and a tone difference of the global tone correction function of a predetermined pixel.
- the tone correction map generator 420 may include a global calculation unit 421 , a smoothing unit 423 , a local calculation unit 425 , and a correction level calculation unit 427 .
- the global calculation unit 421 may calculate a global tone correction function, in which a tone difference between a pixel of the left image and a pixel of the right image, corresponding to pixels other than a ghost pixel in the overlapping region, is reflected, using the ghost information.
- Information about a tone difference in each pixel other than the ghost pixel in the overlapping region may be acquired by calculating the global tone correction function.
- the smoothing unit 423 may smooth a value of the global tone correction function calculated in the global calculation unit 421 , through a smoothing filter. Noise may be included in a value of the calculated global tone correction function during an arithmetic operation. The smoothing unit 423 may cancel the noise through the smoothing filter.
- the local calculation unit 425 may generate the local information of the optimal seam by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam. For example, the local calculation unit 425 may use a window having a size of N ⁇ N for each pixel included in the optimal seam, and calculate an average of tone differences between pixels other than the ghost pixel in the window having a size of N ⁇ N, thereby acquiring the local information of each pixel.
- the correction level calculation unit 427 may calculate a local tone correction level by comparing local information and a value of a global tone correction function, of a pixel included in the optimal seam.
- the global tone correction function may reflect information associated with a tone difference for an entire overlapping region of the left image and the right image.
- the local information may reflect information about a tone difference of a pixel included in the optimal seam so that a pixel included in the optimal seam may have a similar tone difference to a neighboring pixel.
- the correction level calculation unit 427 may calculate the local tone correction level by subtracting a value of the global tone correction function from the local information.
- the tone correction map generator 420 may generate a tone correction map by adjusting the global tone correction function by the local tone correction level.
- the bidirectional corrector 430 may correct the left image and the right image using a final tone correction map generated by applying, to the tone correction map, a weighting based on the left image and a weighting based on the right image.
- the tone correction map may be reflected in the left image and the right image.
- the tone correction map may indicate information of a tone difference for the overlapping region of the left image and the right image.
- a weighting according to a distance between the left image and the overlapping image may be applied to the tone correction map reflected in the left image so that a tone may be naturally formed from the left image.
- a weighting according to a distance between the right image and the overlapping image may be applied to the tone correction map reflected in the right image so that a tone may be naturally formed from the right image.
- the bidirectional corrector 430 may include a weighting, application unit 431 .
- the weighting application unit 431 may apply, to the tone correction map, a weighting varying according to a distance between the left image and the overlapping region.
- the weighting application unit 431 may apply, to the tone correction map, a weighting varying according to a distance between the right image and the overlapping region.
- the bidirectional corrector 430 may correct a tone of the left image and the right image, based on the final tone correction map.
- the image processing apparatus may generate a panoramic image by blending the left image and the right image that are corrected.
- the bidirectional corrector 430 may enable an overall panoramic image to maintain a uniform tone by correcting each tone of the left image and the right image, in comparison with a general process of correcting a tone of an object image according to a tone of a target image.
- FIG. 5 illustrates an operation of generating a ghost map, according to example embodiments.
- An image processing apparatus may receive an input of a left image I 1 and a right image I 2 , and generate a cost map CM 12 for an overlapping region.
- the cost map CM 12 may be generated by reflecting similarity between the left image I 1 and the right image I 2 in the overlapping region. For example, in the cost map CM 12 , a portion expressed in a relatively bright color may indicate a portion having relatively low similarity between the left image I 1 and the right image I 2 , and a portion expressed in a relatively dark color may indicate a portion having relatively high similarity between the left image I 1 and the right image I 2 .
- the image processing apparatus may generate a ghost map GM 12 by binarizing the CM 12 based on an optimal threshold. That is, the optimal threshold may be determined by an optimal threshold estimation process, and the optimal threshold estimation process may correspond to a technology widely used in an image processing field, and thus, further description will be omitted for conciseness.
- Ghost information may be included in the ghost map GM 12 .
- the ghost information may correspond to information unrelated to a tone correction of an image.
- the image processing apparatus may easily generate the ghost map GM 12 from the cost map CM 12 through binarization.
- a bright region 510 may correspond to a region associated with a tone correction in an overlapping region of the left image I 1 and the right image I 2
- a dark region 520 may correspond to a region unrelated to the tone correction in the overlapping region of the left image I 1 and the right image I 2 .
- FIG. 6 illustrates an operation of generating a global tone correction function, according to example embodiments.
- FIG. 6 illustrates a ghost map generated in an overlapping region of a left image and a right image when a panoramic image is generated in a horizontal direction using the left image and the right image.
- a tone difference in a vertical direction may vary by relatively less in the overlapping region.
- generating the global tone correction function for the tone difference in a horizontal direction may be sufficient for expressing a variation of the tone difference of the overlapping region.
- the image processing apparatus may generate the global tone correction function in a horizontal direction by calculating a cumulative mean of tone differences of pixels, other than a ghost pixel in a vertical direction, indicated by a direction of an arrow on the ghost map.
- the tone differences may indicate tone differences between pixels corresponding to the left image and the right image.
- the image processing apparatus may generate a global tone correction function f g (x) in a horizontal direction, using values of the cumulative mean.
- the image processing apparatus may cancel noise included in f′ g (x) by performing a smoothing filtering on f′ g (x).
- f g (x) may be generated as a result of the smoothing filtering performed on f′ g (x).
- f′ g (x) and f g (x) may be expressed by the following Equations.
- I 1 (x, y) denotes a tone of a pixel in the left image
- I 2 (x, y) denotes a tone of a pixel in the right image
- GM 12 (x, y) denotes whether a pixel of the ghost map corresponds to the ghost pixel.
- GM 12 (x, y) may be “0.”
- H denotes a sum of pixels other than the ghost pixel when the cumulative mean is calculated in a vertical direction on the ghost map.
- “h” denotes a number of pixels in a horizontal direction on the ghost map.
- n denotes a size of the smoothing filter.
- a graph 610 illustrates a result of calculating the global tone correction function in a horizontal direction based on information about the ghost map
- a graph 620 illustrates a result of calculating the smoothed global tone correction function.
- FIG. 7 illustrates an operation of generating local information of an optimal seam, according to example embodiments.
- a curved line drawn in a vertical direction on a cost map of FIG. 7 may indicate an optimal seam that occurs when a left image and a right image are joined.
- An image processing apparatus may acquire local information TD(x, y) by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel included in the optimal seam.
- the local information TD(x, y) may indicate information used to correct a tone of a pixel included in the optimal seam to be similar to a neighboring pixel.
- the image processing apparatus may acquire information about the ghost pixel from a ghost map.
- the local information TD(x, y) may be expressed by the following Equation.
- W denotes a number of pixels other than the ghost pixel in a window of a predetermined size
- I 1 (x+j, y+i) denotes a tone of a pixel of the left image in the window
- I 2 (x+j, y+i) denotes a tone of a pixel of the right image in the window
- GM 12 (x+j, y+i) denotes whether a pixel of the ghost map corresponds to the ghost pixel in the window.
- FIG. 8 illustrates a tone correction map, according to example embodiments.
- an image processing apparatus may generate a tone correction map TM(x, y) by compensating a global tone correction function f g (x) by a local tone correction level TC(y).
- the local tone correction level TC(y) may be calculated by subtracting the global tone correction function f g (x) from local information TD(x, y).
- the local tone correction level TC(y) may be expressed as shown below.
- the tone correction map TM(x, y) may be calculated by adding the local tone correction level TC(y) to the global tone correction function f g (x).
- the tone correction map TM(x, y) may be expressed as shown below.
- TM ( x, y ) f g ( x )+ TC ( y )
- a graph 810 illustrates a result obtained by calculating the global tone correction function f g (x), and a graph 820 illustrates a result obtained by calculating the tone correction map TM(x, y).
- FIG. 9 illustrates a bidirectional correction of an input left image and right image, according to example embodiments.
- a weighting may be applied so that a tone correction may be naturally performed from the left image 910 to the overlapping region 930 .
- a weighting may be applied so that a tone correction may be naturally performed from the right image 920 to the overlapping region 930 .
- An optimal seam 940 may be present in the overlapping region 930 .
- a weighting w 1 (x) may be determined based on a portion 931 connecting the left image 910 and the overlapping region 930 .
- the weighting w 1 (x) may be determined in proportion to a width of the overlapping region 930 in a direction 950 .
- the weighting w 1 (x) may be expressed as shown below.
- a weighting w 2 (x) may be determined based on a portion 933 connecting the right image 920 and the overlapping region 930 .
- the weighting w 2 (x) may be determined in proportion to a width of the overlapping region 930 in a direction 960 .
- the weighting w 2 (x) may be expressed as shown below.
- An image processing apparatus may generate a final tone correction map TM 1 (x, y) to be used for a tone correction of the left image 910 by applying the weighting w 1 (x) to the tone correction map TM(x, y).
- the image processing apparatus may generate a final tone correction map TM 2 (x, y) to be used for a tone correction of the right image 920 by applying the weighting w 2 (x) to the tone correction map TM(x, y).
- the final tone correction map TM 1 (x, y) and the final tone correction map TM 2 (x, y) may be expressed as shown below.
- TM 1 ( x, y ) TM ( x, y ) ⁇ w 1 ( x )
- TM 2 ( x, y ) TM ( x, y ) ⁇ w 2 ( x )
- the image processing apparatus may acquire a left image I′ 1 (x, y) having a tone finally corrected by subtracting the final tone correction map TM 1 (x, y) from the left image 910 or adding the final tone correction map TM 1 (x, y) to the left image 910 .
- the image processing apparatus may acquire a right image I′ 2 (x, y) having a tone finally corrected by subtracting the final tone correction map TM 2 (x, y) from the right image 920 or adding the final tone correction map TM 2 (x, y) to the right image 920 .
- the left image I′ 1 (x, y) having a tone finally corrected and the right image I′ 2 (x, y) having a tone finally corrected may be expressed as shown below.
- I′ 1 ( x, y ) I 1 ( x, y ) ⁇ TM 1 ( x, y )
- I′ 2 ( x, y ) I 2 ( x, y )+ TM 2 ( x, y )
- the image processing apparatus may generate a panoramic image by blending the left image I′ 1 (x, y) having a tone finally corrected and the right image I′ 2 (x, y) having a tone finally corrected.
- FIG. 10 illustrates an image processing method, according to example embodiments.
- an image processing apparatus may estimate an optimal seam of a left image and a right image using an optimal seam estimation process.
- the left image and the right image may correspond to input images.
- the optimal seam estimation process may include estimating an optimal seam by connecting similar pixels, based on a level of color similarity and gradient similarity between pixels corresponding to a left image and a right image in an overlapping region of the left image and the right image.
- the image processing apparatus may generate a ghost map by binarizing a cost map.
- a similarity between the left image and the right image may be reflected in the cost map.
- the image processing apparatus may binarize the cost map based on an optimal threshold.
- the image processing apparatus may binarize the cost map by comparing a similarity of each pixel on the cost map with the optimal threshold, thereby generating a ghost map.
- the image processing apparatus may generate local information of the optimal seam.
- the image processing apparatus may generate local information of pixels included in the optimal seam.
- the image processing apparatus may generate the local information by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam.
- Information about the ghost pixel may be acquired from the ghost map.
- the image processing apparatus may calculate a global tone correction function.
- the image processing apparatus may calculate a global tone correction function in a horizontal direction, based on information from the ghost map.
- a tone difference may be reflected in the global tone correction function based on pixels other than the ghost pixel in the overlapping region of the left image and the right image.
- the image processing apparatus may calculate a local tone correction level, based on the local information and the global tone correction function.
- the local tone correction level may be used for enhancing accuracy of a tone correction of a pixel included in the optimal seam on the overlapping region.
- the image processing apparatus may generate a tone correction map by compensating the global tone correction function by the local tone correction level.
- the image processing apparatus may apply, to the tone correction map, a weighting based on the left image and a weighting based on the right image.
- the image processing apparatus may generate a final tone correction map to be used for the left image and a final tone correction map to be used for the right image, based on a weighting applied to the tone correction map.
- the image processing apparatus may correct a tone of the left image and right image using the corresponding final tone correction map.
- FIG. 11 illustrates a panoramic image having a corrected tone according to example embodiments.
- An image 1110 corresponds to a panoramic image generated by estimating an optimal seam of a left image and a right image, and blending the left image and the right image.
- An image 1120 corresponds to a panoramic image generated by correcting a tone of a left image and right image based on a final tone correction map according to example embodiments, and blending the left image and the right image. Referring to FIG. 11 , the image 1120 may have a natural tone when compared to the image 1110 .
- the method of processing an image according to the above-described embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by a computer.
- the media may also include, alone or in combination with the program instructions, data files, data structures, and the like.
- Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM discs and DVDs; magneto-optical media such as optical discs; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like.
- Examples of the magnetic recording apparatus include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape (MT).
- Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
- the described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa.
- the image processing apparatus may include at least one processor to execute at least one of the above-described units and methods.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Stereoscopic And Panoramic Photography (AREA)
- Transforming Electric Information Into Light Information (AREA)
- Studio Devices (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
Description
- This application claims the priority benefit of Korean Patent Application No. 10-2011-0055985, filed on Jun. 10, 2011, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.
- 1. Field
- Example embodiments of the following description relate to an apparatus and method for correcting a tone of a panoramic image, and more specifically, an apparatus and method for performing tone correction before blending a plurality of images to generate a panoramic image.
- 2. Description of the Related Art
- A panoramic image may be generated by blending a plurality of images acquired by photographing a subject at several positions or from several angles. When the plurality of images are blended prior to a tone correction between the plurality of images, an artifact may occur at a seam connecting the plurality of images, which may cause a user to sense a difference because each of the plurality of images may have been acquired under differing conditions, such as, different exposure values, different gradations, and different lens shadings. Thus, a tone correction may need to be performed, among the plurality of images, before blending the plurality of images.
- The example embodiments may include an apparatus for processing an image, the apparatus including a ghost information acquisition unit to acquire, from an overlapping region of a left image and a right image, ghost information based on an optimal threshold and a cost map; a tone correction map generator to generate a tone correction map, based on a global tone correction function generated using the ghost information, and based on a local tone correction level generated using local information of an optimal seam; and a bidirectional corrector to correct the left image and the right image using a final tone correction map generated by applying, to the tone correction map, a weighting based on the left image and a weighting based on the right image.
- The cost map may be generated by overlapping a region of the left image and the right image, such that a similarity between the left image and the right image is reflected on the cost map.
- The left image and the right image may correspond to images acquired by photographing a stationary subject or a moving subject at a plurality of photographing positions to generate a panoramic image.
- The similarity between the left image and the right image may be based on a level of color similarity and gradient similarity between the corresponding pixels in the overlapping region of the left image and the right image.
- The ghost information acquisition unit may include an optimal seam assumption unit to assume a path having a greatest similarity on the cost map to be an optimal seam of the left image and the right image; and a binarization unit to generate a ghost map by binarizing the cost map, based on the optimal threshold that is estimated using an optimal threshold estimation process.
- The tone correction map generator may include a global calculation unit to calculate a global tone correction function, in which a tone difference, between a pixel of the left image and a pixel of the right image, corresponding to pixels other than a ghost pixel in the overlapping region, is reflected using the ghost information.
- The tone correction map generator may further include a smoothing unit to smooth a value of the calculated global tone correction function through a smoothing filter.
- The tone correction map generator may include a local calculation unit to generate the local information of the optimal seam by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam.
- The tone correction map generator may further include a correction level calculation unit to calculate a local tone correction level by comparing local information and a value of a global tone correction function, of a pixel included in the optimal seam.
- The bidirectional corrector may include a weighting application unit to apply, to the tone correction map, a weighting varying according to a distance between the left image and the overlapping region, and a weighting varying according to a distance between the right image and the overlapping region.
- The example embodiments may also include a method of processing an image, the method including acquiring, from an overlapping region of a left image and a right image, ghost information, based on an optimal threshold and a cost map; generating a tone correction map, based on a global tone correction function generated using the ghost information, and based on a local tone correction level generated using local information of an optimal seam; and correcting the left image and the right image using a final tone correction map generated by applying, to the tone correction map, a weighting based on the left image and a weighting, based on the right image.
- The generating of the cost map may include overlapping a region of the left image and the right image, such that a similarity between the left image and the right image is reflected on the cost map.
- The acquiring may include assuming a path having a greatest similarity on the cost map to be an optimal seam of the left image and the right image, and generating a ghost map by binarizing the cost map based on the optimal threshold that is estimated using an optimal threshold estimation process.
- The generating may include calculating a global tone correction function in which a tone difference between a pixel of the left image and a pixel of the right image, corresponding to pixels other than a ghost pixel in the overlapping region is reflected, using the ghost information.
- The generating may include generating the local information of the optimal seam by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam.
- The generating of the tone correction map may further include calculating a local tone correction level by comparing local information and a value of a global tone correction function, of a pixel included in the optimal seam.
- The correcting may include applying, to the tone correction map, a weighting varying according to a distance between the left image and the overlapping region and a weighting varying according to a distance between the right image and the overlapping region.
- According to example embodiments, it is possible to reflect information about mismatched pixels corresponding to a plurality of images, and calculate a relatively accurate tone correction value by considering a ghost phenomenon.
- According to example embodiments, it is possible to maintain a tone of an intermediate level between a left image and a right image by correcting tones of both the left image and the right image that are inputted.
- According to example embodiments, it is possible to reduce an afterimage occurring due to blending since a range of the blending may not be extended to correct a tone.
- According to example embodiments, it is possible to acquire a calculated value at a relatively high speed while calculating a relatively accurate tone correction value by acquiring a ghost map from a cost map based on both of a global tone correction and a local tone correction.
- Additional aspects of embodiments will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosure.
- These and/or other aspects will become apparent and more readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings of which:
-
FIG. 1 , parts (a) and (b), illustrate a panoramic image, according to a conventional art; -
FIG. 2 illustrates an issue occurring due to a moving subject, according to a conventional art; -
FIG. 3 illustrates pixels on the same position in an overlapping region of input images, according to a conventional art; -
FIG. 4 illustrates an apparatus for processing an image, according to example embodiments; -
FIG. 5 illustrates an operation of generating a ghost map, according to example embodiments; -
FIG. 6 illustrates an operation of generating a global tone correction function, according to example embodiments; -
FIG. 7 illustrates an operation of generating local information of an optimal seam, according to example embodiments; -
FIG. 8 illustrates a tone correction map, according to example embodiments; -
FIG. 9 illustrates a bidirectional correction of an input left image and an input right image, according to example embodiments; -
FIG. 10 illustrates an image processing method, according to example embodiments; and -
FIG. 11 illustrates a panoramic image having a corrected tone, according to example embodiments. - Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Embodiments are described below to explain the present disclosure by referring to the figures.
-
FIGS. 1A and 1B illustrate an example of a panoramic image, according to a conventional art. -
FIG. 1A illustrates a panoramic image resulting from matching two images without a tone correction.FIG. 1B illustrates a panoramic image resulting from estimating an optimal seam and matching two images without a tone correction. - An overlapping region of two images acquired by photographing the same subject may have different exposure values or different gradation levels for each image, even though both images correspond to the same subject. Thus, when each image is blended without a tone correction, an unnatural line may occur in an intermediate portion of an image, as illustrated in
FIGS. 1A and 1B . -
FIG. 2 illustrates an issue occurring due to a moving subject, according to a conventional art. - A panoramic image may be generated by blending a target image and an object image. In this instance, a portion inside of a box in the target image and a portion inside of a box in the object image correspond to an overlapping portion. According to a general tone correction process, a tone in the portion inside of the box in the object image may be corrected by referencing the portion inside of the box in the target image.
- Referring to
FIG. 2 , a moving subject not visible from the portion inside of the box in the target image may be present in the portion inside of the box in the object image. Thus, the portion inside of the box in the object image and the portion inside of the box in the target image may contain different content even though the corresponding images are acquired by photographing a subject at the same location. Thus, a tone correction may not be performed since the moving subject included in the portion inside of the box in the object image is not visible from the target image. A tone correction may be performed by referencing the target image, and excluding a portion corresponding to the moving subject. - A general tone correction process may include generating a look-up table of the object image, related to the target image, through histogram matching, and correcting a tone of the object image, based on the look-up table. A general tone correction process may include correcting a region of the object image, in an overlapping region of the object image, and the target image through a histogram used for performing matching, estimating a color conversion matrix using a corrected region of the object image, and correcting a tone of the object image using the estimated color conversion matrix. A general tone correction process may include correcting a tone of the object image by calculating a weighting that uses a tone difference between the object image and the target image in a portion of a seam, and a distance from a pixel to be corrected.
-
FIG. 3 illustrates pixels on the same position in an overlapping region of input images, according to a conventional art. - Similar to a case in which different content is present at the same position due to an occurrence of a moving object, an inaccurate tone correction may be performed due to a matching error of one to two pixels occurring in a boundary of an object, even though an image matching may be performed successfully in an overlapping region of an
input image 310 and aninput image 320 ofFIG. 3 . - A general tone correction process may have an issue, such as, an image becoming darker or brighter as a panorama may be added since a tone of an object image is corrected for a single target image only.
-
FIG. 4 illustrates an apparatus for processing an image, according to example embodiments. - Referring to
FIG. 4 , the apparatus for processing an image (which may be referred to as an image processing apparatus, herein) may include a ghostinformation acquisition unit 410, a tonecorrection map generator 420, and abidirectional corrector 430. - The ghost
information acquisition unit 410 may generate a cost map on an overlapping region of a left image and a right image. In this case, the left image and the right image may correspond to input images. A similarity between the left image and the right image may be reflected on the cost map. To generate a panoramic image, an optimal seam estimation process may be used among processes for naturally joining a left image and a right image. The optimal seam estimation process may include estimating an optimal seam by connecting similar pixels, based on a level of color similarity and gradient similarity between pixels, corresponding to the left image and the right image in an overlapping region of the left image and the right image. The ghostinformation acquisition unit 410 may acquire ghost information based on an optimal threshold. The ghost information may indicate information about a region having a relatively small value of similarity between the left image and the right image. The similarity may be determined, based on various elements indicating a character of an image such as a color, saturation, brightness, and a tone of each image. - The left image and the right image may correspond to images acquired by photographing a stationary subject or a moving subject at a plurality of photographing positions. A panoramic image may be generated by putting the left image and the right image together.
- The ghost
information acquisition unit 410 may include an optimalseam assumption unit 411 and abinarization unit 413. The optimalseam assumption unit 411 may assume a path having a greatest similarity on the cost map to be an optimal seam of the left image and the right image. Thebinarization unit 413 may binarize the cost map, based on the optimal threshold that may be estimated, using an optimal threshold estimation process. A binarization of the cost map may indicate a conversion of the cost map into grayscale. Thebinarization unit 413 may binarize the cost map by comparing similarity of each pixel on the cost map with the optimal threshold, thereby generating a ghost map. For example, thebinarization unit 413 may binarize the cost map to “1” when the similarity of each pixel on the cost map may be greater than the optimal threshold, and may binarize the cost map to “0” when the similarity of each pixel on the cost map may be less than the optimal threshold. In this instance, a portion binarized to “1” on the ghost map may be expressed in a relatively bright color, and a portion binarized to “0” on the ghost map may be expressed in a relatively dark color. - The
binarization unit 413 may generate the ghost map through binarization, and thus, a complicated arithmetic operation may not be used to rapidly generate the ghost map. - The ghost map may include information about a pixel associated with a tone correction and ghost information unrelated to the tone correction. For example, a portion binarized to “1” may indicate pixels associated with the tone correction, and a portion binarized to “0” may indicate a ghost portion unrelated to the tone correction.
- The tone
correction map generator 420 may generate a global tone correction function based on the ghost information in the overlapping region of the left image and the right image. The global tone correction function may reflect all pixels other than a ghost pixel in the overlapping region. For example, when a panoramic image is generated in a horizontal direction using a plurality of images photographed in horizontal directions, a tone difference in a vertical direction may not be an issue while a tone difference in a horizontal direction may be an issue. The tone difference in a vertical direction may be trivial. Thus, when a panoramic image is generated in a horizontal direction, the tonecorrection map generator 420 may generate the global tone correction function by calculating a cumulative mean of tone differences, of portions other than a ghost pixel, in a vertical direction on the overlapping region of the left image and the right image. - The global tone correction function may reflect a tone difference between the left image and the right image in an entire overlapping region.
- The tone
correction map generator 420 may generate local information of the optimal seam. The local information may be generated by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel included in the optimal seam. The local information may indicate information about a tone difference between pixels included in the optimal seam. In this case, the tone difference may refer to a tone difference between a pixel of the left image and a pixel of the right image, corresponding to pixels included in the optimal seam. - The tone
correction map generator 420 may generate a tone correction map based on a global tone correction function and a local tone correction level. The local tone correction level may be determined by comparing the local information and a tone difference of the global tone correction function of a predetermined pixel. - The tone
correction map generator 420 may include aglobal calculation unit 421, a smoothingunit 423, alocal calculation unit 425, and a correctionlevel calculation unit 427. - The
global calculation unit 421 may calculate a global tone correction function, in which a tone difference between a pixel of the left image and a pixel of the right image, corresponding to pixels other than a ghost pixel in the overlapping region, is reflected, using the ghost information. Information about a tone difference in each pixel other than the ghost pixel in the overlapping region may be acquired by calculating the global tone correction function. - The smoothing
unit 423 may smooth a value of the global tone correction function calculated in theglobal calculation unit 421, through a smoothing filter. Noise may be included in a value of the calculated global tone correction function during an arithmetic operation. The smoothingunit 423 may cancel the noise through the smoothing filter. - The
local calculation unit 425 may generate the local information of the optimal seam by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam. For example, thelocal calculation unit 425 may use a window having a size of N×N for each pixel included in the optimal seam, and calculate an average of tone differences between pixels other than the ghost pixel in the window having a size of N×N, thereby acquiring the local information of each pixel. - The correction
level calculation unit 427 may calculate a local tone correction level by comparing local information and a value of a global tone correction function, of a pixel included in the optimal seam. The global tone correction function may reflect information associated with a tone difference for an entire overlapping region of the left image and the right image. The local information may reflect information about a tone difference of a pixel included in the optimal seam so that a pixel included in the optimal seam may have a similar tone difference to a neighboring pixel. For example, the correctionlevel calculation unit 427 may calculate the local tone correction level by subtracting a value of the global tone correction function from the local information. - The tone
correction map generator 420 may generate a tone correction map by adjusting the global tone correction function by the local tone correction level. - The
bidirectional corrector 430 may correct the left image and the right image using a final tone correction map generated by applying, to the tone correction map, a weighting based on the left image and a weighting based on the right image. The tone correction map may be reflected in the left image and the right image. The tone correction map may indicate information of a tone difference for the overlapping region of the left image and the right image. In this instance, a weighting according to a distance between the left image and the overlapping image may be applied to the tone correction map reflected in the left image so that a tone may be naturally formed from the left image. A weighting according to a distance between the right image and the overlapping image may be applied to the tone correction map reflected in the right image so that a tone may be naturally formed from the right image. - The
bidirectional corrector 430 may include a weighting,application unit 431. Theweighting application unit 431 may apply, to the tone correction map, a weighting varying according to a distance between the left image and the overlapping region. Theweighting application unit 431 may apply, to the tone correction map, a weighting varying according to a distance between the right image and the overlapping region. - The
bidirectional corrector 430 may correct a tone of the left image and the right image, based on the final tone correction map. The image processing apparatus may generate a panoramic image by blending the left image and the right image that are corrected. - The
bidirectional corrector 430 may enable an overall panoramic image to maintain a uniform tone by correcting each tone of the left image and the right image, in comparison with a general process of correcting a tone of an object image according to a tone of a target image. -
FIG. 5 illustrates an operation of generating a ghost map, according to example embodiments. - An image processing apparatus may receive an input of a left image I1 and a right image I2, and generate a cost map CM12 for an overlapping region. The cost map CM12 may be generated by reflecting similarity between the left image I1 and the right image I2 in the overlapping region. For example, in the cost map CM12, a portion expressed in a relatively bright color may indicate a portion having relatively low similarity between the left image I1 and the right image I2, and a portion expressed in a relatively dark color may indicate a portion having relatively high similarity between the left image I1 and the right image I2.
- The image processing apparatus may generate a ghost map GM12 by binarizing the CM12 based on an optimal threshold. That is, the optimal threshold may be determined by an optimal threshold estimation process, and the optimal threshold estimation process may correspond to a technology widely used in an image processing field, and thus, further description will be omitted for conciseness.
- Ghost information may be included in the ghost map GM12. In this case, the ghost information may correspond to information unrelated to a tone correction of an image. The image processing apparatus may easily generate the ghost map GM12 from the cost map CM12 through binarization. In the ghost map GM12, a
bright region 510 may correspond to a region associated with a tone correction in an overlapping region of the left image I1 and the right image I2, and adark region 520 may correspond to a region unrelated to the tone correction in the overlapping region of the left image I1 and the right image I2. -
FIG. 6 illustrates an operation of generating a global tone correction function, according to example embodiments. -
FIG. 6 illustrates a ghost map generated in an overlapping region of a left image and a right image when a panoramic image is generated in a horizontal direction using the left image and the right image. When the panoramic image is generated in a horizontal direction, a tone difference in a vertical direction may vary by relatively less in the overlapping region. Thus, generating the global tone correction function for the tone difference in a horizontal direction may be sufficient for expressing a variation of the tone difference of the overlapping region. - The image processing apparatus may generate the global tone correction function in a horizontal direction by calculating a cumulative mean of tone differences of pixels, other than a ghost pixel in a vertical direction, indicated by a direction of an arrow on the ghost map. In this instance, the tone differences may indicate tone differences between pixels corresponding to the left image and the right image.
- The image processing apparatus may generate a global tone correction function fg(x) in a horizontal direction, using values of the cumulative mean. The image processing apparatus may cancel noise included in f′g(x) by performing a smoothing filtering on f′g(x). fg(x) may be generated as a result of the smoothing filtering performed on f′g(x). f′g(x) and fg(x) may be expressed by the following Equations.
-
- In this case, I1(x, y) denotes a tone of a pixel in the left image, I2(x, y) denotes a tone of a pixel in the right image, GM12(x, y) denotes whether a pixel of the ghost map corresponds to the ghost pixel. When the pixel of the ghost map corresponds to a ghost pixel, GM12(x, y) may be “0.” H denotes a sum of pixels other than the ghost pixel when the cumulative mean is calculated in a vertical direction on the ghost map. “h” denotes a number of pixels in a horizontal direction on the ghost map. “n” denotes a size of the smoothing filter.
- Referring to
FIG. 6 , agraph 610 illustrates a result of calculating the global tone correction function in a horizontal direction based on information about the ghost map, and agraph 620 illustrates a result of calculating the smoothed global tone correction function. -
FIG. 7 illustrates an operation of generating local information of an optimal seam, according to example embodiments. - A curved line drawn in a vertical direction on a cost map of
FIG. 7 may indicate an optimal seam that occurs when a left image and a right image are joined. An image processing apparatus may acquire local information TD(x, y) by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel included in the optimal seam. The local information TD(x, y) may indicate information used to correct a tone of a pixel included in the optimal seam to be similar to a neighboring pixel. The image processing apparatus may acquire information about the ghost pixel from a ghost map. The local information TD(x, y) may be expressed by the following Equation. -
- In this case, W denotes a number of pixels other than the ghost pixel in a window of a predetermined size, I1(x+j, y+i) denotes a tone of a pixel of the left image in the window, I2(x+j, y+i) denotes a tone of a pixel of the right image in the window, and GM12(x+j, y+i) denotes whether a pixel of the ghost map corresponds to the ghost pixel in the window.
-
FIG. 8 illustrates a tone correction map, according to example embodiments. - Referring to
FIG. 8 , an image processing apparatus may generate a tone correction map TM(x, y) by compensating a global tone correction function fg(x) by a local tone correction level TC(y). In this instance, the local tone correction level TC(y) may be calculated by subtracting the global tone correction function fg(x) from local information TD(x, y). The local tone correction level TC(y) may be expressed as shown below. -
TC(y)=TD(x, y)−f g(x) - The tone correction map TM(x, y) may be calculated by adding the local tone correction level TC(y) to the global tone correction function fg(x). The tone correction map TM(x, y) may be expressed as shown below.
-
TM(x, y)=f g(x)+TC(y) - A
graph 810 illustrates a result obtained by calculating the global tone correction function fg(x), and agraph 820 illustrates a result obtained by calculating the tone correction map TM(x, y). -
FIG. 9 illustrates a bidirectional correction of an input left image and right image, according to example embodiments. - Referring to
FIG. 9 , when a tone correction map TM(x, y) is determined on anoverlapping region 930 of aleft image 910 and aright image 920, a weighting may be applied so that a tone correction may be naturally performed from theleft image 910 to theoverlapping region 930. A weighting may be applied so that a tone correction may be naturally performed from theright image 920 to theoverlapping region 930. Anoptimal seam 940 may be present in theoverlapping region 930. - A weighting w1(x) may be determined based on a
portion 931 connecting theleft image 910 and theoverlapping region 930. The weighting w1(x) may be determined in proportion to a width of theoverlapping region 930 in adirection 950. The weighting w1(x) may be expressed as shown below. -
- A weighting w2(x) may be determined based on a
portion 933 connecting theright image 920 and theoverlapping region 930. The weighting w2(x) may be determined in proportion to a width of theoverlapping region 930 in adirection 960. The weighting w2(x) may be expressed as shown below. -
- An image processing apparatus may generate a final tone correction map TM1(x, y) to be used for a tone correction of the
left image 910 by applying the weighting w1(x) to the tone correction map TM(x, y). The image processing apparatus may generate a final tone correction map TM2(x, y) to be used for a tone correction of theright image 920 by applying the weighting w2(x) to the tone correction map TM(x, y). The final tone correction map TM1(x, y) and the final tone correction map TM2(x, y) may be expressed as shown below. -
TM 1(x, y)=TM(x, y)×w 1(x) -
TM 2(x, y)=TM(x, y)×w 2(x) - The image processing apparatus may acquire a left image I′1(x, y) having a tone finally corrected by subtracting the final tone correction map TM1(x, y) from the
left image 910 or adding the final tone correction map TM1(x, y) to theleft image 910. The image processing apparatus may acquire a right image I′2(x, y) having a tone finally corrected by subtracting the final tone correction map TM2(x, y) from theright image 920 or adding the final tone correction map TM2(x, y) to theright image 920. For example, the left image I′1(x, y) having a tone finally corrected and the right image I′2(x, y) having a tone finally corrected may be expressed as shown below. -
I′ 1(x, y)=I 1(x, y)−TM 1(x, y) -
I′ 2(x, y)=I 2(x, y)+TM 2(x, y) - The image processing apparatus may generate a panoramic image by blending the left image I′1(x, y) having a tone finally corrected and the right image I′2(x, y) having a tone finally corrected.
-
FIG. 10 illustrates an image processing method, according to example embodiments. - In
operation 1010, an image processing apparatus may estimate an optimal seam of a left image and a right image using an optimal seam estimation process. In this case, the left image and the right image may correspond to input images. The optimal seam estimation process may include estimating an optimal seam by connecting similar pixels, based on a level of color similarity and gradient similarity between pixels corresponding to a left image and a right image in an overlapping region of the left image and the right image. - In
operation 1020, the image processing apparatus may generate a ghost map by binarizing a cost map. A similarity between the left image and the right image may be reflected in the cost map. The image processing apparatus may binarize the cost map based on an optimal threshold. The image processing apparatus may binarize the cost map by comparing a similarity of each pixel on the cost map with the optimal threshold, thereby generating a ghost map. - In
operation 1030, the image processing apparatus may generate local information of the optimal seam. In particular, the image processing apparatus may generate local information of pixels included in the optimal seam. In this instance, the image processing apparatus may generate the local information by calculating an average of tone differences between pixels other than a ghost pixel in a window of a predetermined size around a pixel of the optimal seam. Information about the ghost pixel may be acquired from the ghost map. - In
operation 1040, the image processing apparatus may calculate a global tone correction function. The image processing apparatus may calculate a global tone correction function in a horizontal direction, based on information from the ghost map. A tone difference may be reflected in the global tone correction function based on pixels other than the ghost pixel in the overlapping region of the left image and the right image. - In
operation 1050, the image processing apparatus may calculate a local tone correction level, based on the local information and the global tone correction function. The local tone correction level may be used for enhancing accuracy of a tone correction of a pixel included in the optimal seam on the overlapping region. - In
operation 1060, the image processing apparatus may generate a tone correction map by compensating the global tone correction function by the local tone correction level. - In
operation 1070, the image processing apparatus may apply, to the tone correction map, a weighting based on the left image and a weighting based on the right image. - In
operation 1080, the image processing apparatus may generate a final tone correction map to be used for the left image and a final tone correction map to be used for the right image, based on a weighting applied to the tone correction map. - In
operation 1090, the image processing apparatus may correct a tone of the left image and right image using the corresponding final tone correction map. -
FIG. 11 illustrates a panoramic image having a corrected tone according to example embodiments. - An
image 1110 corresponds to a panoramic image generated by estimating an optimal seam of a left image and a right image, and blending the left image and the right image. Animage 1120 corresponds to a panoramic image generated by correcting a tone of a left image and right image based on a final tone correction map according to example embodiments, and blending the left image and the right image. Referring toFIG. 11 , theimage 1120 may have a natural tone when compared to theimage 1110. - The method of processing an image according to the above-described embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM discs and DVDs; magneto-optical media such as optical discs; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of the magnetic recording apparatus include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape (MT). Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa.
- Further, according to an aspect of the embodiments, any combinations of the described features, functions and/or operations can be provided.
- Moreover, the image processing apparatus, as shown in
FIG. 4 , for example, may include at least one processor to execute at least one of the above-described units and methods. - Although embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.
Claims (17)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020110055985A KR101742120B1 (en) | 2011-06-10 | 2011-06-10 | Apparatus and method for image processing |
KR10-2011-0055985 | 2011-06-10 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20120314945A1 true US20120314945A1 (en) | 2012-12-13 |
US9105107B2 US9105107B2 (en) | 2015-08-11 |
Family
ID=45977178
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/398,444 Active 2033-08-30 US9105107B2 (en) | 2011-06-10 | 2012-02-16 | Apparatus and method for image processing |
Country Status (5)
Country | Link |
---|---|
US (1) | US9105107B2 (en) |
EP (1) | EP2533193B1 (en) |
JP (1) | JP5960513B2 (en) |
KR (1) | KR101742120B1 (en) |
CN (1) | CN102819824B (en) |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130329002A1 (en) * | 2012-06-06 | 2013-12-12 | Apple Inc. | Adaptive Image Blending Operations |
US8902335B2 (en) | 2012-06-06 | 2014-12-02 | Apple Inc. | Image blending operations |
US8957944B2 (en) | 2011-05-17 | 2015-02-17 | Apple Inc. | Positional sensor-assisted motion filtering for panoramic photography |
US20150172620A1 (en) * | 2013-12-16 | 2015-06-18 | National Chiao Tung University | Optimal dynamic seam adjustment system and method for image stitching |
US20150164330A1 (en) * | 2013-12-16 | 2015-06-18 | N22 E.U. | Ultrasound imaging system and method for generateing a blended cine loop |
US9088714B2 (en) | 2011-05-17 | 2015-07-21 | Apple Inc. | Intelligent image blending for panoramic photography |
US20150326753A1 (en) * | 2013-01-22 | 2015-11-12 | Tencent Technology (Shenzhen) Company Limited | Image tone adjustment method, apparatus thereof and computer storage medium |
US9247133B2 (en) | 2011-06-01 | 2016-01-26 | Apple Inc. | Image registration using sliding registration windows |
US20160227188A1 (en) * | 2012-03-21 | 2016-08-04 | Ricoh Company, Ltd. | Calibrating range-finding system using parallax from two different viewpoints and vehicle mounting the range-finding system |
US9589350B1 (en) * | 2013-05-30 | 2017-03-07 | 360 Lab Llc. | Utilizing three overlapping images for exposure correction during panoramic image stitching |
US20170243384A1 (en) * | 2016-02-19 | 2017-08-24 | Mediatek Inc. | Image data processing system and associated methods for processing panorama images and image blending using the same |
US9762794B2 (en) | 2011-05-17 | 2017-09-12 | Apple Inc. | Positional sensor-assisted perspective correction for panoramic photography |
US9832378B2 (en) | 2013-06-06 | 2017-11-28 | Apple Inc. | Exposure mapping and dynamic thresholding for blending of multiple images using floating exposure |
US20180005410A1 (en) * | 2016-06-29 | 2018-01-04 | Xiaoyi Technology Co., Ltd. | System and method for adjusting brightness in multiple images |
US10306140B2 (en) | 2012-06-06 | 2019-05-28 | Apple Inc. | Motion adaptive image slice selection |
US20210215783A1 (en) * | 2017-08-08 | 2021-07-15 | Shanghai United Imaging Healthcare Co., Ltd. | Method, device and mri system for correcting phase shifts |
US11234042B2 (en) | 2017-11-14 | 2022-01-25 | Samsung Electronics Co., Ltd. | Display device, control method therefor and recording medium |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6467787B2 (en) | 2014-05-27 | 2019-02-13 | 株式会社リコー | Image processing system, imaging apparatus, image processing method, and program |
WO2016199608A1 (en) * | 2015-06-12 | 2016-12-15 | ソニー株式会社 | Information processing device and information processing method |
CN106940877B (en) * | 2016-01-05 | 2021-04-20 | 富士通株式会社 | Image processing apparatus and method |
CN106254844B (en) * | 2016-08-25 | 2018-05-22 | 成都易瞳科技有限公司 | A kind of panoramic mosaic color calibration method |
CN107920252B (en) * | 2016-10-11 | 2021-11-12 | 阿里巴巴集团控股有限公司 | Panoramic video data processing method, device and system |
CN107016647B (en) * | 2017-04-26 | 2019-11-22 | 武汉大学 | Panoramic picture color tone consistency correcting method and system |
KR102302245B1 (en) * | 2019-11-07 | 2021-09-14 | 부산대학교 산학협력단 | Apparatus and Method for Eliminating Panoramic Dual and Ghost Image using Deep Learning |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6157747A (en) * | 1997-08-01 | 2000-12-05 | Microsoft Corporation | 3-dimensional image rotation method and apparatus for producing image mosaics |
US20100271512A1 (en) * | 2009-04-23 | 2010-10-28 | Haim Garten | Multiple exposure high dynamic range image capture |
US20110310970A1 (en) * | 2010-06-18 | 2011-12-22 | Snu R&Db Foundation | Method, apparatus and computer-readable medium processing frames obtained by multiple exposures |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH10155104A (en) * | 1996-11-22 | 1998-06-09 | Canon Inc | Compound eye image pickup method and device and storage medium |
US6618511B1 (en) | 1999-12-31 | 2003-09-09 | Stmicroelectronics, Inc. | Perspective correction for panoramic digital camera with remote processing |
US7409105B2 (en) | 2003-10-22 | 2008-08-05 | Arcsoft, Inc. | Panoramic maker engine for a low profile system |
JP2005175620A (en) * | 2003-12-08 | 2005-06-30 | Canon Inc | Image processing method and apparatus |
KR100724134B1 (en) | 2006-01-09 | 2007-06-04 | 삼성전자주식회사 | Method and apparatus for providing panoramic view with high speed image matching and mild mixed color blending |
JP4676385B2 (en) | 2006-05-31 | 2011-04-27 | 株式会社日立国際電気 | Image composition method, image composition apparatus, and image composition program |
JP2008003683A (en) | 2006-06-20 | 2008-01-10 | Matsushita Electric Ind Co Ltd | Image generation device and its method and recording medium |
US7697839B2 (en) | 2006-06-30 | 2010-04-13 | Microsoft Corporation | Parametric calibration for panoramic camera systems |
KR100790887B1 (en) * | 2006-09-22 | 2008-01-02 | 삼성전자주식회사 | Apparatus and method for processing image |
US7839422B2 (en) * | 2006-12-13 | 2010-11-23 | Adobe Systems Incorporated | Gradient-domain compositing |
JP2010108238A (en) | 2008-10-30 | 2010-05-13 | Seiko Epson Corp | Image processor, image processing method, and program |
JP2010147634A (en) | 2008-12-17 | 2010-07-01 | Sanyo Electric Co Ltd | Image synthesizer |
US8339475B2 (en) * | 2008-12-19 | 2012-12-25 | Qualcomm Incorporated | High dynamic range image combining |
US20100194851A1 (en) * | 2009-02-03 | 2010-08-05 | Aricent Inc. | Panorama image stitching |
US8698875B2 (en) | 2009-02-20 | 2014-04-15 | Google Inc. | Estimation of panoramic camera orientation relative to a vehicle coordinate frame |
JP2010220002A (en) | 2009-03-18 | 2010-09-30 | Sony Corp | Image pickup device and image pickup method |
JP2010283502A (en) | 2009-06-03 | 2010-12-16 | Topseed Technology Corp | Intellectual monitoring system and intellectual monitoring method |
JP5418127B2 (en) | 2009-10-09 | 2014-02-19 | ソニー株式会社 | Image processing apparatus and method, and program |
JP5359783B2 (en) * | 2009-10-28 | 2013-12-04 | ソニー株式会社 | Image processing apparatus and method, and program |
KR20110052124A (en) * | 2009-11-12 | 2011-05-18 | 삼성전자주식회사 | Method for generating and referencing panorama image and mobile terminal using the same |
-
2011
- 2011-06-10 KR KR1020110055985A patent/KR101742120B1/en active IP Right Grant
-
2012
- 2012-02-16 US US13/398,444 patent/US9105107B2/en active Active
- 2012-03-12 CN CN201210064144.4A patent/CN102819824B/en active Active
- 2012-03-19 EP EP12160024.1A patent/EP2533193B1/en not_active Not-in-force
- 2012-06-08 JP JP2012131000A patent/JP5960513B2/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6157747A (en) * | 1997-08-01 | 2000-12-05 | Microsoft Corporation | 3-dimensional image rotation method and apparatus for producing image mosaics |
US20100271512A1 (en) * | 2009-04-23 | 2010-10-28 | Haim Garten | Multiple exposure high dynamic range image capture |
US20110310970A1 (en) * | 2010-06-18 | 2011-12-22 | Snu R&Db Foundation | Method, apparatus and computer-readable medium processing frames obtained by multiple exposures |
Non-Patent Citations (4)
Title |
---|
Heo, Y.S. - "Ghost-Free High Dynamic Range Imaging" - ACCV Computer Vision conference, Nov. 2010, pages 486-500 * |
Kang, S. - "High Dynamic Range Video" - ACM 2003, pages 319-325 * |
Meylan, L. - "Tone Mapping for High Dynamic Range Images" - EPFL, 2006, pages 125-127 and 135-140 * |
Uyttendaele, M. - "Eliminating Ghosting and Exposure Artifacts in Image Mosaics" 2001 IEEE, pages 509-516 * |
Cited By (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8957944B2 (en) | 2011-05-17 | 2015-02-17 | Apple Inc. | Positional sensor-assisted motion filtering for panoramic photography |
US9762794B2 (en) | 2011-05-17 | 2017-09-12 | Apple Inc. | Positional sensor-assisted perspective correction for panoramic photography |
US9088714B2 (en) | 2011-05-17 | 2015-07-21 | Apple Inc. | Intelligent image blending for panoramic photography |
US9247133B2 (en) | 2011-06-01 | 2016-01-26 | Apple Inc. | Image registration using sliding registration windows |
US20160227188A1 (en) * | 2012-03-21 | 2016-08-04 | Ricoh Company, Ltd. | Calibrating range-finding system using parallax from two different viewpoints and vehicle mounting the range-finding system |
US20130329002A1 (en) * | 2012-06-06 | 2013-12-12 | Apple Inc. | Adaptive Image Blending Operations |
US8902335B2 (en) | 2012-06-06 | 2014-12-02 | Apple Inc. | Image blending operations |
US10306140B2 (en) | 2012-06-06 | 2019-05-28 | Apple Inc. | Motion adaptive image slice selection |
US9098922B2 (en) * | 2012-06-06 | 2015-08-04 | Apple Inc. | Adaptive image blending operations |
US20150326753A1 (en) * | 2013-01-22 | 2015-11-12 | Tencent Technology (Shenzhen) Company Limited | Image tone adjustment method, apparatus thereof and computer storage medium |
US9736336B2 (en) * | 2013-01-22 | 2017-08-15 | Tencent Technology (Shenzhen) Company Limited | Image tone adjustment method, apparatus thereof and computer storage medium |
US9589350B1 (en) * | 2013-05-30 | 2017-03-07 | 360 Lab Llc. | Utilizing three overlapping images for exposure correction during panoramic image stitching |
US9832378B2 (en) | 2013-06-06 | 2017-11-28 | Apple Inc. | Exposure mapping and dynamic thresholding for blending of multiple images using floating exposure |
US9480403B2 (en) * | 2013-12-16 | 2016-11-01 | General Electric Company | Medical imaging system and method for generating a blended cine loop |
US9251612B2 (en) * | 2013-12-16 | 2016-02-02 | National Chiao Tung University | Optimal dynamic seam adjustment system and method for image stitching |
US20150164330A1 (en) * | 2013-12-16 | 2015-06-18 | N22 E.U. | Ultrasound imaging system and method for generateing a blended cine loop |
US20150172620A1 (en) * | 2013-12-16 | 2015-06-18 | National Chiao Tung University | Optimal dynamic seam adjustment system and method for image stitching |
US20170243384A1 (en) * | 2016-02-19 | 2017-08-24 | Mediatek Inc. | Image data processing system and associated methods for processing panorama images and image blending using the same |
CN107103583A (en) * | 2016-02-19 | 2017-08-29 | 联发科技股份有限公司 | Image data processing system and correlation technique and associated picture fusion method |
US20180005410A1 (en) * | 2016-06-29 | 2018-01-04 | Xiaoyi Technology Co., Ltd. | System and method for adjusting brightness in multiple images |
US10713820B2 (en) * | 2016-06-29 | 2020-07-14 | Shanghai Xiaoyi Technology Co., Ltd. | System and method for adjusting brightness in multiple images |
US20210215783A1 (en) * | 2017-08-08 | 2021-07-15 | Shanghai United Imaging Healthcare Co., Ltd. | Method, device and mri system for correcting phase shifts |
US11624797B2 (en) * | 2017-08-08 | 2023-04-11 | Shanghai United Imaging Healthcare Co., Ltd. | Method, device and MRI system for correcting phase shifts |
US11234042B2 (en) | 2017-11-14 | 2022-01-25 | Samsung Electronics Co., Ltd. | Display device, control method therefor and recording medium |
Also Published As
Publication number | Publication date |
---|---|
CN102819824B (en) | 2017-03-01 |
EP2533193A1 (en) | 2012-12-12 |
US9105107B2 (en) | 2015-08-11 |
JP2013005439A (en) | 2013-01-07 |
KR101742120B1 (en) | 2017-05-31 |
JP5960513B2 (en) | 2016-08-02 |
CN102819824A (en) | 2012-12-12 |
EP2533193B1 (en) | 2019-01-02 |
KR20120136813A (en) | 2012-12-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9105107B2 (en) | Apparatus and method for image processing | |
US8957984B2 (en) | Ghost artifact detection and removal in HDR image processsing using multi-scale normalized cross-correlation | |
EP2261853B1 (en) | Image processing apparatus, medium, and method | |
WO2018176925A1 (en) | Hdr image generation method and apparatus | |
US8995781B2 (en) | Method and apparatus for deblurring non-uniform motion blur using multi-frame including blurred image and noise image | |
US20110280475A1 (en) | Apparatus and method for generating bokeh effect in out-focusing photography | |
US9508153B2 (en) | Distance measurement apparatus, imaging apparatus, distance measurement method, and program | |
US20140294299A1 (en) | Image processing apparatus and method | |
US20160071253A1 (en) | Method and apparatus for image enhancement | |
CN111340749B (en) | Image quality detection method, device, equipment and storage medium | |
US8295639B2 (en) | Image correction apparatus, image correction method, and image correction program | |
US8885937B2 (en) | Exposure measuring method and apparatus based on composition for automatic image correction | |
EP2536123B1 (en) | Image processing method and image processing apparatus | |
US9258490B2 (en) | Smoothing of ghost maps in a ghost artifact detection method for HDR image creation | |
Zhang et al. | Depth enhancement with improved exemplar-based inpainting and joint trilateral guided filtering | |
US9466007B2 (en) | Method and device for image processing | |
US8218870B2 (en) | Method for segmenting images with intensity-based label propagation and probabilistic of level sets | |
EP3070669A1 (en) | Method and apparatus for color smoothing in an alpha matting process | |
US20130286289A1 (en) | Image processing apparatus, image display apparatus, and image processing method | |
Lee et al. | Multi-image high dynamic range algorithm using a hybrid camera | |
CN117710269A (en) | Underwater polarization imaging method based on linear polarization degree and color correction | |
Hach et al. | Phase-aware candidate selection for time-of-flight depth map denoising | |
CN114862700A (en) | Method, device, equipment and medium for suppressing speckle noise of hologram | |
Liu et al. | Three-stage fusion framework for single image dehazing | |
JP6155609B2 (en) | Parallax calculating apparatus and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHA, JOON HYUK;MOON, YOUNG SU;JEON, YOUNG SUN;AND OTHERS;REEL/FRAME:027831/0018 Effective date: 20111213 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |