JP2006511900A - Reduction of resolution difference of separated colors - Google Patents

Abstract

Certain embodiments disclosed provide digital image processing to reduce resolution differences between multiple separated colors (images) of a film frame (eg, a red flickering flame, etc.). Use. A location in the red image can be selected using information from another image. The selected location can be modified using information from the other image. The selection can include comparing an edge feature in the first image with a corresponding edge feature in another image. The modification performs a wavelet transform of the two images and applies the transform to the first image with specific coefficients (or a function of those coefficients) generated by applying the transform to the second image. Copying to the coefficients generated by. The copied coefficient can be correlated with the selected location. Other disclosed techniques can be modified from the above and applied to other fields.

Description

  The particular system disclosed relates primarily to image processing, and more particularly to reducing distortion that appears due to resolution differences in separated colors.

  Movie color film is a relatively recent development. Until the advent of cinematographic color film in the 1950s, the process of producing color movies included capturing color information into two or more reels of black and white film. In the first Technocolor 3-color film separation process, a specially designed movie camera was loaded with 3 reels of black and white film. The light entering through the lens was separated into the three primary colors of light, each recorded on a black and white film on a separate reel. After the three reels have been developed, three photographic negatives representing the yellow (opposite blue), cyan (opposite red), and magenta (opposite green) portions of the original scene Was created.

  In addition to creating color separations by the first Technocolor process, color separations have been created and used for long-term storage of color films. This is because the shelf life of black and white film for movies is generally much longer than that of color film. In this process, a cinema color film was used to expose a single reel of black and white film to sequentially record red, green and blue. That is, each frame was printed three times on the final reel so that the separation colors were formed sequentially.

  In a movie studio, the photographic process performed in a film development lab can be used to recombine the three separate colors into a single reel of color film. When each of the three separated colors is divided into separate reels, the size and position of each source reel are adjusted at once using an optical film printer. Specifically, the process has three stages. First, a magenta source reel is projected through the appropriate color filter onto the final reel. Then, the last reel is rewound, the next source reel is loaded, the size is adjusted, and the color filter is changed. This process is repeated until all three separation colors are printed on one final reel by the optical film printer. The resulting final reel is referred to as an intermediate positive (“IP”). At this point, the colors are expressed in red, green, and blue (opposite colors of cyan, magenta, and yellow, respectively).

  Technicor's three-color film separation process, like other processes, is subject to various distortions in the film, including, for example, resolution differences. The difference in resolution occurs because, for example, there is generally a large difference in the resolution or sharpness of the three film separation colors depending on the nature of the optical path and the lens coating of the Technicoror camera. The cyan filter is usually placed behind the yellow filter. This is a so-called bipack arrangement. Light passing through the yellow filter is filtered before reaching the cyan filter, and is badly diffused. As a result, the separation color of yellow (opposite color of blue) generally has a higher resolution than the separation color of cyan (opposite color of red). Since the separation color of magenta (opposite color of green) is not created in the bi-pack arrangement, the resolution is generally equivalent to the separation color of yellow (opposite color of blue). This resolution difference can cause red fringing and blurring around the edges in the picture or image.

(wrap up)
The embodiments described below address the problem of low resolution of the red separation color by increasing the resolution of the selected portion of the red separation color. Information from a higher resolution separation color (blue or green) is used to select a portion for increasing the resolution. For example, such information may include information that a particular edge in a higher resolution separation color has been determined to match an edge in a red separation color. In this case, the specific edge in the red separation color can be selected. After selecting that part, apply the wavelet transform to the red separation color using the information generated by applying the wavelet transform to the higher resolution separation color to increase the resolution of the selected part Correct the matching information generated by. For example, various coefficients (or a function of these coefficients) generated by applying the wavelet transform to the higher resolution separation color, and a set of coefficients generated by applying the wavelet transform to the red separation color Can be copied. This is the case when those coefficients affect the selected part.

  This embodiment provides an efficient automatic process that reduces resolution differences with respect to color film color separations. In addition, this process determines where in the image the resolution difference needs to be reduced or corrected with minimal human intervention.

  Many implementations can be characterized by including “where” and “how” operations. A “where” action determines where to modify the image, which can be determined, for example, by determining a portion that modifies one or more characteristics of the image. The “how” action determines how to modify a portion of the image identified by the “where” action. For example, frequency-based information, time-based information, or both can be used in either or both of the “where” operation and the “how” operation. May also be inter-frame information.

  According to one aspect, reducing the resolution difference selects a first image that includes first information about a scene and selects a second image that includes second information about the scene. Including. There is a resolution difference between a part of the first image and a part of the second image. When determining a location to modify the characteristics of the first image to reduce the resolution difference, the location is in a portion of the first image, and the determination is made from a portion of the second image. Based at least in part on the information obtained. The resolution difference is reduced by modifying the characteristic at the determined location in the portion of the first image.

The first image and the second image may be digital images. Further, the location can include a pixel, and the characteristic can include a luminance value of the pixel or a function of the luminance value.
The first image and the second image may include a film frame separation color or may be extracted from a composite color image. This color composite image can be generated from the separated colors of the film frame. The first image may include a red separation color, and red fringing may occur due to a difference in resolution. In the first image, it is possible to determine an uncorrected portion whose characteristics are not corrected.

  Determining the location can include comparing one or more features of the edges in the first image with one or more features of the edges in the second image. Based on the result of this comparison, the edge can be selected as a location to be corrected.

  Determining the location can also include selecting one or more edges to modify. Selecting one or more edges to modify can include selecting a single edge pixel of an edge that includes a plurality of edge pixels for one of the one or more edges. Selecting one or more edges to modify comprises (i) comparing one or more features of the first image edge to one or more features of the second image edge; , (Ii) selecting the edge as the edge to modify based on the result of the comparison. The one or more features may include a feature selected from the group consisting of edge location, edge direction, edge range, direction in which the luminance is changing, and luminance cross-range. The edge range to be corrected can be determined for each selected edge.

  Multiple edges can be selected for correction. Two of the selected edges can be connected based on the characteristics of the two selected edges, and an edge range can be determined for the selected and connected edges. Connecting the two selected edges can be the spatial proximity between the two selected edges or the luminance difference between each particular pixel of the two selected edges and the two selected edges. It can be based on one or more of the luminance differences between specific pixels spatially located between the edges. Determining the edge range for the selected and connected edges can be based on the edge range that would have been determined for each selected edge prior to connection.

  Based on the size of the selected edge, the selected edge can be deselected. Determining a location in a portion of the first image can be based on information obtained at least in part from the portion of the second image. This information can be dedicated to the first direction. Modifying the property of the location can include generating a modified first image. In the corrected first image, a portion where the characteristic is corrected can be determined. This determination can be based on information obtained at least in part from the second image. Information obtained from the second image can be used in a second direction orthogonal to the first direction. The characteristic can be modified at a location in the modified first image.

The first image can include an image modified with information obtained from the second image. Selecting the second image can include selecting the second image from the set of images based on one or more criteria. The one or more criteria may include luminance information and resolution information.
The location can be determined automatically or interactively. The feathering technique can be applied to the first image area including the correction portion. The feathering technique can be applied after correcting the part.

  Modifying the characteristics of the location in the first image includes applying a first wavelet transform to a portion of the first image to generate a result, and applying a second to a portion of the second image. Applying a wavelet transform of. One or more coefficients generated by applying the first wavelet transform are modified based on the one or more coefficients generated by applying the second wavelet transform, and the correction result is Can be generated.

  In the first image, it is possible to determine an uncorrected portion where the characteristic is not corrected. A digital image can be generated by applying the inverse wavelet transform to the correction result of the first wavelet transform. When the characteristic is corrected at the non-corrected portion in the digital image, the characteristic of the non-corrected portion can be returned to the original value.

  An inverse wavelet transform can be applied to the correction result, and it can be confirmed whether the resolution difference has been reduced between a part of the first image and a part of the second image. An inverse wavelet transform can be applied to the correction result to generate another result, or a feathering technique can be applied to a part of the other result including the determined correction portion. Applying the feathering technique can include linearly interpolating the luminance values among some of the other results.

  According to another aspect, selecting an edge includes accessing a first image and accessing a second image. The second image includes information complementary to the information in the first image. The edge features in the first image are compared with the corresponding edge features in the second image. Based on the result of the comparison, an edge in the first image is selected as an edge to be corrected.

  The selected edge can be modified using the resolution information for the corresponding edge in the second image. Modifying the selected edge can include modifying the information generated by applying the first wavelet transform to a portion of the selected edge. This modification can be based on information generated by applying a second wavelet transform to a portion of the corresponding edge. The first wavelet transform may be the same as the second wavelet transform.

  Modifying the information generated by applying the first wavelet transform applies the first wavelet transform using one or more coefficients generated by applying the second wavelet transform. Substituting one or more coefficients generated by doing so.

  Using one or more coefficients generated by applying the second wavelet transform scales the coefficients generated by applying the second wavelet transform, and converts the scaled coefficients to Generating. The scaled coefficients can be used to replace the coefficients generated by applying the first wavelet transform.

  According to another aspect, modifying the characteristics of the image includes accessing a first image that includes first information about a scene and a second image that includes second information about the scene. Access. There is a difference in resolution between part of the first image and part of the second image. In order to reduce the resolution difference, a location where the characteristics of the first image are corrected is determined. This determination is based on a time domain comparison of a portion of the first image and a portion of the second image. The correction of the characteristic of the part is performed by correcting the information generated by applying the wavelet transform to a part of the first image. This correction of information is performed based on information generated by applying wavelet transform to a part of the second image.

  According to another aspect, reducing the resolution difference includes selecting a first image that includes first information about a scene and a second image that includes second information about the scene. Selecting. There is a difference in resolution between part of the first image and part of the second image. A location for modifying the characteristics of the first image is determined, but the location is in a portion of the first image. The resolution difference is reduced by using the information from the second image to modify the characteristics of the location in the first image.

  The first image and the second image may be digital images. The location can include a pixel, and the characteristic can include a luminance value of the pixel or a function of the luminance value. Information used when correcting the characteristics of the part can include resolution information from the second image.

  Modifying the characteristics of the location in the first image can be achieved by using the information generated by applying the second transformation to the part of the second image, to the part of the first image. Modifying the information generated by applying one transform can be included. Each transform can include a wavelet transform.

  Modifying the information generated by applying the first transform applies the first wavelet transform to a coefficient at a specific location in the result generated by applying the second wavelet transform. Copying to a specific place in the result generated by the above, or scaling and copying. For example, modifying the information generated by applying the first transform may convert the coefficients in each non-baseband subband generated by applying the second wavelet transform to the first wavelet. It can include copying to the corresponding location in the result generated by applying the transformation or scaling and copying.

  Alternatively, modifying the information generated by applying the first transform may include converting the coefficients in each non-baseband subband generated by applying the first wavelet transform to the second wavelet. It can include modifying based on the coefficients at corresponding locations in the result generated by applying the transform. Modifying the coefficients in each non-baseband subband is generated by applying a second wavelet transform to each non-baseband subband generated by applying the first wavelet transform. Copying the coefficient at the corresponding location in the result. The one or more coefficients to be copied can be scaled before copying.

A specific location can be associated with a location whose characteristics are to be modified. Each coefficient that is copied, scaled, or modified can be associated with the location to be modified.
The first image and the second image are film frame separation colors or can be extracted from a composite color image. This color composite image can be generated from the separated colors of the film frame. The first image may be a red separation color, and red fringing may occur due to a difference in resolution.

  Uncorrected parts that do not correct the characteristics can be determined. Modifying the characteristics of the location in the first image using the information of the second image can be obtained by using one or more coefficients generated by applying a wavelet transform to the first image. And modifying based on one or more coefficients generated by applying a wavelet transform to the image. A correction result can be generated by the correction. An inverse wavelet transform can be applied to this correction result to generate a result image, and it can be determined whether or not the characteristics of the uncorrected portion are not corrected in the result image. If it has been corrected, the characteristics of the uncorrected part can be restored to the original values.

  The feathering technique can be applied to the region of the first image including the portion whose characteristics are to be corrected. The feathering technique can be applied after correcting the characteristics of the part. Applying the feathering technique can include linearly interpolating the luminance values within the region.

  Modifying the characteristics of the location can include performing the transformation only in the first direction to generate a modified first image. The modified first image can be transformed only in a second direction orthogonal to the first direction to generate a modified version of the modified first image. Determining a location in a portion of the first image can be based on information obtained at least in part from the portion of the second image. This information can be dedicated to the first direction.

The first image can include an image modified with information obtained from the second image. Selecting the second image can include selecting the second image from the plurality of images based on one or more criteria that can include luminance value information and resolution information.
The location determination can be made automatically or interactively, and the determination can be based on information in the second image. The characteristic of the part can be automatically corrected.

  Determining the location can include selecting one or more edges to modify. For example, for one of the one or more edges, a single edge pixel of an edge that includes multiple edge pixels can be selected.

  Selecting one or more edges to modify compares one or more features of the edges in the first image with one or more features of the edges in the second image Can include. Based on the result of this comparison, the edge can be selected as the edge to be corrected. The one or more features may include a feature selected from the group consisting of edge location, edge direction, edge range, direction in which the luminance is changing, and luminance cross-range.

  The edge range to be corrected can be determined for each selected edge. Based on the size of the selected edge, the selected edge can be deselected. Multiple edges can be selected for correction. Two of the selected edges can be connected based on the characteristics of the two selected edges. An edge range can be determined for the selected and connected edges.

  The two selected edges are either the spatial proximity between the two selected edges or the brightness difference between each particular pixel of the two selected edges and the two selected edges. Connections can be made based on one or more of the luminance differences between specific pixels located spatially. The edge range of the selected and connected edges can be determined for each selected edge based on the edge range that would have been determined prior to connection.

  According to another aspect, modifying the edge includes accessing the first and second images, the second image having information complementary to the information in the first image. Including. An edge is selected in the first image, and the edge is corrected based on information in the second image.

  Modifying the selected edge may include using resolution information about the edge in the second image that may correspond to that edge in the first image. Modifying the selected edge can be accomplished by using the information generated by applying the first wavelet transform to a portion of the edge in the first image to match one of the corresponding edges in the second image. Modifying based on information generated by applying a second wavelet transform to the part.

  Modifying the information generated by applying the first wavelet transform may include copying coefficients or scaled coefficients from the result generated by applying the second wavelet transform. it can. This coefficient or scaled coefficient can be copied to the result generated by applying the first wavelet transform.

  According to another aspect, reducing the resolution difference is to access a first image that includes first information about a scene and to a second image that includes second information about the scene. There is a resolution difference between a part of the first image and a part of the second image. In order to reduce the resolution difference based on the comparison in the time domain between a part of the first image and a part of the second image, a part of the first image whose characteristic is to be corrected is Determine in one image. This resolution difference corrects the information generated by applying the wavelet transform to a part of the first image based on the information generated by applying the wavelet transform to a part of the second image. This is reduced by modifying the characteristics of the part.

The apparatus can include a computer readable medium that stores instructions that, when executed on a machine, cause the various operations described above to be performed. The apparatus can include a processing device that is coupled to a computer-readable medium and that executes stored instructions.
One or more embodiments are shown in the accompanying drawings and the following description. Other embodiments will be apparent from the description, drawings, and claims.
This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application, including color drawings, will be provided by the US Patent and Trademark Office upon request and payment of the necessary fee.

(Detailed explanation)
FIG. 1A shows a portion of a scene where red fringing occurs around various objects. FIG. 1B is a color image of a portion of FIG. 1A showing that there is red fringing around the edge along the pair of white trousers 110 and the edge along the boot 120. For example, region 130 shows red fringing on the right side of boot 120. Region 140 also shows red fringing on white trousers 110. Red fringing is the result of the resolution difference between the separated colors of the color image. As described above, the red fringing may be caused by the photographing process.

  FIG. 2 is the same color image as FIG. 1B, but is the result of processing the color separation using a technique described below that reduces red fringing. As shown in FIG. 2, the red fringing around the white trousers 110 is greatly reduced or eliminated, particularly in the region 140 above the contoured edges of the white trousers 110. The red fringing of the region 130 along the boot 120 is also significantly reduced or eliminated. Further, processing the image of FIG. 1B to reduce red fringing does not cause any visible degradation in resolution or image quality in other aspects.

  FIG. 3 is a block diagram of a system 300 that reduces resolution differences. As an overview, the system 300 mainly performs the following two operations. (1) Identify a location whose resolution should be corrected. (2) Correct the resolution of those places. In the following description, reference is made mainly to individual images (frames), but it will be understood from the description that this process is repeated for each image in the reel.

(Digitizing device)
System 300 receives three separate color images C (cyan), M (magenta), and Y (yellow), and three digital color component images (“digital images”) R _D (red), G _D (green). , And B _D (blue). The subscript D indicates that the image is digitized. Accordingly, the digitizing device 310 includes the steps of reversing the photographic negative (C, M, Y) into a photographic positive and digitizing the resulting positive into three digital images (R _D , G _D , B _D ). Execute.

Each digital image includes an array of pixels having a width and a height. Within the array, the position of each pixel can be expressed as follows.
(X, y), where 0 ≦ x ≦ width, 0 ≦ y ≦ height

Each pixel location has an associated gray level value that can be expressed as:
I (x, y), where 0 ≦ I (x, y) ≦ I_max

  I (x, y) represents the luminance of a specific color (for example, red, green, or blue) at the pixel position (x, y). I_max represents the maximum possible luminance value of the pixel. For example, in the case of 16-bit data, I_max = 65,535.

(Pretreatment device)
The system 300 further includes a pre-processing device 320 that receives the digital images R _D , G _D , and B _D from the digitizing device 310 and generates modified digital images R, G, and B. The pre-processing device 320 is an optional device in the system 300 and performs a pre-processing operation on one or more of the three digital images R _D , G _D , and B _D. For example, one or more of the images R _D , G _D , and B _D can be smoothed, modified using registration algorithms to align with other digital images that make up the frame, or particle removal Can be processed using an algorithm. In one embodiment, the preprocessor 320 operates on all three digital images R _D , G _D , and B _D and (i) particle removal by smoothing, (ii) registration, and (iii) Perform further smoothing operations. If pre-processing device 320 is not included in system 300, images R, G, and B are the same as images R _D , G _D , and B _D , respectively.

(Classifier)
The system 300 further includes a classifier 330 that receives the digital images R, G, and B from the preprocessor 320 and outputs a classification map for one of those images. For example, the classification map C _R identifies in the red digital image, one increases the resolution or plurality of locations (pixel). C _R, each pixel is "fix" rather than luminance values (M), except that includes a label of "optionally modifying the" (PM), or "not fix" (NM), red digital image Can be represented by the same sequence. “Correction” indicates that the pixel luminance value of the corresponding pixel position in the red digital image should be corrected. “Correcting in some cases” means that, for example, depending on the result of an additional test, It indicates that there is a possibility of correcting the pixel luminance value, and “do not correct” indicates that the pixel luminance value should not be corrected.

In one embodiment described in detail below, the PM label is only a temporary label, and the output of the classifier 330 is a classification map where the label attached to the pixel is M or NM and there is no PM label. it is a C _R. Using a provisional PM label allows an implementation that captures and uses the fact that a particular pixel may have passed at least some of the tests that indicate that the pixel should be modified.

  Whether to modify a given pixel location can be determined based on various factors. Since red fringing is a phenomenon related to edges, the classification device 330 uses the edge information to determine which pixels to modify. Edge information can be obtained in one or more dimensions for the edge. For example, the classification device 330 can use edge information related to a two-dimensional edge. Conversely, the classifier 330 can perform two iterations. In the first iteration, for example, edge information related only to the horizontal edge can be used, and in the second iteration, for example, edge information related only to the vertical edge can be used.

  Whether to modify a given pixel in the red digital image is determined based not only on the information in the red digital image R but also on the information in one or more of the other digital images G and B it can. The classifier 330 determines which other digital image to use to provide information used to correct the red digital image. A digital image used for providing the information is referred to as a reference digital image. Various criteria can be used to select the reference digital image. For example, as shown in the following example, luminance value information and resolution information can be used.

In one embodiment, if both the green and blue digital images have edges that coincide with the edges in the red digital image, the edge transition (from one to more pixels) The digital image (green or blue) with the lower average brightness at the last part is selected as the reference digital image for that edge. In another embodiment, if both the green and blue digital images have edges that match the edges in the red digital image, the higher resolution digital image (green or blue) is used as the reference digital image. Selected. There are various definitions of resolution (or definition). For example, the resolution of a set of pixels is determined by dividing the range of brightness values present in those pixels (ie, the maximum brightness value minus the minimum brightness value) divided by the number of pixels in the set. Can be defined. In a further embodiment, the selection criteria for selecting a reference digital image to be used for one particular edge can be different from the selection criteria to be used for another edge. If it is possible to use a plurality of reference digital images, classification map C _R generated by the classifier 330, as well as M information or NM information of each edge pixel, what reference digital image for each edge pixel Information about what is used can also be provided.

  FIG. 4 is a flow chart of a process 400 performed in one embodiment of the classifier 330 to determine whether to correct a pixel in the red digital image. Process 400 includes obtaining 410 a set of edges (edge maps) for R and one or more of G and B. This edge map can be obtained by using, for example, an edge detection filter (for example, a Canny filter). Pixels constituting the edge are called edge pixels or edge transition pixels. Edge information can be combined or divided. For example, the edge information can be divided into a set of information for capturing edges in the orthogonal direction. In one embodiment, horizontal edge information and vertical edge information are acquired separately. FIGS. 5 and 6 show edge maps of a simple red digital image and a simple reference digital image, respectively.

  FIG. 5 shows a sample edge map 500 of a red digital image having only a one-dimensional horizontal edge for simplicity. The edge map 500 shows four edges. The four edges are (i) a one-dimensional horizontal edge covering pixels (1,3) to (3,3), and a primary covering (ii) (5,1) to (9,1) pixels. Cover the original horizontal edge, the one-dimensional horizontal edge that covers pixels (iii) (3,8) to (4,8), and the pixels (iv) (7,6) to (8,6) One-dimensional horizontal edge.

  FIG. 6 shows a sample edge map 600 of a simple reference digital image. The edge map 600 shows two edges. The two edges cover the (i) one-dimensional horizontal edge that covers the pixels from (1,4) to (3,4), and (ii) the pixels from (5,3) to (9,3) One-dimensional horizontal edge.

  Process 400 includes determining (420) descriptive criteria for the edge map of the red digital image and determining (430) descriptive criteria for the edge map of the reference digital image. The descriptive criteria can be determined for each edge pixel within the edge, or can be determined in common for multiple edge pixels within the edge (ie, for all edge pixels that are at most within the edge). In the following, the term edge is used to refer to one edge pixel within an edge, and to refer to multiple edge pixels within an edge (ie, all edge pixels that are at most within the edge). There is. As a descriptive reference, for example, in a red digital image and a reference digital image, (i) whether the edge is a horizontal and / or vertical edge, (ii) the edge transition is high to low in a given direction. Whether to change to luminance or from low to high luminance, (iii) edge position, (iv) edge range, (v) luminance range traversing within the edge, (vi) pixel luminance of the edge and Various other functions of pixel location are included.

  “Edge range” means a set of pixels that define an edge transition. The edge range can be determined from a given edge using various factors (eg, luminance values, etc.). The edge range of a particular edge may be affected by whether there is another edge near that edge. The edge range can be determined in one or more directions. For example, the edge range can be determined for one or two dimensions depending on whether the edge map includes a one-dimensional edge or a two-dimensional edge. An edge range can also be defined for a single edge pixel.

  FIG. 7 shows a sample edge range 700 of four edges in the edge map 500. The edge range is shown by a cross hatch to distinguish it from the actual edge itself, but the edge itself is also considered as a part of the edge range. As shown in the figure, the edge range of the edge starting from (1, 3) extends upward by two pixels and downward by two pixels. The edge range of the edge starting from (5, 1) extends upward by one pixel and downward by three pixels. The edge range starting from (3, 8) extends upward by one pixel and downward by one pixel. The edge range of the edge starting from (5, 1) extends upward by one pixel and downward by two pixels.

  FIG. 8 shows a sample edge range 800 of two edges in the edge map 600. As in FIG. 7, the edge range is shown by a cross hatch to distinguish it from the actual edge itself, but the edge itself is also considered as a part of the edge range. As shown in the figure, the edge range of the two edges extends one pixel both upward and downward.

  Process 400 includes, for each edge in the red digital image edge map, determining whether each edge matches an edge in the reference digital image edge map (440). The term “match” not only indicates that the spatial positions of the edges correspond to each other, but also that there is a resolution difference between that edge of the red digital image and that edge of the reference digital image. It is also used to indicate that the edge of the red digital image is considered a candidate for correction. In other implementations, another criterion may be used (eg, only considering whether the edges correspond spatially) to determine “match”. Factors used in edge evaluation can provide information related to, for example, spatial relationships, the presence of resolution differences, or both. Whether or not the edges match can be determined by various methods, for example, by using not only the edge comparison information but also the individual information of each edge. In one embodiment, to compare edge descriptive criteria, for example, whether the edge directions are equivalent, and whether the edge range luminance value transitions are in the same direction (eg, low to high brightness, or high brightness) To low brightness), whether the brightness ranges of the edge range are equal, and whether the edges meet certain distance criteria (for example, an edge or a specified portion of an edge (start, end, middle, etc.) is a certain distance from each other Etc.).

  In an embodiment, one or more of a variety of other factors can be used to determine whether edges match. Such factors include, for example, (i) the distance between the location of the edge in the red digital image where the luminance value is maximum and the location of the edge in the reference digital image where the luminance value is maximum or minimum, (Ii) the distance between the portion of the edge in the red digital image where the luminance value is minimum and the portion of the edge in the reference digital image where the luminance value is minimum or maximum; (iii) (the red digital image The edge range (or part of an edge range) of this red neighbor edge (if there is another edge in the red digital image (referred to as a neighbor edge) within a certain close distance of the current edge in question) The difference between the average luminance in the red digital image and the average luminance in the reference digital image, measured against, (iv) the edge range of the red edge or reference edge Difference between the average luminance at the red edge and the average luminance at the reference edge, measured for (or part of the edge range), (v) the luminance range traversed by the reference edge and the luminance traversed by the red edge The ratio to the range, (vi) the difference between the luminance value of the red edge and the reference edge (or the difference of the average luminance value in the pixel range) at various locations. Further, as such factors, there is frequency information or resolution information in the wavelet domain in addition to the spatial domain information. In an embodiment, which factor to use can be determined based on specific characteristics of the red and reference edges. For example, different factors can be used depending on whether the red edge and / or the reference edge has a maximum luminance value that exceeds a certain threshold. As another example, depending on whether there is an edge pixel in the reference digital image that matches one or more edge pixels in the vicinity of the edge pixel in a non-reference digital image (eg, a red digital image), Different factors can be used.

  In determining whether two edges match, one or more factors can be analyzed and combined in various ways. For example, the determination may be (i) a binary result, such as whether a particular threshold has been satisfied, or (ii) a large number, such as, for example, whether most of the factors under consideration have yielded a favorable result. Factor analysis (but if those factors may be considered important), (iii) probabilistic discrimination such as probabilistic evaluation of the likelihood of two edges being matched (however, the evaluation varies) Based on various factors). In one embodiment with probabilistic discrimination, labeling a pixel as a PM pixel can indicate that the pixel is likely to be a matching pixel but is less than the probability that the M pixel will match. . The above examples are not necessarily mutually exclusive.

  As with determining match, the determination of whether a pixel should be labeled as a PM pixel can be made according to multiple factors or a single factor. The decision as to whether the PM pixel will eventually become an M pixel or an NM pixel can be made according to the subsequent operation. For example, as described below, the determination of how to finally label a PM pixel can be made by operations such as continuity, removal, and relabeling.

  The operation 440 will be described with reference to FIGS. As described above, in the edge map 500, 12 edge pixels define four edges. In this example, each edge pixel in the edge map 500 (and its associated edge range (in this example, a vertically extending edge range)) is examined, and these edge pixels (and associated range) are edged. Determine if it matches an edge pixel (and associated range) in the map 600. In other implementations, a plurality of edge pixels (up to the entire edge) of the edge can be examined together to determine if those edges match the edges in the edge map 600.

  In FIG. 5, the edge pixels are located from (1,3) to (3,3), from (5,1) to (9,1), from (3,8) to (4,8). And at locations (7, 6) to (8, 6). The distance between the two pixels is acceptable, and the positions (1,3), (2,3), (3,3), (5,1), (6,1), (7,1), and (8 , 1) for the edge pixel at position (1, 4), (2, 4), (3,4), (5, 3), (6, 3), (7, 3) in the edge map 600. Assuming that any other matching condition is satisfied when compared to the edge pixels at (8, 3), respectively, these edge pixels are labeled as M pixels. For the sake of explanation, it is assumed that the edge pixel at the position (9, 3) in the edge map 500 may coincide with the edge pixel at the position (9, 3) in the edge map 600. Assume that the edge pixels at positions (3, 8) and (4, 8) do not match any edge pixel in edge map 600. For the sake of illustration, assume that the edge pixels at positions (7, 6) and (8, 6) may match the edge pixels in edge map 600.

  Process 400 includes labeling (450) each edge pixel of the edge map of the red digital image. These pixels can be labeled as, for example, M pixels if there is a matching reference edge, PM pixels if there is a matching reference edge, and NM pixels otherwise. .

  FIG. 9 shows an edge map 900 obtained by applying operation 450 to edge map 500. In FIG. 9, 8 at positions (1,3), (2,3), (3,3), (5,1), (6,1), (7,1), and (8,1). Two pixels are labeled as M pixels. This is because these edge pixels (and the edge ranges associated with them) have matching pixels in the edge map 600 of the reference digital image. The pixel at position (9, 1) is labeled as a PM pixel. The two pixels at positions (3,8) and (4,8) are labeled as NM pixels, and the two pixels at positions (7,6) and (8,6) are labeled as PM pixels Yes.

  Process 400 includes providing continuity between M and PM pixels in the edge map (460). For example, a specific range of neighborhoods can be defined in the horizontal and / or vertical direction around each M and PM pixel in the edge map of the lower resolution image. Within this neighborhood, if a pixel is identified as an M pixel or PM pixel, its direction (eg, horizontal, vertical, or diagonal) is between that pixel and the M or PM pixel for which the neighborhood is defined. A specific pixel group existing along the (direction) can be similarly labeled as an M pixel or a PM pixel. This is to provide continuity between M pixels or PM pixels. In one such implementation, operation 460 treats all PM pixels as M pixels. That is, all pixels that are relabeled in operation 460 are relabeled as M pixels instead of PM pixels.

  FIG. 10 shows an edge map 1000 obtained by applying operation 460 to the edge map 900 using a neighborhood whose horizontal and vertical ranges are both 5. Since both the horizontal and vertical ranges are 5, the vicinity of the pixel (3, 3) is indicated by a thick outline. The continuity operation 460 changes the pixel (4, 2) to an M pixel as shown in the figure. This is because the pixel (4, 2) is an M pixel (5, 1) and its neighborhood is defined as M. This is because it is between the pixels (3, 3). In the edge map 1000, the vicinity of the pixel (7, 6) whose horizontal and vertical ranges are both 5 is also indicated by a thick outline. However, none of the pixels in the vicinity of pixel (7,6) is changed by continuity operation 460. Pixels (3, 8) and (4, 8) are not explicitly labeled with NM pixels, because it is understood that all unlabeled pixels are NM pixels. .

  Continuity between edge pixels can be provided in various other ways. As an example, if the luminance difference between two edge pixels and the luminance difference between a specific group of pixels spatially located between the two edge pixels are both below a specific threshold, There is a method of connecting two edge pixels.

  Process 400 includes removing 470 spurious M or PM pixels in the edge map. In one embodiment, the extra set of M or PM pixels is removed to a smaller set so that the inclusion of pixels that may be misclassified as M or PM pixels is reduced. To do. Such misclassification can occur, for example, due to noise. Various criteria can be used to perform this removal operation.

  FIG. 11 is a flow chart of a process 1100 used by the classification device 330 to remove M or PM pixels. Process 1100 includes identifying (1110) a set of connected M pixels (and optionally PM pixels). A connected set is defined as a set of M or PM pixels where every M or PM pixel may be in the vicinity of at least one other M or PM pixel in the set, respectively. Is done. M or PM pixels are in the vicinity if they are adjacent to each other in the horizontal, vertical, or diagonal directions. In edge map 1000, there are 8 M pixels and 1 neighboring PM pixel in one connected set, and the remaining 2 PM pixels in another connected set.

  Process 1100 includes determining the number of M pixels (1120) and determining the number of PM pixels (1125) for a given connected set. The process 1100 compares (1130) the number of M and / or PM pixels in the connected set to one or more thresholds and one or more of those thresholds. Re-labeling (1140) M or PM pixels in the connected set as NM pixels. In one embodiment, the sum of the number of M pixels and the number of PM pixels is compared to a size threshold. If this sum is less than the size threshold, re-label those M and PM pixels as NM pixels. Conversely, if the sum is greater than or equal to the size threshold, the PM pixels in the connected set are relabeled to M pixels. Operations 1130 and 1140 may further (or alternatively) include comparing the ratio of M pixels to PM pixels in the connected set to a ratio threshold. If the ratio of M pixels to PM pixels is less than the ratio threshold, the M pixels or PM pixels in the connected set are relabeled to NM pixels. Conversely, if the ratio is greater than or equal to the ratio threshold, the PM pixels in the connected set are relabeled to M pixels.

  In the example of FIG. 10, if the size threshold is set to 3, the connected set of M pixels and one PM pixel are not relabeled as NM pixels, and the connected set of PM pixels is re-labeled as NM pixels. Labeled. If a ratio threshold is applied and the ratio threshold is 8 or less M pixels for one PM pixel, the connected set of M pixels remains as M pixels, and the position (9, The neighboring PM pixels in 1) are relabeled as M pixels and the connected set of PM pixels is relabeled as NM pixels.

  Returning to FIG. 4, process 400 further includes providing an edge range for all M pixels in the edge map (480). For M pixels that are edge pixels in the edge map of the red digital image, the edge range can be interpreted, for example, as the original edge range of the red digital image. The original edge range can be determined, for example, in one dimension (eg, horizontal or vertical) or two dimensions. For M pixels that are not edge pixels, such as those labeled as M pixels in operation 460, the edge range can be interpreted as a function of the edge range of one or more neighboring M pixels, for example.

  In one implementation, the similarity of the edge ranges of a connected set of M pixels is achieved by specifying an equivalent edge range for all M pixels in the connected set. The edge range can be determined, for example, based on a function of the original edge range of M pixels in the connected set having the original edge range. For example, the median, average, minimum, or maximum value of the original edge range can be used as the edge range of all M pixels in the connected set. These median, average, or other types of statistics, determined by collecting M pixels in the connected set, determine the edge range of each M pixel in the connected set. It can also be used as a starting point.

  FIG. 12 shows an edge range 1200 obtained by applying operation 480 to the M pixels of the edge map 1000. The original edge range of M pixels, which are also edge pixels, is shown in FIG. The edge range in FIG. 7 has the median of the range of the upper one pixel and the median of the range of the lower three pixels. These two median values of the original edge range are used for the edge range given in FIG. 12, and the same upper and lower ranges are given to the M pixels labeled in operation 460.

  Process 400 includes labeling (490) all edge range pixels in the edge map as M pixels and all other pixels as NM pixels. FIG. 13 shows a classification map 1300 generated by labeling the edge range pixels of FIG. 12 as M pixels, but it is understood that pixel positions without labels are NM pixels.

(Correction device)
Returning to FIG. 3, the system 300 further includes a correction device 340. The correction device 340 receives (i) the classification map _CR and the three digital images R, G, and B, (ii) increases the resolution of the location identified by the classification map, and (iii) labels as (M1 _R ) Output a modified digital color component image of the red digital image. The retouching device 340 corrects the resolution content of the red digital image using the resolution information from the reference digital image. Since the red digital image and the reference digital image have matching edges, the resolution information from the reference digital image (which is typically a higher resolution digital image than the red digital image) is used to correct the red digital image. Can be guided. The correcting device 340 generates resolution information by applying wavelet transform to the reference digital image, and uses the resolution information to correct information generated by applying wavelet transform to the red digital image. The wavelet transform is useful because the transform generates multiple representations (subbands) with different resolutions of the entire image, and the resolution information at a specific spatial position corresponds to each subband. This is because it is correlated with a specific wavelet coefficient at the position.

  FIG. 14 is a flow chart of a process 1400 that the correction device 340 performs to correct the red digital image. Process 1400 includes performing separate wavelet transforms (1410) on the red digital image and the reference digital image. This conversion can be done, for example, one dimension at a time or two dimensions at a time, and in either case, better results are obtained with a particular implementation. In one implementation that performs two passes, the transformation is one-dimensional in each pass and captures information corresponding to edges having diagonal directions. In the first pass, conversion is performed line by line, and horizontal resolution information is captured. In the next pass, the column is converted and vertical resolution information is captured.

  FIG. 15 is a table 1500 showing the results of a simple example targeting one row. The row “x” is a row of the classification map, and indicates that the row x has 16 pixels and 7 of them are M pixels (the luminance value of the row x of the digital image is not shown). . Seven M pixels are at positions 1-6 and 9, and the remaining pixels are NM pixels (NM pixels are not labeled). In general, a one-level, one-dimensional, row-by-row wavelet transform of a digital image has two total 16 coefficients (not shown) correlated or associated with the digital image row x. A subband is generated. One subband represents low resolution information and the other subband represents high resolution information. In a four-level one-dimensional wavelet transform of a digital image, five subbands are generated. Here, there is one coefficient for subband 0 (baseband), one coefficient for subband 1, two coefficients for subband 2, four coefficients for subband 3, and eight coefficients for subband 4. The position in row x, correlated with each of these 16 coefficients, is indicated by the number displayed in table 1500.

  In general, in the wavelet transform that the correction device 340 performs on the entire digital image, for each row of the digital image, “z” coefficients corresponding to the highest subband (“z” is half the number of pixels in the row). ) “Z / 2” coefficients corresponding to the second highest subband, and “z / 4” coefficients corresponding to the third highest subband (and so on) are generated. Although this wavelet coefficient is said to be correlated or associated with the spatial region of the digital image (in this example, a particular row or part of a row), in general, the resolution component identified by the wavelet coefficient is spatial It is not limited to that area.

  Process 1400, after calculating the wavelet transform, determines which wavelet coefficient locations of the digital image generated by applying the wavelet transform to the red digital image are to be modified. This determination is performed individually for each row and for each subband in each row.

Process 1400 includes determining (1420) which wavelet coefficients in the highest subband to modify for a particular row in the red digital image. In one embodiment, when the position of a particular row of the classification map C _R "2 * j" or "2 * j + 1" is M pixels, in the position in the highest sub-band of the wavelet transform "j" Modify the coefficients (where “j” is greater than or equal to 0 and is measured from the start of the highest wavelet subband).

  The operation 1420 will be described using the table 1500. The highest subband is subband 4, and position 0 is corrected because position 1 in row x of the classification map is M pixels. Similarly, positions 1, 2, 3, and 4 are modified, but positions 5, 6, and 7 are not modified.

  Process 1400 includes determining (1430) which wavelet coefficients in the remaining subbands to modify for the same particular row in the red digital image. In one embodiment, the remaining subbands are processed from high to low. If the position “2 * j” or “2 * j + 1” in the previous (higher) subband is modified, the coefficient at position “j” in the next higher subband of the wavelet transform To correct.

  The operation 1430 will be described using the table 1500. Continuing from the previous description, the next highest subband is subband 3 and position 0 (2 * j) (or position 1 (2 * j + 1)) in subband 4 has been modified so that position 0 (j = 0) is corrected. Similarly, positions 1 and 2 are corrected, but position 3 is not corrected.

Next, subband 2 is processed. Since position 0 (2 * j) (or position 1 (2 * j + 1)) in subband 3 has been corrected, position 0 (j = 0) is corrected. Similarly, position 1 is also corrected.
Next, subband 1 is processed. Since position 0 (j = 0) (or position 1 (2 * j + 1)) in subband 2 has been corrected, position 0 (j = 0) is corrected.
Finally, subband 0 is processed. Since position 0 (j = 0) in subband 1 is corrected, position 0 (j = 0) is corrected. In one implementation, the coefficients for subband 0 are not modified.

  Process 1400 includes modifying (1440) the coefficients at the positions determined above. In one embodiment, the coefficients are modified by replacement. In particular, the position coefficients determined in the results generated by applying the wavelet transform to the reference digital image are determined in the results generated by applying the wavelet transform to the red digital image. Copies (or scales and copies) to the coefficient group at the specified position. Therefore, at the determined coefficient position, the resolution information of the red digital image is replaced with the resolution information (or a function of the resolution information) of the reference digital image. To scale a coefficient is to multiply the coefficient by a specific number.

Note that the correction device 340 only corrects the red (lower resolution) digital image and not the reference (higher resolution) digital image.
Process 1400 includes performing an inverse wavelet transform (1450) on the modified result of applying the wavelet transform to the red digital image. This inverse transform produces a modified red digital image (labeled M1 _{R in} system 300).

(Post-processing equipment)
Returning to FIG. 3, the system 300 further includes a post-processing device 350. Post-processing device 350 receives M1 _R , C _R , and three digital images R, G, and B and generates a modified M1 _R (referred to as M2 _R ). The post-processing device 350 performs three operations, but in other embodiments, one or more of the three operations can be omitted and another operation added.

First, the post-processing device 350, M1 _R pixels corresponding to the NM pixels of C _R confirms that it has not been modified from their luminance values in the red digital image. If any of those M1 _R pixels have been modified, the luminance values of those pixels are returned to the original values given in the red digital image.

  The NM pixel may be corrected by changing the wavelet coefficient by the correction device 340. As already mentioned with respect to table 1500, the factor 0 has been modified in subbands 4, 3, 2, 1, and 0. As can be seen from table 1500, all five of these factors affect NM pixel 0 in row x of the digital image. Therefore, NM pixel 0 can be expected to be modified. In addition, these coefficients are associated with or correlated with a particular group of pixels in the digital image, but generally these coefficients also produce an effect on surrounding pixels (this effect is referred to as a diffusion effect). Thus, for example, coefficient 3 in subband 4 can be expected to affect not only NM pixel 7 but also NM pixel 8 in row x of the digital image.

Second, the post-processing device 350 calculates the resolution of M1 _R or a portion of M1 _R and confirms that the resolution has not decreased as a result of the correction. If the resolution is degraded, various actions can be initiated or performed. For example, when the resolution decreases at a specific pixel position due to the correction, these pixel values can be returned to the original luminance values. Alternatively, any of the parameters of the classification device 330 and / or the correction device 340 can be changed and the operation of the device can be performed again. For example, it or change to change the parameter that determines the C _R, to change the reference digital image, the method of selecting and / or modifying the wavelet coefficients for modifications.

  Third, the post-processing device 350 “feathers” (or similar) using different reference digital images (Refl and Ref2 respectively) at the boundary between M and NM pixels, or between two M pixels. Smoothing) attempts to reduce the visual impact of one or more discontinuities that may exist. Feathering can be applied, for example (as described below in some embodiments) to locations that contain one type of pixel (eg, pixels on one side of a transition boundary) or locations that contain multiple types of pixels ( For example, the present invention can be applied to a pixel in a neighborhood extending over the whole transition boundary between different types of pixels. In one embodiment, feathering is performed along a vertical boundary between two pixel regions that are adjacent in the horizontal direction, as well as along a horizontal boundary between two pixel regions that are adjacent in the vertical direction. To do.

In general, the size of the feathering range affects the rate at which the luminance values of the pixels at the boundary merge from one value to the other. Various techniques can be used to determine the size of the range for each row or column corresponding to the NM / M (or Refl / Ref2) pixel transition. In one implementation, the size of the feathering range is determined based not only on the brightness values of the modified red image M1 _R , but also on the brightness values of R, G, and B. For example,

The red brightness value of the NM pixel in the NM / M transition after the red image is processed by the correction device is M1 _R (i, j), and the NM / M transition after the red image is processed by the correction device. Assuming that the red luminance value of the M pixel at is M1 _R (i, j + l), the value diff can be defined by the following equation.
diff = abs (M1 _R (i, j) −M1 _R (i, j + 1))

This is the absolute value of the red luminance difference between the red value of the NM pixel and the red value of the M pixel. Furthermore, when a particular metric is applied to the (R, G, B) pixel luminance value at position (i, j) and (i, j + 1) and the M1 _R pixel luminance value at position (i, j + 1) The value diff2 can be defined as 1 or 0 based on whether the metric is met. For example, as its particular metric, abs (M1 _R (i, j + 1) -R (i, j + 1)), abs (G (i, j) -G (i, j + 1)), and abs (B (i , J) -B (i, j + l)).

Therefore, the range can be defined as follows:
E1 = (constant * diff) (if diff2 = 1) or 0 (if diff2 = 0)
However, the constant (constant) is 0 or more, but is generally less than 1.

FIG. 16 shows a simple example targeting a part of one column i. In FIG. 16, the pixel (i, j) is an NM pixel, and the pixel (i, j + l) is an M pixel. M1 _R (i, j) = 130, M1 _R (i, j + 1) = 90, and diff = 40. Further, diff2 is set to 1. If the value of “constant” is 0.1, E1 = 4. This includes from M1 _R (i, j + l) to M1 _R (i, j + 4).

This range can be further improved based on the similarity in brightness of neighboring pixels that have received the same designation. In one embodiment, for the pixel (i, k) (j + 1 <k ≦ j + E1, k is an integer), given the M pixel at position (i, j + l) and the range E1 in the example above. Perform the following operations.
1. It is checked whether the pixel (i, k) is the same type of designation (M pixel or NM pixel) as the immediately preceding pixel (i, kl).
2a. If operation 1 is not satisfied, go to operation 4.
2b. The (R, G, B) luminance value of the pixel (i, k) is compared with the (R, G, B) luminance value of the pixel (i, k−1).
3. If the comparison in operation 2b meets a specific metric (different from the metric given diff2 above), k is incremented by 1 and the operation returns to operation 1.
4). If operation 1 is not satisfied, or if the comparison of operation 2b does not meet a specific metric, the range of pixels (i, k) that satisfy operation 3 is defined as a range.

  The above algorithm can be applied to the columns shown in FIG. In the first pass of this algorithm, k = j + 2. In operation 1, pixel (i, j + 2) and pixel (i, j + 1) are compared, and both are M pixels. In operation 2b, the (R, G, B) luminance value of the pixel (i, j + 2) is compared with the (R, G, B) luminance value of the pixel (i, j + l). For example, as this comparison, the absolute value of the difference between the respective color components can be obtained. In this example, the comparison results in (3, 3, 2) (ie, 128-125, 127-130, 118-116). can get. In operation 3, it is determined from the result (3, 3, 2) whether or not the measurement standard is satisfied. As this measurement standard, for example, the maximum tolerance of each component can be used. For example, the value of the maximum tolerance can be 6. In such an embodiment, the metric is met because the difference (3, 3, and 2) for each color component is less than the maximum tolerance of 6.

  In the second pass of this algorithm, k = j + 3. In operation 1, pixel (i, j + 3) and pixel (i, j + 2) are compared, and both are M pixels. If the comparison is an absolute value difference between the luminance values in the operation 2b, the comparison result is that the R component is 7 (125-118) and the G component and B component are both 1. In operation 3, if the metric is a maximum difference of 5, the result of 7 does not meet the metric. In operation 4, pixels (i, j + 1) and (i, j + 2) are defined as ranges. This is less than the previously determined range 4.

  If the maximum tolerance is 7 in the above example, pixel (i, j + 3) meets the metric and the third pass of the algorithm is performed. However, in the third pass, since the pixel (i, j + 4) is an NM pixel, the operation 1 is not satisfied, and the range includes the pixel (i, j + l) to the pixel (i, j + 3). End with (i, j + 3).

  The feathering range can be extended from end to end of both the M pixel position group and the NM pixel position group, from end to end of the M pixel position group only, or from end to end of the NM pixel position group only (or Similarly, when there is a Ref1 / Ref2 boundary transition, similarly, from both ends of the Refl pixel position group and the Ref2 pixel position group, from the end of the Refl pixel position group only to the end, or only the Ref2 pixel position group Can be spread from end to end). When the feathering range is expanded not only from the M pixel position to the end of the NM pixel position, for example, the same processing as the above-described operation can be applied to NM pixels near the NM / M transition boundary.

  When the feathering range is obtained, the luminance value of the pixels in this range can be corrected. In one implementation, new brightness values are obtained by linearly interpolating between brightness values associated with the last pixel of each feathering range within the feathering range. However, a new luminance value can be obtained within the feathering range using various methods.

  FIG. 17 shows a two-dimensional example 1700 that identifies a plurality of pixels subject to the above-described feathering approach near an NM / M boundary transform. For simplicity of display, NM / M pixels are not labeled. Instead, the transition between NM / M pixels is indicated by a thick solid line. Pixels that are affected by the feathering technique in both the horizontal and vertical directions are labeled “S”.

As already briefly described, the one-dimensional wavelet transform can be applied only in one direction by the correction device 340 using the classification map in which edges are described only in one direction. Using the uncorrected red digital image R in the system 300, a new diagonal classification map can be obtained. Correction device 340 receives the new classification map and "old" M2 _R, using a new classification map, application of one-dimensional wavelet transform in the oblique direction to the old M2 _R. Correct the M2 _R, to obtain a new M1 _R. A new M1 _R is sent to the aftertreatment device 350. In the post-processing apparatus 350, the final M2 _R is generated. In other embodiments, the results of multiple passes (diagonal, etc.) can be combined in various ways.

In another embodiment, similar to the above, the one-dimensional wavelet transform is applied only in one direction by a correction device 340 that uses a classification map that describes edges in only one direction. However, in this alternative embodiment, M2 is fed back to the _R classification device 330, the classification unit 330, using the M2 _R instead of the red digital image R, the new shows the M pixels associated with diagonal edge Get a classification map.

(Synthesizer)
Returning to FIG. 3, the system 300 further includes a synthesizer 360. Synthesizer 360, M2 and _R, receives two digital images of more high resolution, to generate these three digital images combined with a composite (synthetic) color image (frame). As already mentioned, an optical printer can be used to combine three digital images to produce a composite color image. In general, when an optical printer is used, the resolution decreases for all three colors. However, if a laser film printer is used, the decrease can be avoided.

(Other embodiments)
Returning to digitizing device 310, in one embodiment, I (x, y) represents the logarithm of the actual luminance value at pixel location (x, y). In another embodiment, (i) perform one or more of various other transformations (eg, a positive to negative conversion instead of or in addition to a negative to positive conversion, etc.). (Ii) no conversion at all, (iii) receiving digitized data so that it does not need to be digitized, and / or (iv) digitizing possible and another digitized component data Receives synthesized data that can be extracted. For example, (iv) includes composite duplex from film color separations and composite color images. If the high resolution separation color and the low resolution separation color are predetermined (or designated) or determined by the digitizing device 310, the digitizing device 310 need only digitize the two separated colors. Furthermore, none of the other blocks in the system 300 need to receive a digital representation of the separation colors that are not in use.

  Returning to the classifier 330, implementations may, for example, simply select a green or blue digital image, or select a digital image that meets certain criteria (eg, a digital image with the highest resolution, etc.). Can determine the reference digital image. The resolution can be obtained by various methods. For example, transform frequency information can be used to identify high-frequency content in digital images or separated colors, or spatial domain techniques can be used to examine edge gradients and other features of edges that indicate high frequency or high resolution. You can. You can apply a resolution-determining technique to determine which separation color or digital image has the lowest resolution, but for cinematography applications, it is generally the red resolution that has the lowest resolution. Note that it may be considered.

  Furthermore, it is not necessary to fix the reference digital image for a given image (frame). For example, the reference digital image may be changed according to the edge or pixel. In addition, if it is determined that a particular edge in the lower resolution image contains matching edges in other digital images, which digital image (or which combination of digital images) is more To determine if it is a preferred reference, the reference digital image can be repeated for two candidates in multiple passes at the classifier 330. In one embodiment, the same reference digital image is always used for a set of connected edge pixels. The reference digital image can be determined based on a reference digital image selected for each edge pixel group, that is, a reference digital image selected by the majority of the edge pixel groups.

  In these examples where multiple reference digital images can be used in an image, the classifier 320 further uses the information used to modify the selected portion of the lower resolution image as multiple reference digital images. It can be specified which one or more of these should be provided. The designation of the reference digital image and the identification of the selected portion to which the given reference digital image is applied can be provided in the classification map as described above or provided separately.

  In other embodiments, low resolution digital images are determined for different portions or features of the frame. In one such implementation, multiple passes are performed through the system 300, each pass processing features of a particular digital image with the lowest resolution of the three digital images.

  The resolution of a digital image (separated color) can be obtained using various criteria, including the criteria described so far. Such criteria can further include, for example, information obtained from wavelet transforms and other transforms. When the resolutions of the two digital images are equivalent, in order to determine a reference digital image for a specific edge from the two, for example, a digital image having a lower average luminance value at the end of the edge range is selected. Even if it is not a digital image with the highest resolution, it can be selected as a reference digital image. For example, a digital image may have some other characteristic that makes it suitable as a reference digital image in certain applications.

  As already mentioned, the classifier 330 can use other features in addition to (or instead of) the edges to determine which pixels to modify. Such features include, for example, luminance value features (eg, luminance values above a certain threshold), object shape, wavelet coefficient values, previously identified as misaligned by some other image processing algorithm Information from the neighboring frame (ie, a time-based feature), etc., indicating that there has been a corrected pixel in the corresponding region and the corresponding region in the neighboring frame. Time-based features can use information from one or more frames that precede or follow the frame in time. In such an implementation, the fact that most of the part of a frame, including edges and other features, remains constant from frame to frame can be exploited.

  Alternative implementations of the classifier 330 may perform the process 400 using one or more of various edge detection methods in addition to (or instead of) the aforementioned Canny filter. For example, the process 400 can obtain a set of edges by identifying transition pixels at the boundary between high and low brightness pixels. For example, if the luminance value of a certain pixel is higher than the threshold value, the pixel is designated as high luminance, and if not, the low luminance pixel can be differentiated by designating the pixel as low luminance. Such a threshold comparison can be made for R, G, and B images.

  Various functions performed by the classification device 330 or other devices need not be performed in all implementations. For example, it is not necessary to achieve continuity between M or PM images in the edge map, and it is not necessary to remove spurious M or PM pixels from the edge map. Further, in the intermediate stages of the various functions performed by the classifier 330, the pixels can be limited to M labels or NM labels only (ie, no PM label need be assigned to every pixel).

  Various functions performed by the classifier 330 can also be performed multiple times in some implementations. For example, the continuity and / or removal steps can be applied multiple times at various different stages during the processing performed by the classifier 330. In an alternative embodiment, the continuity and / or removal steps can be applied after the first set of edges (or edge map) has been generated and the edges of the non-reference digital image and the reference digital image. Before applying the various criteria tests to the edges to determine if they match (ie, if they must be modified).

  In operations 420 and 430 of process 400, the slope of the change in luminance value can be used as a descriptive criterion. The gradient information can also be used for other situations and other purposes. For example, it can be used when determining an edge range (by comparing gradient information) or when determining whether edges match.

  Various criteria can also be used when determining whether to modify a particular M pixel processing method differently than other M pixels in the correction step. If it is determined that the processing method for a particular edge or edge pixel is different, that information can also be provided to the classification map and others. For example, the classification device may change the scaling factor used in the correction operation based on the characteristics of the edge pixel and the range associated with the edge pixel and the characteristic of the edge pixel corresponding to the reference digital image and the range associated therewith. Can be determined.

  Returning to the correction device 340, the resolution content of the image can be corrected using time domain analysis, frequency domain analysis, and / or wavelet domain analysis. In the above embodiment, the range of resolution correction can be limited by using wavelet transform and preventing NM pixels from being corrected in the time domain. Other transforms, particularly transforms in which frequency information or resolution information is correlated with spatial information or temporal information (eg, short-term Fourier transform) can also be used. Furthermore, other types of transformations may be used in a given implementation. In addition, time-based methods (eg, frame-by-frame analysis) already described in the context of classification can also be used to modify an image (frame). Such analysis for each frame includes, for example, many of the methods already described.

  Using wavelet (or other) transforms, the coefficients can be combined in a variety of ways (e.g., copying as described above, or performing functions such as scaling). In addition, the coefficients can be modified based on other factors in the subband. In addition, when given a coefficient diffusion effect, in one embodiment, it is not associated (or correlated) with a particular M pixel, but is expected to affect that particular M pixel by the diffusion effect. Deliberately change the coefficient. In an embodiment, instead of performing the transformation on the entire digital image, the transformation can be performed on a portion of the digital image.

  The wavelet transform used in one embodiment of the correction device 340 is a digital wavelet transform using subsampling. Other implementations of wavelet transforms, or other transforms can also be used.

  Returning to the post-processor 350, various other operations can be performed to verify or improve the resulting resolution. For example, the post-processing device 350 can allow modification of the NM pixel and / or check whether the M pixel has been modified in an advantageous manner. In contrast, it should also be clarified that in the embodiment no post-processing needs to be performed.

  In an embodiment, it is not necessary to process every edge of the red digital image. For example, in one embodiment, in a red digital image, only those edges whose resolution is below a specified threshold or whose resolution is lower than the corresponding edge in the reference digital image by a specified threshold will be corrected. To do.

  The embodiments and techniques described herein can be applied to various applications where it is necessary to reduce the distortion generated from a plurality of separated colors with resolution differences. Examples are spectral separation and non-spectral separation. Spectral separation can be performed, for example, (1) for example, for color film applications that capture various color frequencies, (2) for astronomical applications, for example, that capture radio and / or optical frequencies, (3) for example, various magnetic frequencies (MRI) ), X-ray frequency, and sound (ultrasound) frequency are used in medical applications. As these examples illustrate, spectral separation can be taken from a variety of frequency sources including, for example, electromagnetic waves, sound waves, and the like. Non-spectral separation can be obtained from variations in pressure, temperature, energy, power, etc., for example.

  The embodiments and techniques described herein can also be applied to composite color images, ie, images having multiple color components. For example, a video image may have red, green, and blue components combined into one “composite” image. These components can be separated to form a separation color, and one or more of the embodiments and techniques described herein can be applied to those separation colors.

  Implementations and functions can be implemented in a process, device, or combination of devices. For example, such devices include computers and other processing devices that can process instructions using, for example, a processor, programmable logic device, application specific integrated circuit, controller chip, and the like. The instructions can be, for example, in the form of software or firmware. The instructions can be stored on a computer readable medium (eg, disk, random access memory, read only memory, etc.).

  With reference to FIG. 18, a system 1800 for implementing various disclosed functions includes a processing device 1810 coupled to a computer-readable medium 1820. The computer readable medium 1820 stores instructions 1830 that are processed by the processing device 1810. The process performs various disclosed functions.

  For example, they can be selected by accessing separated colors (digital images). The selection of the separation color (digital image) can be performed, for example, by selecting a file or a representation (for example, a display) of the separation color (digital image). Other representations can be provided, for example, by various user interfaces.

  The digital image group described above includes information covering the same object as a whole. For example, each red, blue, and green digital image contains information (red information, blue information, or green information) that covers the entire frame. Such information (red, blue, and green) can be referred to as “complementary” information because it relates to the same object. In other implementations, digital images covering various objects (eg, areas of the scene being photographed, body parts, sky parts, etc.) can be used.

  In other implementations, a group of digital images in which only a portion of each digital image covers the desired object can be used. For example, the red separation color can be used only for capturing foreground information, and the reference separation color can be used for capturing both foreground information and background information.

  Embodiments can also capture desired objects at various angles and distances into multiple digital images. Such a situation can be the case when using the camera at different distances and angles to shoot different color components, or using different telephoto lenses to shoot the same whole part of the sky.

  In various implementations, for example, one or more operations, functions, or features are performed automatically. “Automatic” means running substantially without human intervention, ie, substantially non-interactively. As an example of automatic processing, there is processing that operates by itself after being started by a human operator. For example, electronic technology, optical technology, mechanical technology, and other technologies can be used as the automatic embodiment.

  The functional blocks, operations, and other disclosed features can be performed in combination in various orders and combinations and can be enhanced by other features not explicitly disclosed. A reference to a part of an image or other object can include the entire image or other object.

  A number of embodiments have been described above. However, it will be understood that various modifications may be made without departing from the spirit and scope of the claims of the present invention. Accordingly, other embodiments are within the scope of the appended claims.

It is an image in which red fringing appears due to the difference in resolution of the separated colors. 1B is a partial color image of FIG. 1A showing red fringing. It is the color image of FIG. 1B to which the process which reduces a resolution difference was performed. It is a block diagram of the system which reduces a resolution difference. It is a flowchart of the process which determines whether a pixel is corrected. An edge map of a simple low resolution digital image. It is an edge map of a simple reference digital image. It is a figure which shows the edge range of the edge of FIG. It is a figure which shows the edge range of the edge of FIG. FIG. 7 illustrates the edges of the edge map of FIG. 5 with edges that match the edge map of FIG.

FIG. 10 shows the edge map of FIG. 9 after applying a continuity operation. It is a flowchart of the process which removes a "correction" pixel. It is a figure which shows the edge range of the edge map of FIG. It is a classification map which shows a "correction" pixel. 6 is a flow chart of a process for modifying a selected portion of a low resolution digital image. It is a table | surface which shows the correlation of the wavelet coefficient and pixel in the line of a digital image. Fig. 2 is a map of a portion of a sequence of images that provides specific information used in a feathering operation. It is a classification map which shows the pixel which can be smoothed. FIG. 6 is a block diagram of a system that implements the disclosed functions.

Claims (28)

A method for reducing the resolution difference;
Selecting a first image containing first information about the scene;
Selecting the second image having a resolution difference between a portion of the first image and a portion of the second image and including second information about the scene;
In order to reduce the resolution difference, a portion of the part of the first image that modifies characteristics of the first image is at least partially from the part of the second image. Determining based on the obtained information; and reducing the resolution difference by modifying the characteristic at the determined location in the portion of the first image.   The method of claim 1, wherein determining the location includes selecting one or more edges to modify. Selecting one or more edges to modify;
Comparing one or more features of the edge in the first image with one or more features of the edge in the second image; and based on the result of the comparison, the edge 3. The method of claim 2, comprising selecting as an edge to correct. Selecting the one or more edges to modify includes selecting the plurality of edges to modify;
The method further comprises connecting two of the selected edges based on characteristics of the selected two edges; and determining an edge range of the selected and connected edges. 2. The method according to 2. Modifying the characteristics of the location in the first image;
Applying a first wavelet transform to the portion of the first image to generate a result;
Applying a second wavelet transform to the portion of the second image; and applying one or more coefficients generated by applying the first wavelet transform to the second wavelet transform. The method of claim 1, comprising modifying based on the one or more coefficients generated by applying to produce a modified result.   The method of claim 5, further comprising determining an uncorrected portion in the first image that does not modify the characteristic. Applying a reverse wavelet transform to the modification result of the first wavelet transform to generate a digital image; and determining whether the characteristic has not been modified at the uncorrected portion of the digital image. The method of claim 6 further comprising:   The method of claim 7, further comprising returning the characteristic of the uncorrected location to an original value if the property is modified at the uncorrected location in the digital image. Applying an inverse wavelet transform to the correction result; and determining whether the resolution difference between the portion of the first image and the portion of the second image is reduced. The method of claim 5 further comprising: Applying an inverse wavelet transform to the modified result to generate another result; and further applying a feathering technique to a portion of the another result, including the determined portion to be modified. The method of claim 5.   The method of claim 10, wherein applying the feathering technique comprises linearly interpolating luminance values within the portion of the another result. A method for modifying the characteristics of an image;
Accessing a first image containing first information about the scene;
Accessing the second image having a resolution difference between a portion of the first image and a portion of the second image and including second information about the scene;
In order to reduce the resolution difference, a characteristic of the first image is modified based on a time domain comparison of the portion of the first image and the portion of the second image. Determining the location; and information generated by applying wavelet transform to the portion of the first image, and generating information by applying wavelet transform to the portion of the second image. Modifying the property of the location by modifying based on information. At least as a result when executed by the machine;
Selecting a first image containing first information about the scene;
Selecting the second image having a resolution difference between a portion of the first image and a portion of the second image and including second information about the scene;
In order to reduce the resolution difference, a portion of the part of the first image that modifies characteristics of the first image is at least partially from the part of the second image. An apparatus comprising: a computer readable medium storing instructions for determining based on the obtained information; and reducing the resolution difference by modifying the characteristic of the location. A method for reducing the resolution difference;
Selecting a first image containing first information about the scene;
Selecting the second image having a resolution difference between a portion of the first image and a portion of the second image and including second information about the scene;
Determining a location in a portion of the first image to modify a characteristic of the first image; and using information from the second image, Reducing the resolution difference by modifying the characteristic of the location.   Modifying the characteristic of the location in the first image uses the information generated by applying a second transformation to the portion of the second image, The method of claim 14, comprising modifying information generated by applying a first transform to the portion of the image. The first transform includes a first wavelet transform;
The second transform includes a second wavelet transform; and further modifying the information generated by applying the first transform results from applying the second wavelet transform 16. The method of claim 15, comprising copying a coefficient at a specific location in to the specific location in the result generated by applying the first wavelet transform.   The method of claim 16, wherein the particular location is associated with the location that modifies the property. The first transform includes a first wavelet transform;
The second transform includes a second wavelet transform; and further modifying the information generated by applying the first transform is generated by applying the second wavelet transform. 16. The method of claim 15, comprising scaling and copying the coefficients in each subband of the baseband to the corresponding locations in the result generated by applying the first wavelet transform. Modifying the property of the location;
15. The method of claim 14, comprising performing the transformation only in the first direction; and generating a modified first image. Performing a transformation on the modified first image only in a second direction orthogonal to the first direction; and generating a modified version of the modified first image. 20. The method of claim 19, comprising. At least as a result when executed by the machine;
Selecting a first image containing first information about the scene;
Selecting the second image having a resolution difference between a portion of the first image and a portion of the second image and including second information about the scene;
Determining a location in a portion of the first image to modify a characteristic of the first image; and using information from the second image, An apparatus comprising a computer readable medium storing instructions for reducing the resolution difference by modifying the characteristic of the location. A device storing an image with increased resolution, wherein the image with increased resolution;
A second component image, wherein the second component image is modified to reduce a resolution difference between the first component image and the second component image. The apparatus includes a part having the specified characteristics, and wherein the part is determined based on position determination information obtained at least partially from the first component image. The resolution enhanced image includes a color composite image including first color information relating to a first color and second color information relating to a second color;
The first component image includes the first color information relating to the first color of the color composite image; and the second component image relates to the second color of the color composite image. 23. The apparatus of claim 22, comprising the second color information. The resolution can be increased by combining the display of each set of separated colors including a set of separated colors and the set of separated colors including a first separated color and a second separated color. Displayed images;
23. The apparatus of claim 22, wherein the first component image includes the first separation color; and the second component image includes the second separation color.   23. The portion of the portion is determined based on a time domain comparison of the portion of the second component image and the corresponding portion of the first component image. Equipment.   23. The apparatus of claim 22, wherein the characteristic is modified using information of a first component from the first component image. Generating information of the first component by applying a wavelet transform to a specific portion of the first component image corresponding to the portion of the second component image; and further, the second component image 27. The apparatus of claim 26, wherein the characteristic is modified by modifying second component information generated by applying a wavelet transform to the portion of the first component using the first component information. . A device storing an image with increased resolution, wherein the image with increased resolution;
A second component image, wherein the second component image reduces a resolution difference between the first component image and the second component image; An apparatus including a portion having characteristics modified using information of a first component from the first component image.

Images