JP4445953B2 - Image correction method and apparatus using stamp - Google Patents

Description

  The present invention relates to an image correction method and apparatus using a stamp having a plurality of pixel sizes.

  There is a remarkable spread of information home appliances such as mobile phones, digital cameras, videos and DVDs. The performance of embedded hardware such as LSI and DSP is increasing year by year. For this reason, various software applications that have been running on a personal computer can also be run in real time in a mobile phone.

  Camera shake correction in a mobile phone is one of correction methods based on typical image processing. Erasing unnecessary telop characters on a DVD is also one of correction methods based on image processing.

  At times, in order to implement an algorithm in hardware, it is often necessary to perform an operation by converting a floating point operation into an integer. In order to achieve high speed by solving this mathematical equation, special measures are required.

  On the other hand, when camera shake correction is performed, the image frame becomes one size smaller, so image correction that returns to the original size is required. In addition, regarding a method for erasing unnecessary characters and telops freely, correction to an easy-to-view image is required.

  Conventional image correction is often based on a numerical solution problem (such as Non-Patent Document 1) based on a partial differential equation or the like, and for example, a personal computer is used. However, this image correction is limited to the case where the correction target is rich in low spatial frequency components.

  On the other hand, in the case of a correction target with a clear edge structure and outline, correction up to a high spatial frequency component is necessary, and a method called texture-synthesis in Non-Patent Document 2 is known.

  However, this method has a drawback that the operation cost associated with pattern matching is several times that of the method of Non-Patent Document 1. Although depending on the numerical operation, the method of Non-Patent Document 1 uses an exponential function, and thus there is a problem that it is not easy to simply convert the algorithm from the floating point type to the integer type.

  In the numerical solution problem, the method of correcting the image by the time evolution of the partial differential equation is central. For this reason, a lot of memory, space, and time differential calculations are required, and real-time processing is difficult in a model such as a cellular phone that does not have a high CPU.

As a simple method, a certain amount of image correction can be performed by using a moving average method, a linear interpolation method, or the like. However, there arises a problem that the corrected area is blurred and a problem that correction is performed with an artificial texture different from the surrounding image.
M.Beralmio, G.Sapiro, C.Ballester, and V.Caselles, Computer Graphics, "image inpainting", Proceeding of SIGGRAPH 2000, pp.417-424, July 2000. Li-Yi Wei and M. Levoy, "Texture systhesis over arbitrary manifold surfaces", Proceeding of SIGGRAPH 2001, pp.355-360, Aug. 2001

  The background of the problem setting solved by the present invention has the following problems.

  First problem) Real-time processing is required even on a low-performance CPU such as a mobile phone. Second problem) Depending on the numerical operation, a lot of memory is required. Third problem) High spatial frequency components cannot be corrected due to low image correction. Fourth problem) Due to the algorithm, it cannot be expressed as an integer.

  In view of the above, it is an object of the present invention to provide an image correction method and apparatus that can compensate for missing regions in an image at high speed without using sophisticated mathematical equations.

  In the present invention, for example, high speed is realized as a weighting process for pixels based on filter design, which is extremely low in calculation cost.

  In addition, a complicated image pattern can be handled by applying the stamp size step by step from a small size to a large size in accordance with the surrounding texture feature amount.

  In addition, correction and interpolation to fine textures can be performed at higher speed by hierarchical image formation.

  According to the present invention, a missing region in an image can be compensated at high speed without using sophisticated mathematical equations.

  As a result, areas such as telop and superimpose can be automatically replaced with the original background symbols.

  In addition, it is possible to automatically and effectively replace the original image without a sense of incongruity without performing the conventional blurring operation on the telop area.

  In addition, since simple processing is repeated, high-speed processing in a mobile phone is possible.

  From the above effects, it can be considered to be the core of new image conversion technology in the digital industry in Japan and overseas in the digital camera, digital video, and digital television from home to professional in the broadband age.

  Embodiments of an image correction apparatus and method using a stamp according to the present invention will be described below with reference to the drawings.

  FIG. 1 is a configuration diagram of an image correction apparatus according to the present embodiment.

  This apparatus includes a data input unit 100, a region separation unit 110, a gray value storage setting unit 120, an end determination unit 130, and an image display unit 140. This apparatus is a so-called computer, which reads a computer program and executes processing corresponding thereto.

  In this apparatus, the data input means 100 inputs an image G from a mobile phone camera or a digital camera. The image G is composed of gray values of each pixel. In the monochrome image G, the pixel color is not specified, while in the color image G, red, green, and blue are specified. For example, a gray value of 8-bit gradation is set in the pixel. Here, since processing that does not depend on the color or the number of bits will be described, reference to the matter is omitted.

  The area separating unit 110 erases telops and unnecessary objects from the image G using, for example, a character recognition technique. In addition, the region separation unit 110 separates a region missing due to packet loss, for example. Here, a region where a telop is present or a region where a telop is missing is called a region K.

  The gray value storage setting unit 120 sequentially performs image correction (gray value setting) on the region K. Details will be described later.

  When the image correction is sequentially performed and the similarity between the current area K and the previous area K before the image correction is higher than a predetermined threshold, the end determination unit 130 performs the image correction. Terminate.

  It is to be noted that the image G is converted into an image for each of a plurality of resolutions, and the gradation value setting (image correction) of the image having the lowest resolution is started. You may make it carry out sequentially. Thereby, processing time can be shortened.

  The image display means 140 displays the image after such image correction.

  FIG. 2 is a flowchart of processing performed by the gray value storage setting unit 120. FIG. 3 is a diagram showing the state of the processing.

  First, a stamp (here, a square stamp having a length of 3 pixels in the vertical direction and a length of 3 pixels in the horizontal direction) is set to the initial initial position (the initial initial position is the upper left of the region K here) (S1: FIG. 3 (a)).

  Next, the gray values of all the pixels that have been set at the position of the stamp St are stored (S2).

  Next, the stamp St is moved, for example, by one pixel toward the pixel for which the gray value is set, here, downward (S3: FIG. 3B). That is, the position of the stamp St before and after the movement is partially overlapped.

Direction before movement, here multiple directions except top, here left (illustrated in FIG. 3 (c)), right (illustrated in FIG. 3 (d)), bottom, bottom right, bottom left (these are not illustrated) ) When the stamp St is shifted by, for example, one pixel, the similarity between the post-shift position and the initial position is calculated (S4). That is, the similarity between the post-shift position and the initial position is calculated so that the positions of the stamps St before and after the shift partially overlap. For example, the degree of similarity (R) is obtained by equation (1).

I _ave and J _ave are the average of the gray value at the initial position and the average of the gray value at the position after the shift, respectively.

  I (i, j) and J (i, j) are the shades of pixels at the initial position in the horizontal direction i and the vertical direction j (that is, the X coordinate in the stamp is i and the Y coordinate is j). The gray value of the pixel at the position in the left-right direction i and the up-down direction j (that is, the position where the X coordinate in the stamp is i and the Y coordinate is j) in the post-shift position.

X _SIZE and Y _SIZE are the size of the stamp St in the horizontal direction (X direction) (here, 3 (pixels)) and the size of the stamp St in the vertical direction (Y direction) (here, 3 (pixels)).

  Next, it is determined whether or not the highest similarity is higher than a predetermined threshold TH (S5).

  FIG. 4 is a diagram showing the distribution of similarity at the position near the stamp St after the movement at S3. Although the degree of similarity varies depending on the vicinity position, in S5, it is determined whether the highest degree of similarity (position of ◯) is higher than a predetermined threshold TH.

  Returning to FIG. 2, if YES is determined in S5, the stamp St is positioned at the post-shift position (next initial position) corresponding to the highest similarity (for example, the position of FIG. 3D), and there. The gray value stored in S2 is set to the setting target pixel (S6). Here, the gray value of the pixel having the same position in the stamp St is set. If the gray value has already been set in the area K, it is overwritten or an average of both gray values is set.

  If it is determined NO in S5, the stamp St is positioned at the next initial position (S7). For example, it is positioned at the right position by one stamp St (three pixels) from the immediately preceding initial position (FIG. 3E). In this way, the position of the stamp St is changed to the right.

  After S6 and S7, it is determined whether or not the stamp St exceeds the right end of the region K (S8). If NO is determined, the process returns to S2. If YES is determined, it is determined whether or not the lower end is exceeded if the stamp St is moved down by one (S9). If NO is determined, the stamp St is set to the next initial position (vertical direction). Then, it is positioned one stamp below, and in the left-right direction (position at the left end of the region K) (S10: FIG. 3 (f)), and the process returns to S2. If it is determined YES in S9, the process in FIG.

  Subsequently, the same processing is performed until now, but the following points are different.

  In the processing so far, the position that is in contact with the upper side of the upper left end of the region to be set is set as the initial position, and here, the position that is in contact with the lower side of the lower left end is set as the initial position. Further, the place where the stamp St is moved downward is moved upward here. Further, the similarity when the stamp St is shifted to the left, right, lower, lower right, and lower left is calculated. Here, the similarity when the stamp St is shifted to the left, right, upper, upper right, and upper left is calculated.

  Subsequently, the same processing is performed until now, but the following points are different.

  In the processing so far, a stamp St having a length of 3 pixels in the vertical direction and 3 pixels in the horizontal direction is used. Here, for example, 4 pixels in the vertical direction and 4 pixels in the horizontal direction are used. A stamp St having a length is used. Further processing may be performed by increasing the stamp St in stages.

  FIG. 5 is a schematic diagram of an image showing a state of correction by the conventional method and the present apparatus.

  Conventionally, there is a linear interpolation method for setting a gray value in a direction perpendicular to the region K, and the conventional method cannot reproduce a texture that is incident obliquely on the region K (FIG. 5A). Then, since the gray value is set at the post-shift position with the highest similarity, such an oblique texture can be reproduced (FIG. 5B).

  FIG. 6 is an actual image showing a state of correction by the conventional method and this apparatus.

  FIG. 6A is a mandrill image (original image) which is a standard image in the coding field because it is composed of a complex pattern with various textures and wide bands.

  FIG. 6B is generated on the assumption that a packet loss has occurred in a packet transmitted in the display order when this image is displayed by scanning from the left end to the right end every 16 pixels. It is an image. The image includes four missing areas composed of pixels whose gray values have not been set due to packet loss.

  FIG. 6C shows an image obtained by correction by this apparatus. The four missing areas were interpolated well, and an image almost the same as the original image was obtained.

  FIG. 6D is an image obtained by correction by a conventional method (linear interpolation method). Four defective areas were blurred and interpolated, resulting in an incomplete image. The linear interpolation method linearly interpolates the missing area based on the gray value of the area in contact with the missing area. The longer the distance, the more pixels to be set between known pixels. In addition, since it has the characteristics of a low-pass filter, even when trying to interpolate a texture having a structure such as an edge component, fine information is converted into blurred information. This is the principle of blurring.

  FIG. 7 is a diagram illustrating an evaluation result regarding image quality.

  Regarding the image quality, an evaluation result when the image is corrected with a low spatial frequency pattern and an evaluation result when the image is corrected with a high spatial frequency pattern are shown.

  In detail, the result of the subjective evaluation performed by five persons about restoreability, continuity, etc. is shown. Evaluation result “1” was the worst and evaluation result “5” was the best.

  In addition, here, the evaluation result by the conventional method (linear interpolation method) and the evaluation result by this apparatus are shown. In the conventional method, since the texture structure is broken, the evaluation result with a pattern of a high spatial frequency has deteriorated to near “1”. In addition, the evaluation results in the pattern of high spatial frequency were good from “4” to “5”.

  On the other hand, in this apparatus, the evaluation results in both patterns were good, “4” to “5”. The evaluation results with the high spatial frequency pattern were better than those with the conventional method. This is because this device is based on known images, which is a clear advantage.

  In the present embodiment, the shape of the square stamp St may be another shape such as a rectangle.

  In the present embodiment, the correction is performed while scanning a rectangular area such as the area K so-called line-sequential scanning, but the correction may be performed while scanning so as to go around the correction target area. Thereby, image correction can be performed in a region having an arbitrary shape.

  A computer program that causes the apparatus to execute the above image correction method is recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or a magnetic tape, or a communication network such as the Internet. And can be widely distributed.

It is a block diagram of the image correction apparatus which concerns on this Embodiment. It is a flowchart of the process which the gray value storage setting means 120 performs. It is a figure which shows the mode of the process of FIG. It is the figure which showed distribution of the similarity degree in the vicinity position of stamp St. It is the model of the image which shows the mode of correction | amendment by the conventional method and this apparatus. It is an actual image which shows the mode of correction by the conventional method and this apparatus. It is a figure which shows the evaluation result about an image quality.

Explanation of symbols

100 Data input means 110 Area separation means 120 Gray value storage setting means 130 End determination means 140 Image display means G Image to be corrected K Area to be corrected St Stamp

Claims (9)

An image correction method using a stamp that inputs an image including an area including pixels having grayscale values and an area including pixels to be set, and sets the gray value to the pixels using a stamp having a plurality of pixel sizes. ,
Storing the gray value of the pixel by positioning the stamp at an initial position including the set pixel;
Step position Ru partially overlapping the stamps before and after and the shift by moving the stamp towards the pixel gray value is set,
Calculating the similarity between the post-shift position and the initial position when the stamp is shifted in a plurality of directions excluding the direction before movement;
When the highest similarity is higher than a predetermined threshold, the step of setting the gray value of the post-shift position corresponding to the similarity by the stored gray value is performed a plurality of times while changing the initial position,
When the highest similarity in the previous round is not higher than the threshold, the initial position is made different from the initial position in the previous round, while the highest similarity in the previous round is higher than the threshold and the shift position is shaded. An image correction method using a stamp, characterized in that when a value is set, the post-shift position is set as an initial position.   2. An image correction method using a stamp, wherein the image correction method according to claim 1 is performed by changing a size of the stamp.   The image correction method using a stamp according to claim 1 or 2, wherein the region including the pixels to be set is a region obtained by cutting out a telop in an image by a character recognition technique.   4. The image is converted into an image for each of a plurality of resolutions, and the gradation value setting of an image having a resolution higher than the resolution is sequentially performed starting from the gradation value setting of the image having the lowest resolution. An image correction method using the stamp according to any one of the above. An image correction apparatus using a stamp that inputs an image including an area composed of pixels with grayscale values and an area composed of pixels to be set, and sets grayscale values for the pixels using a stamp having a plurality of pixel sizes. ,
Means for inputting the image;
Means for detecting an area composed of pixels to be set;
Means for positioning a stamp at an initial position including a set pixel and storing the gray value of the pixel;
When the stamp is moved toward the pixel for which the gray value is set , the stamp positions before and after the shift are partially overlapped , and the stamp is shifted in a plurality of directions excluding the direction before the movement, A step of calculating the similarity of the initial position and, when the highest similarity is higher than a predetermined threshold value, setting the gray value of the post-shift position corresponding to the similarity by the stored gray value is the initial position With means to perform multiple times while changing,
When the highest similarity in the previous round is not higher than the threshold, the initial position is made different from the initial position in the previous round, while the highest similarity in the previous round is higher than the threshold and the shift position is shaded. An image correction apparatus using a stamp, characterized in that when a value is set, the post-shift position is set as an initial position.   6. The image correction apparatus using a stamp according to claim 5, wherein the image correction method performed by the image correction apparatus according to claim 5 is performed by changing a size of the stamp.   The image correction apparatus using a stamp according to claim 5 or 6, wherein the region including the pixels to be set is a region obtained by cutting out a telop in an image by a character recognition technique.   8. The image is converted into an image for each of a plurality of resolutions, and the gradation value setting for an image having a resolution higher than the resolution is sequentially performed, starting with the gradation value setting for the image having the lowest resolution. An image correction apparatus using the stamp according to any one of the above. An image correction method using a stamp that sets a gray value for a pixel with a stamp having a size of multiple pixels is input. A computer program to be executed by an apparatus,
Storing the gray value of the pixel by positioning the stamp at an initial position including the set pixel;
Step position Ru partially overlapping the stamps before and after and the shift by moving the stamp towards the pixel gray value is set,
Calculating the similarity between the post-shift position and the initial position when the stamp is shifted in a plurality of directions excluding the direction before movement;
When the highest similarity is higher than a predetermined threshold, the step of setting the gray value of the post-shift position corresponding to the similarity by the stored gray value is performed a plurality of times while changing the initial position,
When the highest similarity in the previous round is not higher than the threshold, the initial position is made different from the initial position in the previous round, while the highest similarity in the previous round is higher than the threshold and the shift position is shaded. A computer program for causing an image correction apparatus to execute an image correction method using a stamp, characterized in that when a value is set, the position after shifting is set as an initial position.