JP5327199B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing method and an image processing apparatus, and more particularly to an image processing method and an image processing apparatus that perform resolution conversion in units of one frame.

  With the recent increase in size and definition of display devices, higher resolution is also demanded for video content. Needless to say, there is a tendency for high-definition shooting of newly produced content, but there is a growing demand for viewing content that has been produced with conventional non-high resolution in a higher resolution form. . In many current television apparatuses, non-high-resolution content is increased by interpolation techniques to match only the number of pixels with a high-definition panel, but high resolution in the original sense has not been practically performed.

  On the other hand, attempts to generate a high-resolution image from a low-resolution image have been made for a long time, and a method for generating a high-resolution image by integrating low-resolution images having misalignment of multiple frames has been studied extremely widely. Yes. In addition to the technique described in Patent Document 1, many techniques for creating a high-definition moving picture from a low-resolution moving picture have been studied. There are also a number of high resolution techniques that use repetitive parts and similar parts in a frame.

JP 2004-56789 A

  Many conventional high resolution techniques rely on interpolation by filters or integration of similar parts. In interpolation using a filter, the output image generally tends to be flat, and it is difficult to express the sharpness of the original image in detail as it is. The “texture” is an element constituting an image, and is composed of a single pixel or a pixel group. In the case of a technique based on the integration of similar parts, the result tends to be uneven, with a part that succeeds and a part that fails even in the same image. In any case, the texture pattern similarity between input and output cannot be secured stably.

  The present invention has been made in view of the above points, and provides an image processing method and an image processing apparatus that can suppress a reduction in sharpness from an original image before enlargement as much as possible with respect to the texture of the image after enlargement. Objective.

  In order to achieve the above object, the image processing method of the present invention extracts pixel value data of a first predetermined number of pixels in a first block unit including a first predetermined number of pixels from a supplied image. An image extraction step, a texture measurement step for measuring a statistic of pixel value data of the first predetermined number of pixels extracted in the image extraction step, and a pixel of the first predetermined number of pixels measured in the texture measurement step A noise generating step for generating noise data corresponding to the size of the second block corresponding to a second predetermined number of pixels larger than the first predetermined number and having a statistic equivalent to the statistic of the value data And dividing the second block of the second predetermined number of noise data generated in the noise generation step into a plurality of sub-blocks, and one sub-block corresponds to one pixel of the first block Then, the difference between the pixel value of the first block and the average of the pixel values in the sub-block is minimized, and the difference sum of the pixel values between adjacent pixels in the second block of noise data is minimized. As described above, the present invention includes a rearrangement step for rearranging the noise data, and an output step for outputting the noise data after rearrangement by the rearrangement step as an enlarged image.

  In order to achieve the above object, the image processing apparatus of the present invention obtains pixel value data of a first predetermined number of pixels in a first block unit composed of a first predetermined number of pixels from a supplied image. Extracting image extracting means, texture measuring means for measuring the statistic of pixel value data of the first predetermined number of pixels extracted by the image extracting means, and first predetermined number of pixels measured by the texture measuring means Noise that generates noise data corresponding to the size of the second block corresponding to a second predetermined number of pixels larger than the first predetermined number and having a statistic equivalent to the statistic of the pixel value data of A second block composed of a generation unit and a second predetermined number of noise data generated by the noise generation unit is divided into a plurality of sub-blocks, and one sub-block corresponds to one pixel of the first block As the first block The noise data so as to minimize the difference between the pixel value of the pixel and the average of the pixel values in the sub-block, and to minimize the difference sum of the pixel values between adjacent pixels in the second block of noise data. The image processing apparatus includes: a sorting unit that performs sorting; and an output unit that outputs noise data after sorting by the sorting unit as an enlarged image.

  According to the present invention, it is possible to suppress a reduction in sharpness from an original image before enlargement as much as possible with respect to the texture of the image after enlargement.

It is a block diagram of one embodiment of an image processing device of the present invention. It is a flowchart of one embodiment of an image processing method of the present invention. It is a figure which shows the arrangement | sequence of 3x3 pixel extracted from the input image. It is a figure which shows 6x6 pixel arrangement | sequence of the data generate | occur | produced as noise. It is a figure which shows an example of the pixel value of 3x3 pixel group extracted from the input image. It is a figure which shows an example of the value of the noise of 6x6 pixel arrangement | sequence before rearrangement. It is a figure which shows the structure of a subblock. It is a figure which shows the adjacent pixel which calculates the difference sum of the pixel value in the case of no edge. It is a figure which shows an example of the noise value of a 6x6 pixel arrangement | sequence after rearrangement. It is a figure which shows the adjacent pixel which calculates the difference sum of the pixel value in the case with an edge.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of an embodiment of an image processing apparatus according to the present invention, and FIG. 2 is a flowchart of an embodiment of an image processing method according to the present invention.

  In FIG. 1, an image processing apparatus 10 according to the present embodiment includes a pixel extraction unit 11, an average / deviation measurement unit 12, a noise generation unit 13, a rearrangement unit 14, an image insertion unit 15, a fine adjustment unit 16, and a diffusion filter 17. Configured to receive a low-resolution image before image processing as an input, and output a high-resolution image enlarged to 200% size. Here, the image processing apparatus 10 processes only the luminance of the image, but it can also be applied to color differences and primary color systems.

  The pixel extraction unit 11 includes a total of nine pixels from the supplied low-resolution image, that is, three pixels in the vertical direction and three pixels in the horizontal direction (hereinafter also referred to as 3 × 3 pixels) including eight pixels around the target pixel. Pixels are extracted in units of one block. FIG. 3 shows an arrangement of the first block of 3 × 3 pixels. In FIG. 3, the pixel indicated by “4” at the center is the target pixel.

  The average / deviation measurement unit 12 performs texture analysis by measuring the average and standard deviation as texture statistics for the input 3 × 3 pixels. The noise generation unit 13 converts noise having an average and standard deviation equivalent to that of the input image into data for the image size of the second block of 6 pixels in the vertical direction and 6 pixels in the horizontal direction (hereinafter also referred to as 6 × 6 pixels). Occurs as. FIG. 4 shows the arrangement of the second block of 6 × 6 pixels. The generation of noise by the noise generator 13 is based on a known normal random number.

  The rearrangement unit 14 uses 2 × 6 × 6 pixels that constitute one second block as described later so that the noise generated by the noise generation unit 13 can be used as a part of the enlarged image. A unit of 4 pixels divided by × 2 pixels is a sub-block, and each sub-block corresponds to one pixel of the first block of 3 × 3 pixels of the input low-resolution image, and 3 × 3 pixels of the input low-resolution image The pixels are rearranged to minimize the difference between the average pixel value and the average pixel value in the sub-block.

  When the edge is present in the 3 × 3 pixels of the input low-resolution image, the image insertion unit 15 does not need to satisfy the Markov property (the attribute value depends only on the attribute value in the immediate vicinity) across the edge. When rearranging the pixels, an image that allows discontinuity across the edge is inserted. In addition, said edge is a case where the luminance pixel difference value between adjacent pixels is a large value exceeding the set threshold value. The fine adjustment unit 16 performs fine adjustment of the enlarged image according to the original image quality of the pixel value. The diffusion filter 17 performs filtering to suppress roughness specific to random noise in the input enlarged image.

  Next, the operation of the signal processing apparatus 10 of the present embodiment shown in FIG. 1 will be described with reference to the flowchart of FIG.

  The pixel extraction unit 11 extracts 3 × 3 pixels in the array shown in FIG. 3 centering on the target pixel as a part of the original image from the input low-resolution image (step S1). Subsequently, the average / standard deviation measuring unit 12 measures the average and standard deviation as the statistics of the texture for each first block of 3 × 3 pixels extracted by the pixel extracting unit 11 (step S2). .

  Next, the noise generating unit 13 generates noise having the average and standard deviation measured by the average / standard deviation measuring unit 12 as data corresponding to the size of the second block of 6 × 6 pixels shown in FIG. (Step S3). For example, the noise generator 13 generates a noise N according to the following equation, where a is an input average and σ is a standard deviation.

N = a + σ × BMN (1)
However, in the above formula, BMN is a normal random number generated by a known Box-Muller method.

  In addition, the meaning that the noise generator 13 generates noise having the same average / standard deviation as the input low-resolution image ensures the appearance similar to the input 3 × 3 pixels by having the same average / standard deviation. It can be done. The reason why the generated noise has a 6 × 6 pixel size is to obtain an enlarged pixel having a double size in the horizontal and vertical directions.

  FIG. 5 shows an example of the pixel values of the 3 × 3 pixel group of the input low resolution image. FIG. 6 shows an example of pixel values of a 6 × 6 pixel group of noise having the same average and standard deviation as the input 3 × 3 pixels generated by the noise generating unit 13.

  However, an enlarged image using the noise data generated by the noise generating unit 13 as a pixel value as it is cannot be configured. When viewed as a random texture, this noise ensures a similar appearance to the input 3 × 3 pixels, but when used as a part of an enlarged image, it becomes a pixel value close to the original image when re-reduced. This is because both the first condition and the second condition that the image appears natural must be satisfied.

  As for the first condition, one block of 6 × 6 pixels is divided by a bold line in FIG. 7, and a total of 4 × 2 pixel units of 2 × 2 pixels is defined as a subblock, and the subblock is one pixel of input 3 × 3 pixels. In order to correspond to the above, the average difference between the pixel value of the input 3 × 3 pixels and the pixel value in the sub-block may be minimized. The second condition is satisfied by ensuring the Markov property (the attribute value depends only on the attribute value in the immediate vicinity). Specifically, the sum total of pixel value differences between adjacent pixels connected by the double arrows in FIG. 8 is calculated and minimized.

  The difference between the average of the pixel value of the input 3 × 3 pixels and the pixel value in the sub-block and the “minimization” of the difference sum of the pixel values between adjacent pixels are an arrangement of noise data that minimizes them. Is to find a replacement. Therefore, the rearrangement unit 14 performs rearrangement according to the 6 × 6 pixel noise data and the proximity pixel correlation (step S4). FIG. 9 shows an example of the result of sorting by the sorting unit 14.

  However, when there is an edge in the input 3 × 3 pixels, it is not necessary to satisfy the Markov property across the edges, and therefore, a corresponding portion is not calculated in the sum of the difference of pixel values between adjacent pixels, and is discontinuous. Allow sex. The presence / absence of this inter-pixel calculation is indicated by the presence / absence of a double arrow in FIG. 10, (a) shows a horizontal edge, (b) shows a vertical edge, (c) shows an example of 6 × 6 pixels when there is an upper left-lower right edge, and (d) a lower left-upper right edge. FIG. 10A shows a case where a horizontal edge exists between the pixel group in the third row and the pixel group in the fourth row, and FIG. 10B shows the pixel group in the third column and the fourth column. A case where a vertical edge exists between the eye pixel group is shown. In FIG. 10C, an oblique edge exists in a pixel group on a diagonally upward straight line connecting the upper left pixel and the lower right pixel. FIG. 10D shows the upper left pixel and the upper right pixel. An example in which an edge in a diagonal direction exists in a pixel group on a straight line rising diagonally to the right connecting an end pixel.

  The presence / absence / direction of the edge may be calculated using a known edge detection operator for each input 3 × 3 pixels. The image insertion unit 15 performs the above processing on the entire image and performs insertion into the image (step S5).

  Next, the fine adjustment unit 16 finely adjusts the enlarged image obtained by the image insertion unit 15 so that the re-reduced image matches the original image (step S6). That is, the fine adjustment unit 16 uses the pixel values Y [j * 2] [i * 2] and Y [j *] of the four pixels of the enlarged image corresponding to one pixel of the original pixel value y [j] [i]. 2] [i * 2 + 1], Y [j * 2 + 1] [i * 2], Y [j * 2 + 1] [i * 2 + 1] are generated, and the average (ave and The following processing is performed to match the original image pixel value. However, in the following formula, ratio = y [j] [i] / ave.

Y (j * 2) [i * 2] = y [j] [i] + (Y [j * 2] [i * 2] −ave) * ratio (2)
Y (j * 2) [i * 2 + 1] = y [j] [i] + (Y [j * 2] [i * 2 + 1] −ave) * ratio (3)
Y (j * 2 + 1) [i * 2] = y [j] [i] + (Y [j * 2 + 1] [i * 2] −ave) * ratio (4)
Y (j * 2 + 1) [i * 2 + 1] = y [j] [i] + (Y [j * 2 + 1] [i * 2 + 1] −ave) * ratio (5)
Then, the diffusion filter 17 performs diffusion filter filtering on the enlarged image finely adjusted by the fine adjustment unit 16 (step S7). The diffusion filter 17 performs smoothing by repeatedly applying filtering based on an arithmetic expression represented by the following expression to the input enlarged image, and suppresses roughness peculiar to random noise. However, in the following expression, Y [j] [i] indicates the pixel value of one pixel of the enlarged image.

Y [j] [i] = Y [j] [i] + ε (-8 * Y [j] [i] + Y [j-1] [i-1] + Y [j-1] [i] + Y [j-1] [i + 1]
+ Y [j] [[i-1] + Y [j] [i + 1] + Y [j + 1] [i-1] + Y [j + 1] [i] + Y [j + 1] [i + 1]) (6)
The second term on the right side of the equation (6) is that the coefficient ε is added to a difference value obtained by subtracting a value eight times the pixel value Y [j] [i] of the target pixel from the sum of the pixel values of the eight pixels around the target pixel. Multiplyed value. Note that the coefficient ε and the number of repetitions in the equation (6) are experimentally determined. For example, the coefficient ε is a value in the range of 0.005 to 0.2, and the number of repetitions is 2 to 5 times.

  Thus, according to the image processing apparatus 10 of the present embodiment, for each 3 × 3 pixel block of the supplied low-resolution image, 6 × having a statistic equivalent to the statistic of the texture. Noise data having a block size of 6 pixels is generated. According to the image processing apparatus 10 of the present embodiment, the block of noise data is divided into a plurality of sub-blocks, and one sub-block corresponds to one pixel of input 3 × 3 pixels, and the input 3 × 3 The noise data is rearranged and rearranged so as to minimize the average difference between the pixel value of the pixel and the pixel value in the sub-block and to minimize the difference sum of the pixel values between adjacent pixels of the noise data. Since a high-resolution image is obtained from the subsequent data, it is possible to suppress the reduction in sharpness from the original image before enlargement as much as possible with respect to the texture of the image after enlargement.

  The present invention is not limited to the case where the image processing apparatus 10 in the block diagram of FIG. 1 is configured by hardware, and includes an image processing program that causes a computer to execute the processing of each step in the flowchart of FIG. To do. In this case, the image processing program may be taken into the computer from a recording medium or may be taken into the computer via a network.

DESCRIPTION OF SYMBOLS 11 Image extraction part 12 Average / standard deviation measurement part 13 Noise generation part 14 Rearrangement part 15 Image insertion part 16 Fine adjustment part 17 Diffusion filter

Claims (2)

An image extracting step of extracting pixel value data of the first predetermined number of pixels in a first block unit including a first predetermined number of pixels from the supplied image;
A texture measurement step of measuring a statistic of pixel value data of the first predetermined number of pixels extracted in the image extraction step;
A second predetermined number of pixels having a statistic equivalent to the statistic of the pixel value data of the first predetermined number of pixels measured in the texture measurement step and greater than the first predetermined number A noise generation step for generating noise data corresponding to the size of the second block corresponding to
The second block including the second predetermined number of noise data generated in the noise generation step is divided into a plurality of sub blocks, and one sub block corresponds to one pixel of the first block. As a minimum, the difference between the pixel value of the first block and the average of the pixel values in the sub-block is minimized, and the difference sum of pixel values between adjacent pixels in the second block of the noise data A reordering step for reordering the noise data so as to minimize
An output step of outputting the noise data after the rearrangement by the rearrangement step as an enlarged image;
An image processing method comprising: Image extracting means for extracting pixel value data of the first predetermined number of pixels in a first block unit comprising a first predetermined number of pixels from the supplied image;
Texture measuring means for measuring a statistic of pixel value data of the first predetermined number of pixels extracted by the image extracting means;
A second predetermined number of pixels having a statistic equivalent to the statistic of the pixel value data of the first predetermined number of pixels measured by the texture measuring means and greater than the first predetermined number Noise generating means for generating noise data corresponding to the size of the second block corresponding to
The second block consisting of the second predetermined number of noise data generated by the noise generating means is divided into a plurality of sub-blocks, and one sub-block corresponds to one pixel of the first block. As a minimum, the difference between the pixel value of the first block and the average of the pixel values in the sub-block is minimized, and the difference sum of pixel values between adjacent pixels in the second block of the noise data Reordering means for reordering the noise data so as to minimize
Output means for outputting the noise data after rearrangement by the rearrangement means as an enlarged image;
An image processing apparatus comprising:

Images