WO2021158039A1

WO2021158039A1 - Method and device for downscaling image

Info

Publication number: WO2021158039A1
Application number: PCT/KR2021/001488
Authority: WO
Inventors: 박두복; 김동규; 이태미
Original assignee: 삼성전자 주식회사
Priority date: 2020-02-05
Filing date: 2021-02-04
Publication date: 2021-08-12
Also published as: KR102449250B1; KR20210099867A

Abstract

According to an embodiment of the present disclosure, a method for downscaling an image may comprise the steps of: applying a second derivative filter in the horizontal and vertical directions of source pixels in an image to extract contour information about the source pixels; determining pixel values of a plurality of horizontal interpolation pixels and a plurality of vertical interpolation pixels on the basis of the pixel values of the source pixels, the contour information, and a downscaling factor; determining pixel values of a plurality of the interpolation pixels on the basis of the pixel values of the plurality of horizontal interpolation pixels, the pixel values of the plurality of vertical interpolation pixels, and weights; applying an adaptive filter to the source pixels, the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of interpolation pixels to determine pixel values of the central interpolation pixels; and downscaling the image to an image composed of the central interpolation pixels.

Description

Method and apparatus for downscaling video

The present disclosure relates to a method and apparatus for downscaling an image.

A display device is a device capable of displaying an image based on a broadcast signal or image signal/image data of various formats by having a display panel for displaying an image, and is implemented as a TV or a monitor. With the recent development of technology, the size of the screen of the display device has been diversified, and the resolution of the image reproduced on the display device has also been diversified. Therefore, it is necessary to provide the user with a modified image resolution by a non-integer magnification. In particular, when the resolution of the original image is greater than the resolution of the current display device, it is necessary to downscale the image by a non-integer magnification in order to provide the user with an image having the same level of clarity as the original image.

Non-integer scale downscaling of an image can be achieved by interpolating integer pixels to create sub-pixels (or interpolated pixels). Existing technologies for downscaling images with non-integer magnification include Bilinear, Bicubic, and Lanczos interpolation filtering. A problem arises. Accordingly, there is an increasing demand for downscaling an image to a non-integer magnification while reflecting the characteristics of the surrounding area without problems such as aliasing and distortion.

An object of the present disclosure is to provide a user with a downscaled image having the same level of sharpness as the original image by interpolating pixels of the original image based on contour information and applying an adaptive filter to the pixels and interpolated pixels of the original image. .

As a technical means for achieving the above-described problem, the method for downscaling an image includes applying a second-order differential filter in the horizontal and vertical directions of the source pixel of the image to obtain outline information of the source pixel. extracting, determining pixel values of a plurality of horizontal interpolation pixels and pixel values of a plurality of vertical interpolation pixels based on the pixel values of the source pixels, the outline information, and the downscaling factor, the plurality of horizontal interpolation pixels determining pixel values of a plurality of interpolation pixels based on pixel values of the directional interpolation pixels, pixel values of the plurality of vertical interpolation pixels, and weights, the source pixel, the plurality of horizontal interpolation pixels, and the plurality of vertical interpolation pixels The method may include determining a pixel value of a central interpolation pixel by applying an adaptive filter to a directional interpolation pixel and the plurality of interpolation pixels, and downscaling the image to an image composed of the central interpolation pixel.

According to embodiments of the present disclosure, a downscaled image having the same level of sharpness as the original image is provided to the user by interpolating pixels of the original image based on contour information and applying an adaptive filter to the pixels and interpolation pixels of the original image. effect that can be provided.

1 is a block diagram illustrating a display device according to an embodiment of the present disclosure.

2 is a diagram illustrating an example of extracting contour information of a source pixel by applying a second-order differential filter according to an embodiment of the present disclosure.

3 is a flowchart illustrating a process of downscaling an image according to an embodiment of the present disclosure.

4A is a diagram illustrating an example of generating an interpolation pixel according to an embodiment of the present disclosure.

4B is a diagram illustrating an example in which an interpolation pixel is generated according to an embodiment of the present disclosure.

5 is a flowchart illustrating a process of downscaling an image according to an embodiment of the present disclosure.

6A is a diagram illustrating an example of generating an interpolation pixel according to an embodiment of the present disclosure.

6B is a diagram illustrating an example of generating an interpolation pixel according to an embodiment of the present disclosure.

6C is a diagram illustrating an example in which an interpolation pixel is generated according to an embodiment of the present disclosure.

The method of downscaling an image includes extracting contour information of the source pixel by applying a second-order differential filter in horizontal and vertical directions of the source pixel of the image, the pixel value of the source pixel, and the contour determining pixel values of a plurality of horizontally interpolated pixels and pixel values of a plurality of vertically interpolated pixels based on the information and a downscaling factor, the pixel values of the plurality of horizontally interpolated pixels, the plurality of vertically interpolated pixels determining pixel values of a plurality of interpolation pixels based on pixel values and weights of , and applying an adaptive filter to the source pixel, the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of interpolation pixels. It may include determining a pixel value of a central interpolation pixel by applying the method, and downscaling the image to an image composed of the central interpolation pixel.

Hereinafter, various embodiments of the present disclosure are described in more detail with reference to the accompanying drawings. However, this disclosure may be embodied in many different forms and should not be construed as limited to any specific structure or function presented in the following description. In contrast, these embodiments are provided to detail and complete the description of the present disclosure. Following the description of the present disclosure, it will be apparent to those skilled in the art that the scope of the present disclosure includes any embodiment of the present disclosure disclosed herein, regardless of whether this embodiment is implemented independently or in conjunction with any other embodiment. will be. For example, the methods and apparatus disclosed herein may be implemented in practice using any of the embodiments disclosed herein. It should also be understood that any embodiment of the present disclosure may be implemented using one or more elements set forth in the appended claims.

And in order to clearly describe the embodiments of the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification. Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. .

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

The word "exemplary" is used herein in the sense of "used as an illustration or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the accompanying drawings.

1 is a block diagram illustrating a display apparatus 100 according to an embodiment of the present disclosure. The display apparatus 100 may include an outline information extraction unit 110 , an interpolation unit 120 , and a downscaling unit 130 .

In the present disclosure, it is described that elements in the display apparatus 100 perform each operation for clarity of operation, but the outline information extractor 110 , the interpolator 120 , and the downscaling unit 130 are all at least It may be implemented by one processor.

The contour information extractor 110 may extract contour information of the image from the input image. The input image may be defined by a two-dimensional grid, where each element of the grid may be referred to as a pixel. Also, in the present disclosure, a pixel of an input image may be referred to as a source pixel. Accordingly, the outline information extractor 110 may extract outline information of each source pixel from pixel information of the 2D source pixel.

A luminance value may be assigned to each source pixel in the digital image. Accordingly, the pixel value may be a luminance value, for example, in an 8-bit image, a pixel value may be 0 to 255, and in a 12-bit image, a pixel value may be one of 0 to 4095. Also, dark pixels may have low pixel values, while bright pixels may have high pixel values.

In the present disclosure, the outline information may be an amount of change in a pixel value of a current source pixel with respect to an adjacent source pixel. In addition, as contour information is extracted, it may be possible to identify a line or boundary of an object. Accordingly, the outline information extraction unit 110 may extract outline information to distinguish between a dark area and a bright area. For example, the outline information extractor 110 may extract outline information, which is a positive value, when the current source pixel is brighter than the adjacent source pixel. Also, as the brightness level of the current source pixel increases, the value of the extracted outline information may also increase. A method of extracting contour information will be described in detail with reference to FIG. 2 .

The interpolator 120 may generate a plurality of sub-pixels (or interpolation pixels) by interpolating a plurality of source pixels existing in at least one patch. In the present disclosure, interpolation may refer to a process of adding a new sub-pixel (or interpolation pixel) between source pixels. According to an embodiment of the present disclosure, the interpolator 120 determines the pixel values of the plurality of horizontal interpolation pixels and the pixel values of the plurality of vertical interpolation pixels based on the pixel values of the source pixels, the outline information, and the downscaling magnification. can Also, according to an embodiment of the present disclosure, the interpolation unit 120 determines pixel values of a plurality of interpolation pixels based on pixel values of a plurality of horizontal interpolation pixels, pixel values of the plurality of vertical interpolation pixels, and weights. can A method of interpolating a source pixel is described in detail with reference to FIGS. 3 to 6B .

The downscaling unit 130 may downscale at least one patch of an image. In the present disclosure, downscaling may refer to an operation of reducing the size of a patch, such as reducing a patch composed of a plurality of pixels to a patch composed of one pixel. For example, the downscaling unit 130 applies an adaptive filter to a source pixel, a plurality of horizontal interpolation pixels, a plurality of vertical interpolation pixels, and a plurality of interpolation pixels to determine a pixel value of a central interpolation pixel, can be downscaled to an image composed of central interpolated pixels.

According to an embodiment of the present disclosure, each of the contour information extraction unit 110 , the interpolation unit 120 , and the downscaling unit 130 receives information on a second-order differential filter, a downscaling magnification, and an adaptive filter from input image information. can be obtained. In addition, information on the second-order differential filter, the downscaling magnification, and the adaptive filter is stored in a memory (not shown) and, if necessary, the contour information extraction unit 110, the interpolation unit 120, and the downscaling unit 130. may be transmitted. In this case, the memory (not shown) should retain data even when power supplied to the display apparatus 100 is cut off, and may be provided as a writable non-volatile memory (Writable Rom) to reflect changes. That is, the memory may be provided with either a flash memory, an EPROM, or an EEPROM.

When the image is downscaled by the downscaling unit 130 , the display apparatus 100 outputs the downscaled image.

According to an embodiment of the present disclosure, the outline information extractor 110 may extract outline information of an image from an input image. In this case, the outline information extractor 110 may extract outline information of a high value in the horizontal direction when the change in pixel values in the horizontal direction is significant. Similarly, when the change in pixel values in the vertical direction is significant, the outline information extractor 110 may extract outline information of a high value in the vertical direction. As a method of extracting the contour information, the first derivative (ie, the gradient) or the second derivative of the pixel value distribution function is used. Here, the filter deriving the first-order differential function of the pixel distribution may be referred to as a first-order differential filter, and the filter deriving the second-order differential function of the pixel distribution may be referred to as a second-order differential filter. When contour information is extracted using the first-order differential filter, coefficients of the first-order differential filter may be determined to find a local directional maxima. In this case, the contour information obtained from the first-order differential filter may include both the position of the pixel from which the contour information is extracted and the directionality of the contour information (also referred to as an edge angle). However, depending on the first-order differential filter in extracting contour information has several problems. Specifically, the outline of an object tends to become wider due to the use of the first-order differential filter. Wide outlines increase the complexity in subsequent software and increase the amount of computation required to detect pixels when attempting to identify objects in the image. Accordingly, in the present disclosure, a process of extracting contour information using a second-order differential filter and downscaling an image based on the extracted contour information will be described.

According to an embodiment of the present disclosure, a second-order differential filter may be applied to a pixel value distribution function of an image in order to extract contour information. In addition, since the same result as the second-order differential function is derived by differentiating the first-order differential function again, the same effect as the second-order differential filter can be obtained by applying the first-order differential filter twice.

By applying a second-order differential filter to a pixel, it is possible to derive a second-order differential distribution function for the transition (zero crossing) of pixel values in the horizontal direction, thereby giving the pixel an outline for the horizontal direction (or x-axis direction) information can be extracted. Next, by applying a second-order differential filter to the pixel, the second-order differential distribution function for the transition (zero crossing) of the pixel values in the vertical direction is derived to obtain the contour information for the vertical direction (or y-axis direction), and the diagonal direction By deriving a second-order differential distribution function for pixel value transition (zero intersection), contour information in the diagonal direction can be extracted. In the present disclosure, it is disclosed that outline information is extracted with respect to a horizontal direction and then outline information is extracted with respect to a vertical direction and a diagonal direction, but the present disclosure is not limited thereto. That is, the direction in which the outline information is extracted is not determined and may be performed in parallel.

In a digital image, since data is arranged at regular intervals, it is impossible to perform a differential operation in a mathematical sense. For this reason, an operation for taking a difference between adjacent pixels using a matrix called a kernel or a mask is performed to facilitate image processing, and thus an operation approximating the differential operation can be performed.

The second-order differential filter of the present disclosure may extract contour information of a source pixel by using the kernels 210 and 230 . In this case, the kernels 210 and 230 may be used for a current source pixel from which contour information is to be extracted and a source pixel adjacent to the current source pixel. Here, the center values 220 and 240 of the kernels 210 and 230 may be reflected in the current source pixel. An example of extracting contour information with respect to the vertical direction of the current pixel using the kernel 210 is as follows:

The sum of all values in the determinant determined through the calculation between the above matrices becomes 510. Accordingly, the second-order differential filter may determine that the contour information of the current source pixel is 510. In this case, positive outline information means that the current source pixel has a pixel value of a higher level than that of an adjacent source pixel when viewed in the vertical direction, and a value of 510 indicates outline information corresponding to another value (for example, 255). It may mean that the amount of change in the pixel value of the current source pixel is larger than that of the corresponding source pixel.

In the present disclosure, it has been described that the outline information in the vertical direction is extracted using the kernel 210 , but the present disclosure is not limited thereto. Accordingly, the kernel 230 may be used to extract the outline information in the vertical direction, and the

kernels

210 and 230 may be used to extract the outline information in the horizontal and diagonal directions.

According to an embodiment of the present disclosure, the second-order differential filter, like the

kernels

210 and 230 , may have a Gaussian-shaped kernel in which the center values 220 and 240, which are the largest values, are reflected in the current source pixel. . Also, the second-order differential filter may have a kernel in which the sum of all kernel values is 0, such as the

kernels

210 and 230 .

When the contour information of the source pixel is extracted by applying the second-order differential filter, the contour information extractor 110 transmits the extracted contour information to the interpolator 120 , and the interpolator 120 may interpolate the source pixel.

According to an embodiment of the present disclosure, downscaling of an image may be performed in units of patches. In the present disclosure, a patch includes a plurality of source pixels, and all interpolation pixels in one patch may be determined based on the same downscaling factor. For example, in one patch, a plurality of horizontal interpolation pixels may be determined based on the same horizontal downscaling factor, and a plurality of vertical interpolation pixels may be determined based on the same vertical downscaling factor. The downscaling factor may be different for each patch, and the source pixel may be overlapped with different patches. Thus, different patches may contain the same source pixel if desired.

Hereinafter, in downscaling an image, a method of downscaling a plurality of source pixels to one interpolation pixel on the premise of one patch will be described.

Referring back to FIG. 3 , in operation S310 , the display apparatus 100 may extract contour information of the source pixel by a second-order differential filter.

The contour information may be an amount of change in a pixel value of a source pixel adjacent to the current source pixel, and contour information corresponding to all source pixels included in one patch may be extracted. Here, the outline information may include outline information for a horizontal direction and a vertical direction. Since the method of extracting the outline information has been described with reference to FIG. 2 , a detailed description thereof will be omitted.

When the outline information of the source pixel is extracted, in operation S320 , the display apparatus 100 may interpolate the source pixel by determining pixel values of a plurality of horizontal direction interpolation pixels and pixel values of a plurality of vertical direction interpolation pixels.

Next, in operation S330 , the display apparatus 100 may interpolate a source pixel by determining pixel values of a plurality of interpolation pixels.

4A is a diagram illustrating an example of generating an interpolation pixel according to an embodiment of the present disclosure. A method of interpolating the source pixel in steps S320 and S330 will be described with reference to FIG. 4A . In addition, in FIG. 4A , for convenience of explanation, the pixel values of the two

horizontal interpolation pixels

420a and 420b and the two

vertical interpolation pixels

430a and 430b for the four

source pixels

410a , 410b , 410c and 410d are shown in FIG. 4A . ) value is determined and a process in which the pixel value of the interpolation pixel 440 is finally determined will be described.

Pixel values of the

horizontal interpolation pixels

420a and 420b and the

vertical interpolation pixels

430a and 430b may be determined by Equation (1) below.

[Equation (1)]

In the present disclosure, subscripts i and j may be indexes for distinguishing pixel positions. Therefore, (i, j) may not be a concept that exactly corresponds to the coordinate values on the x and y coordinates.

_{4A and m ij} in Equation (1) may mean a pixel value (eg, a luminance value) of a source pixel. For example, m ₁₁ may mean a pixel value of the source pixel 410a. Δxx _ij may mean outline information of a source pixel. Specifically, Δxx _ij may mean a change amount of a pixel value in a horizontal direction of the source pixel. For example, Δxx ₁₁ may correspond to a change amount of a pixel value in a horizontal direction of the source pixel 410a with respect to another source pixel adjacent to the source pixel 410a. Also, Δyy _ij may mean outline information of the source pixel. Specifically, Δyy _ij may mean a change amount of a pixel value in the vertical direction of the source pixel. For example, Δyy ₁₁ may correspond to a change amount of a pixel value in the vertical direction of the source pixel 410a with respect to another source pixel adjacent to the source pixel 410a.

In FIG. 4A and Equation (1), k _x may be a horizontal downscaling magnification among the downscaling magnifications, and k _y may be a vertical downscaling magnification among the downscaling magnifications. In this case, k _x and k _y are both positive real numbers between 0 and 1, and their values may be kept constant in determining all interpolated pixels in the same patch.

ΔEx _ij may mean a change amount of a pixel value of a horizontal interpolation pixel, and Ex _ij may mean a pixel value of a horizontal interpolation pixel. For example, Ex ₁₁ may be a pixel value of the horizontal interpolation pixel 420a.

Also, as can be seen from Equation (1), the pixel value of the horizontal interpolation pixel includes the outline information of the left source pixel of the horizontal interpolation pixel, the pixel value of the left source pixel, and the outline information of the right source pixel of the horizontal interpolation pixel. , may be determined based on a pixel value of the right source pixel and a horizontal downscaling factor. For example, the pixel value of the horizontal interpolation pixel 420a is the pixel value m ₁₁ of the source pixel 410a located to the left of the horizontal interpolation pixel 420a, the outline information Δxx ₁₁ , and the horizontal interpolation pixel 420a It may be determined through calculation of _{the pixel value m 12} of the source pixel 410b located to the right of , the outline information Δxx _{12 ,} and the horizontal downscaling magnification k _{x .}

Also, similarly to the horizontal interpolation pixel 420a, the pixel value of the horizontal interpolation pixel 420b is the pixel value m ₂₁ of the source pixel 410c located to the left of the horizontal interpolation pixel 420a and the outline information Δxx ₂₁ , may be determined by calculating _{the pixel value m 22} of the source pixel 410d located on the right side of the horizontal interpolation pixel 420b , the outline information Δxx _{22 ,} and the horizontal downscaling magnification k _{x .}

ΔEy _ij may mean a change amount of _{the pixel value of the interpolation pixel in the vertical direction, and Ey j} may mean the pixel value of the interpolation pixel in the vertical direction. For example, Ey ₁₁ may be a pixel value of the vertical interpolation pixel 430a.

In addition, as can be seen from Equation (1), the pixel value of the vertical interpolation pixel includes the contour information of the upper source pixel of the vertical interpolation pixel, the pixel value of the upper source pixel, and the contour information of the lower source pixel of the vertical interpolation pixel. , may be determined based on a pixel value of the lower source pixel and a vertical downscaling magnification. For example, the pixel value of the vertical interpolation pixel 430a is the pixel value m ₁₁ of the source pixel 410a located above the vertical interpolation pixel 430a, the outline information Δyy ₁₁ , and the vertical interpolation pixel 430a. a it can be determined through calculation of pixel values and contour information ₂₁ m yy △ ₁₂ and in the longitudinal direction of the downscaling ratio ky source pixel (410c) located on the lower side.

Also, similarly to the vertical interpolation pixel 430a, the pixel value of the vertical interpolation pixel 430b is the pixel value m ₁₂ of the source pixel 410b located above the vertical interpolation pixel 430b and the outline information Δyy ₁₂ _{, may be determined through calculation of the pixel value m 22} of the source pixel 410d positioned below the vertical interpolation pixel 430b, the outline information Δyy ₂₂ and the vertical downscaling magnification ky.

When the pixel values of the

horizontal interpolation pixels

420a and 420b and the pixel values of the

vertical interpolation pixels

430a and 430b are determined, the pixel value of the interpolation pixel 440 may be determined.

According to an embodiment of the present disclosure, the pixel value of the interpolation pixel 440 may be determined by Equation (2).

[Equation (2)]

_{E ij} in FIG. 4A and Equation (2) may be a pixel value of an interpolation pixel existing at a position corresponding to the index (i, j). For example, E ₁₁ may correspond to a pixel value of the interpolation pixel 440 .

_{W 1} in Equation (2) corresponds to a weight value of the interpolation pixel 440 of the vertical interpolation pixel 430a, W ₂ is a weight value of the interpolation pixel 440 of the vertical interpolation pixel 430b, W ₃ may correspond to a weight of the interpolation pixel 440 of the horizontal interpolation pixel 420a, and W ₄ may correspond to a weight of the interpolation pixel 440 of the horizontal interpolation pixel 420b. In this case, each weight may correspond to a ratio of a distance of the interpolation pixel 440 to the

adjacent interpolation pixels

420a, 420b, 430a, and 430b. Accordingly, W ₁ may correspond to a horizontal downscaling factor k _x _{, and W 3} may correspond to a vertical downscaling factor k _y .

As can be seen from Equation (2), the pixel value of the interpolation pixel may be determined based on pixel values of the plurality of horizontal interpolation pixels, the pixel values of the plurality of vertical interpolation pixels, and weights. Specifically, the pixel value of the interpolation pixel is the pixel value of the upper horizontal interpolation pixel of the interpolation pixel, the pixel value of the lower horizontal interpolation pixel, the pixel value of the left vertical interpolation pixel, and the pixel value and weight of the right vertical interpolation pixel. can be determined based on For example, the pixel values of interpolation pixel 440 are pixel values Ex ₁₁ , Ex ₂₁ _{of horizontal interpolation pixels 420a, 420b , pixel values Ey 11} , Ey _{21 of}

vertical interpolation pixels

430a, 430b and weights W ₁ ,It may be determined through the operation of _{W 2} , W ₃ , and W _{4 .}

In FIG. 4A , for convenience of explanation, two

horizontal interpolation pixels

420a and 420b and two

vertical interpolation pixels

430a and 430b are determined for the four

source pixels

410a, 410b, 410c, and 410d, and finally A process in which the interpolation pixel 440 is determined has been described. However, the downscaling of the present disclosure may be performed in units of patches. In addition, in the present disclosure, the patch before interpolation may include a source pixel composed of 4x4 pixels. Accordingly, the patch (or extended patch) after the interpolation process may be composed of 7x7 pixels, but is not limited thereto. Specifically, the patch before interpolation may consist of n x n pixels, and the patch after interpolation may consist of (2n-1) x (2n-1) pixels.

The extended patch 460 of beam 4b is a patch after the interpolation process, and the extended patch 460 may include a plurality of source pixels, a plurality of horizontal interpolation pixels, a plurality of vertical interpolation pixels, and a plurality of interpolation pixels. there is. Specifically, as can be seen from FIG. 4B , the extended patch 460 after the interpolation process may be composed of 7x7 pixels, 16 source pixels, 12 horizontal interpolation pixels, 12 vertical interpolation pixels, and 9 It may include interpolation pixels. In this case, the method of interpolating each horizontal interpolation pixel, the vertical interpolation pixel, and the source pixel of FIG. 4B may be performed through the method of interpolating the source pixel described with reference to FIG. 4A .

When the pixel values of the plurality of interpolation pixels are determined, the display apparatus 100 finally applies the adaptive filter to determine the pixel values of the central interpolation pixel 460 and converts the extended patch 460 into the central interpolation pixel 460 . You can downscale the image by downscaling. In the present disclosure, the central interpolation pixel 460 may be a geometrically centered pixel among a plurality of interpolation pixels in the extended patch 460 . Accordingly, the position of the central interpolation pixel 460 may be the same as the position of the interpolation pixel having the _{pixel value E 22 .}

Hereinafter, a method of determining the pixel value of the central interpolation pixel 460 by applying the adaptive filter will be described.

Referring back to FIG. 3 , when pixel values of a plurality of interpolation pixels are determined in steps S320 and S330 , in step S340 , the display apparatus 100 applies an adaptive filter to determine the pixel values of the central interpolation pixel 460 . can

In the present disclosure, the application of the adaptive filter means that the pixel of a specific pixel is applied in consideration of the Euclidean distance between pixels, the position of the pixel in the image, the pixel value, and the variance of the pixel value by applying the filter to the pixel. It can mean determining a value. In addition, application of the adaptive filter may mean that a convolution operation is mathematically performed.

The adaptive filter may perform filtering according to Equation (3):

[Equation (3)]

F(x, y) is a mathematical representation of an adaptive filter, G(x, y) is a mathematical representation of a Gaussian filter, and H(x, y) may be a mathematical representation of a range filter. As can be seen from Equation (3), the adaptive filter F(x, y) of the present disclosure may be derived through a convolution operation of the Gaussian filter G(x, y) and the range filter H(x, y). Also, in the present disclosure, the Gaussian filter G(x, y) may be a Gaussian type bilateral filter. The bidirectional filter is a filter for the purpose of reducing the FIR (Finite Impulse Response) filter error occurring near the contour and preserving the contour information.

The x and y of the Gaussian filter G(x, y) and the range filter H(x, y) may correspond to coordinates in an image of a specific pixel. Also, the Gaussian filter G(x, y) may include a parameter g indicating space variance, and the range filter H(x, y) may include a parameter r indicating a range variance. .

Specifically, the parameter g isIt may be a parameter indicating that a high weight is given to pixels located at a close distance to a pixel located at the center in the extended patch 460 using a Gaussian function. Here, the pixel located at the center does not mean a pixel located in the geometrically correct center, but may mean an interpolation pixel having a center index among interpolation pixels. Accordingly, referring to FIG. 4B , the _{interpolation pixel having the pixel value E 22} may be a pixel located at the center for determining the parameters g and r. The parameter r may be a parameter corresponding to a difference between the current pixel and a pixel distribution located at the center. Accordingly, as the pixel values of the current pixel and the pixel located at the center are similar, a higher weight may be assigned to the current pixel.

Also, I(x, y) in Equation (3) may be a pixel value of a pixel located at (x, y) coordinates, and C may be a pixel value of a pixel located at the center. Accordingly, C may correspond to _{E 22 .}

According to an embodiment of the present disclosure, the pixel value of the central interpolation pixel 470 may be determined by applying an adaptive filter to all pixels in the extended patch 460 . Specifically, the pixel value of the central interpolation pixel 470 may be determined by applying an adaptive filter to all source pixels, horizontal interpolation pixels, vertical interpolation pixels, and interpolation pixels in the extended patch 460 .

Accordingly, the pixel values of the central interpolation pixel 470 are the pixel values of all source pixels in the extension patch 460, the pixel values of the horizontal interpolation pixels, the pixel values of the vertical interpolation pixels, and the pixel values of the interpolating pixels and the adaptive filter. It can be determined through a convolution operation of F(x, y).

When the pixel value of the central interpolation pixel 470 is determined, in operation S350 , the display apparatus 100 may downscale the image to an image including the central interpolation pixel. Specifically, the display apparatus 100 may downscale the image by downscaling the extended patch 460 to the central interpolation pixel 460 . In order to downscale the input image, a plurality of extended patches may be generated and the plurality of extended patches may partially overlap each other, so that the display apparatus 100 may obtain a downscaled image with a non-integer magnification.

According to an embodiment of the present disclosure, in order to determine the pixel value of the central interpolation pixel, not only the horizontal direction interpolation pixel and the vertical direction interpolation pixel, but also the diagonal interpolation pixel may be further considered.

First, in operation S510 , the display apparatus 100 may extract outline information of a source pixel by a second-order differential filter. Here, the outline information may include outline information for a diagonal direction in addition to the outline information for the horizontal and vertical directions. Since the method of extracting the contour information has been described with reference to FIG. 2 , a detailed description thereof will be omitted.

When the outline information of the source pixel is extracted, in step S520 , the display apparatus 100 determines the pixel values of the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of diagonal interpolation pixels to select the source pixels. can be interpolated.

Next, in operation S530 , the display apparatus 100 may interpolate a source pixel by determining pixel values of a plurality of interpolation pixels.

6A is a diagram illustrating an example of generating an interpolation pixel according to an embodiment of the present disclosure. A method of interpolating the source pixel in steps S520 and S530 will be described with reference to FIG. 6A . In addition, in FIG. 6A , for convenience of explanation, the pixel values of the two

horizontal interpolation pixels

420a and 420b and the two

vertical interpolation pixels

430a and 430b with respect to the four

source pixels

410a, 410b, 410c, and 410d are shown in FIG. ) and the pixel values of the two

diagonal interpolation pixels

610a and 610b are determined, and finally the pixel values of the interpolation pixel 620 are determined.

Since the method of determining the pixel values of the

horizontal interpolation pixels

420a and 420b and the pixel values of the

vertical interpolation pixels

430a and 430b has been described with reference to FIG. 3 , a detailed description thereof will be omitted.

Pixel values of the

diagonal interpolation pixels

610a and 610b may be determined by Equation (4) below.

[Equation (4)]

_{6A and m ij} in Equation (4) may mean a pixel value of a source pixel. Also, Δxy _ij and Δyx _ij may mean outline information of a source pixel. Specifically, Δxy _ij may mean a change amount of a pixel value in a diagonal direction having a negative slope of the source pixel. In the present disclosure, a diagonal direction having a negative slope may mean that when one diagonal interpolation pixel is generated and added between two source pixels, an angle formed by the two source pixels is a negative number. For example, Δxy ₁₁ may correspond to a change amount of a pixel value in a diagonal direction of the source pixel 410a with respect to another source pixel adjacent to the source pixel 410a. Here, the diagonal interpolation pixel 610a is generated between the source pixel 410a and the source pixel 410d, and an angle between the source pixel 410a and the source pixel 410d may have a negative slope. Δyx _ij may mean a change amount of a pixel value in a diagonal direction having a positive slope of the source pixel. In the present disclosure, a diagonal direction having a positive slope may mean that when one diagonal interpolation pixel is generated and added between two source pixels, an angle formed by the two source pixels is positive. For example, Δyx ₁₁ may correspond to a change amount of a pixel value in a diagonal direction of the source pixel 410b with respect to another source pixel adjacent to the source pixel 410b. Here, the diagonal interpolation pixel 610b may be generated between the source pixel 410b and the source pixel 410c, and an angle between the source pixel 410b and the source pixel 410c may have a positive slope.

In FIGS. 6A and Equation (4), k _x may be a horizontal downscaling magnification among the downscaling magnifications, and k _y may be a vertical downscaling magnification among the downscaling magnifications. In this case, k _x and k _y are both positive real numbers between 0 and 1, and their values may be kept constant in determining all interpolated pixels in the same patch.

ΔExy _ij may mean a change amount of a pixel value of a diagonal interpolation pixel having a negative slope, _{and Exy ij may mean a pixel value of a diagonal interpolation pixel having a negative slope.} For example, Exy ₁₁ may be a pixel value of the diagonal interpolation pixel 610a.

In addition, as can be seen from Equation (4), the pixel value of the diagonal interpolation pixel having a negative slope is the contour information of the left-upper source pixel of the diagonal interpolation pixel having a negative slope, and the left-upper source pixel of the pixel value. It may be determined based on the pixel value, the contour information of the right-lower source pixel of the diagonal interpolation pixel having a negative slope, the pixel value of the right-lower source pixel, the horizontal downscaling magnification, and the vertical downscaling magnification. For example, the pixel value of the diagonal interpolation pixel 610a is the pixel value m ₁₁ of the upper-left source pixel 410a of the diagonal interpolation pixel 610a, the outline information Δxy ₁₁ , and the diagonal interpolation pixel 610a _{The pixel value m 22} of the right-lower source pixel 410d of , Δxy ₂₂ , may be determined through calculation of a horizontal downscaling magnification k _{x and a vertical downscaling magnification ky.}

ΔEyx _ij may mean a change amount of a pixel value of a diagonal interpolation pixel having a positive slope, _{and Eyx ij may mean a pixel value of a diagonal interpolation pixel having a positive slope.} For example, Eyx ₁₁ may be a pixel value of the diagonal interpolation pixel 610b.

Also, similar to a diagonal interpolation pixel having a negative slope, the pixel value of a diagonal interpolation pixel having a positive slope is the contour information of the right-top source pixel of the diagonal interpolation pixel having a positive slope, right-top It can be determined based on the pixel value of the source pixel, the contour information of the left-bottom source pixel of the diagonal interpolation pixel having a positive slope, the pixel value of the left-bottom source pixel, the horizontal downscaling factor, and the vertical downscaling factor there is. For example, as can be seen through Equation (4), the pixel value of the diagonal interpolation pixel 610b is the pixel value m ₁₂ of the right-upper source pixel 410b of the diagonal interpolation pixel 610b and the outline information Δyx ₁₂ _{, the pixel value m 21} of the left-lower source pixel 410c of the diagonal interpolation pixel 610b and the contour information Δyx ₂₁ , the horizontal downscaling factor k _x and the vertical downscaling factor ky through the calculation can be decided.

According to an embodiment of the present disclosure, when a diagonal interpolation pixel is further considered, the pixel value of the interpolation pixel 620 may be determined by Equation (5).

[Equation (5)]

_{E ij} in FIG. 6B and Equation (5) may be a pixel value of an interpolation pixel existing at a position corresponding to the index (i, j). For example, E ₁₁ may correspond to a pixel value of the interpolation pixel 640 .

_{W 1} in Equation (5) corresponds to a weight value of the interpolation pixel 620 of the vertical interpolation pixel 430a, W ₂ is a weight value of the interpolation pixel 620 of the vertical interpolation pixel 430b, W ₃ may correspond to the weight of the interpolation pixel 620 of the horizontal interpolation pixel 420a, and W ₄ may correspond to the weight of the interpolation pixel 440 of the horizontal interpolation pixel 620b. In this case, each weight may correspond to a ratio of a distance of the interpolation pixel 440 to the

adjacent interpolation pixels

420a, 420b, 430a, and 430b. Accordingly, W ₁ may correspond to a horizontal downscaling factor k _x _{, and W 3} may correspond to a vertical downscaling factor k _y . Also, W ₅ may correspond to a weight of the interpolation pixel 440 of the diagonal interpolation pixel 610a and W ₆ may correspond to a weight of the interpolation pixel 440 of the diagonal interpolation pixel 610b. In this case, the _{values of W 5} and W ₆ may be determined by the Pythagorean theorem.

As can be seen from Equation (5), when diagonal interpolation pixels are further considered, the pixel values of the interpolation pixels are pixel values of a plurality of horizontal interpolation pixels, pixels of a plurality of vertical interpolation pixels, and a plurality of diagonals. The directional interpolation may be determined based on pixel values and weights of pixels. Specifically, the pixel value of the interpolation pixel is the pixel value of the upper horizontal interpolation pixel of the interpolation pixel, the pixel value of the lower horizontal interpolation pixel, the pixel value of the left vertical interpolation pixel, the pixel value of the right vertical interpolation pixel, negative It may be determined based on a pixel value of a diagonal interpolation pixel having a slope, a pixel value of a diagonal interpolation pixel having a positive slope, and a weight. For example, the pixel values of the interpolation pixel 620 are the pixel values Ex ₁₁ , Ex ₂₁ _{of the horizontal interpolation pixels 420a, 420b , the pixel values Ey 11} , Ey _{12 of the}

vertical interpolation pixels

430a, 430b diagonal _{pixel values E xy11} , E _yx11 and weight W _{1 of} directional

interpolated pixels

610a , 610b ,It may be determined through the operation of _{W 2} , W ₃ , W _{4 , W5 , and W6 .}

Similar to FIG. 4B , after steps S520 and S530 , a patch composed of 4x4 pixels may be expanded into an extended patch 630 . The extended patch 630 is a patch after the interpolation process, and may include a plurality of source pixels, a plurality of horizontal interpolation pixels, a plurality of vertical interpolation pixels, a plurality of diagonal interpolation pixels, and a plurality of interpolation pixels. In this case, the method of interpolating each horizontal interpolation pixel, the vertical interpolation pixel, the diagonal interpolation pixel, and the source pixel of FIG. 6CB is performed through the method of interpolating the source pixel described with reference to FIGS. 5, 6A, and 6B. can be

When the pixel values of the plurality of interpolation pixels are determined, the display apparatus 100 finally applies the adaptive filter to determine the pixel values of the central interpolation pixel 640 and converts the extended patch 630 into the central interpolation pixel 640 . You can downscale the image by downscaling. In the present disclosure, the central interpolation pixel 660 may be a geometrically centered pixel among a plurality of interpolation pixels in the extended patch 630 . Accordingly, the position of the central interpolation pixel 640 may be the same as the position of the interpolation pixel having the _{pixel value E 22 .}

Referring back to FIG. 5 , when the pixel values of the plurality of interpolation pixels are determined in steps S520 and S530 , the display apparatus 100 applies the adaptive filter in step S540 to determine the pixel values of the central interpolation pixel 640 . there is.

According to an embodiment of the present disclosure, the pixel value of the central interpolation pixel 640 may be determined by applying an adaptive filter to all pixels in the extended patch 630 , except for the diagonal interpolation pixel. Specifically, the pixel value of the central interpolation pixel 640 may be determined by applying an adaptive filter to all source pixels, horizontal interpolation pixels, vertical interpolation pixels, and interpolation pixels in the extended patch 630 . Since the method of determining the pixel value of the central interpolation pixel by applying the adaptive filter has been described with reference to FIG. 3 , a detailed description thereof will be omitted.

When the pixel value of the central interpolation pixel 640 is determined, in operation S550 , the display apparatus 100 may downscale the image to an image including the central interpolation pixel. Specifically, the display apparatus 100 may downscale the image by downscaling the extended patch 630 to the central interpolation pixel 640 . In order to downscale the input image, a plurality of extended patches may be generated and the plurality of extended patches may partially overlap each other, so that the display apparatus 100 may obtain a downscaled image with a non-integer magnification.

Various embodiments of the present disclosure include one or more instructions stored in a storage medium (eg, memory) readable by a machine (eg, the display device 100 or a computer). It can be implemented as software. For example, the processor (eg, the processor 120 ) of the device may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' is a tangible device and only means that it does not include a signal (eg, electromagnetic wave), and this term means that data is semi-permanently stored in the storage medium. and temporary storage. For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices. It can be distributed (eg downloaded or uploaded) directly or online between devices (eg smartphones). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable app) is stored at least in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.

The disclosed embodiments have been described with reference to the accompanying drawings as described above. Those of ordinary skill in the art to which the present disclosure pertains will understand that the present disclosure may be practiced in other forms than the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

A method for downscaling an image, comprising:

extracting contour information of the source pixel by applying a second-order differential filter in horizontal and vertical directions of the source pixel of the image;

determining pixel values of a plurality of horizontal interpolation pixels and pixel values of a plurality of vertical interpolation pixels based on the pixel values of the source pixels, the outline information, and a downscaling factor;

determining pixel values of a plurality of interpolation pixels based on pixel values of the plurality of horizontal interpolation pixels, pixel values of the plurality of vertical interpolation pixels, and weights;

determining a pixel value of a central interpolation pixel by applying an adaptive filter to the source pixel, the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of interpolation pixels; and

downscaling the image to an image composed of the central interpolated pixels.
The method of claim 1,

The method of claim 1, wherein the outline information is a pixel value change amount with respect to an adjacent source pixel of a current source pixel.
The method of claim 1,

The pixel value of each horizontal interpolation pixel includes outline information of a left source pixel of the horizontal interpolation pixel, a pixel value of the left source pixel, outline information of a right source pixel of the horizontal interpolation pixel, and a pixel value of the right source pixel and a method for downscaling an image, which is determined based on a horizontal downscaling factor.
The method of claim 1,

The pixel value of each vertical interpolation pixel includes outline information of an upper source pixel of the vertical interpolation pixel, a pixel value of the upper source pixel, outline information of a lower source pixel of the vertical interpolation pixel, and a pixel value of the lower source pixel and a method for downscaling an image, which is determined based on a vertical downscaling factor.
The method of claim 1,

The pixel value of each interpolation pixel is the pixel value of the upper horizontal interpolation pixel of the interpolation pixel, the pixel value of the lower horizontal interpolation pixel, the pixel value of the left vertical interpolation pixel of the interpolation pixel, and the pixel value of the right vertical interpolation pixel and a method for downscaling an image, which is determined based on the weight.
The method of claim 1,

The second-order differential filter has a Gaussian-shaped kernel and the sum of the kernel values is 0. The method of downscaling an image.
The method of claim 1,

wherein the weight is determined based on a distance of each interpolated pixel to an adjacent interpolated pixel.
The method of claim 1,

The step of extracting the outline information of the source pixel comprises:

and extracting contour information of the source pixel by applying the second-order differential filter in a diagonal direction of the source pixel,

The step of determining pixel values of the plurality of interpolation pixels includes:

determining pixel values of a plurality of diagonal interpolation pixels based on the pixel values of the source pixels, the outline information, a horizontal downscaling magnification, and a vertical downscaling magnification; and

determining pixel values of the plurality of interpolation pixels based on pixel values of the plurality of horizontal interpolation pixels, pixel values of the plurality of vertical interpolation pixels, and pixel values and weights of the plurality of diagonal interpolation pixels; A method of downscaling an image, including.
9. The method of claim 8,

Among the pixel values of the plurality of diagonal interpolation pixels, a first diagonal interpolation pixel value includes contour information of a left-upper source pixel of the first diagonal interpolation pixel, a pixel value of the left-upper source pixel, and the first diagonal interpolation pixel value. 1 is determined based on contour information of a right-lower source pixel of a diagonal interpolation pixel, a pixel value of the right-lower source pixel, the horizontal downscaling factor, and the vertical downscaling factor,

Among the pixel values of the plurality of diagonal interpolation pixels, a second diagonal interpolation pixel value includes contour information of a right-upper source pixel of the second diagonal interpolation pixel, a pixel value of the right-upper source pixel, and the second diagonal line. A method for downscaling an image, which is determined based on contour information of a left-bottom source pixel of a directional interpolation pixel, a pixel value of the left-bottom source pixel, the horizontal downscaling factor, and the vertical downscaling factor.
9. The method of claim 8,

wherein the weight is determined based on a distance of each interpolation pixel to an adjacent interpolated pixel and a distance to an adjacent source pixel.
The method of claim 1,

The method of downscaling an image, wherein the adaptive filter is determined through a convolution operation of a bilateral filter and a range filter.
The method of claim 1,

The central interpolation pixel is a geometrically centered pixel among the plurality of interpolation pixels,

The downscaling includes downscaling a 7x7 patch composed of the source pixel, the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of interpolation pixels to one pixel composed of the central interpolation pixel. A method for downscaling an image, comprising the steps of:
A display device for downscaling an image, comprising:

an outline information extraction unit for extracting outline information of the source pixel by applying a second-order differential filter in horizontal and vertical directions of the source pixel of the image;

determine pixel values of a plurality of horizontal interpolation pixels and pixel values of a plurality of vertical interpolation pixels based on the pixel values of the source pixels, the outline information, and a downscaling factor;

an interpolator configured to determine pixel values of a plurality of interpolation pixels based on pixel values of the plurality of horizontal interpolation pixels, pixel values of the plurality of vertical interpolation pixels, and weights; and

a pixel value of a central interpolation pixel is determined by applying an adaptive filter to the source pixel, the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of interpolation pixels, and the image is converted to the central interpolation pixel A display device for downscaling an image, comprising a downscaling unit for downscaling the configured image.
14. The method of claim 13,

wherein the outline information is a pixel value change amount with respect to an adjacent source pixel of a current source pixel.
A computer-readable recording medium storing instructions for downscaling an image, comprising:

extracting contour information of the source pixel by applying a second-order differential filter in horizontal and vertical directions of the source pixel of the image;

determining pixel values of a plurality of horizontal interpolation pixels and pixel values of a plurality of vertical interpolation pixels based on the pixel values of the source pixels, the outline information, and a downscaling factor;

determining pixel values of a plurality of interpolation pixels based on pixel values of the plurality of horizontal interpolation pixels, pixel values of the plurality of vertical interpolation pixels, and weights;

determining a pixel value of a central interpolation pixel by applying an adaptive filter to the source pixel, the plurality of horizontal interpolation pixels, the plurality of vertical interpolation pixels, and the plurality of interpolation pixels; and

A computer-readable recording medium having stored thereon instructions for performing an operation of downscaling the image to an image composed of the central interpolation pixel.