KR102340942B1 - Method for Image Processing and Display Device using the same - Google Patents

Abstract

The present invention provides a display device including a display panel, a driving unit, and a control unit. The display panel displays an image. The driver drives the display panel. The control unit controls the driving unit and image-processes an image input from the outside. The control unit calculates the first grid size corresponding to each point of the input image according to the target resolution in consideration of the salience value corresponding to a visually important part in the input image, and considers the second grid size in consideration of temporal consistency , calculates the final grid size based on the first and second grid sizes, generates a grid map, and expresses and generates a reassigned image to be finally output based on the grid map and the input image.

Description

Image processing method and display device using same

The present invention relates to an image processing method and a display device using the same.

Recently, Light Emitting Display (LED), Liquid Crystal Display (LCD), Field Emission Display (FED), Electrophoretic Display (EPD), Electrowetting The use of display devices such as an electro-wetting display (EWD) and a quantum dot display (QD) is increasing.

The display device described above may be implemented as a TV, an image player, a personal computer (PC), a home theater, a smart phone, an augmented/virtual reality device, and the like. The display device is implemented in various forms as in the previous example and has aspect ratios of various sizes.

Although display devices are produced in various forms as described above, video content is actually created to fit the aspect ratio of a specific target. For example, since image content is standardized as a form of an image format suitable for a specific target such as a television, when implemented in a smartphone, an aspect ratio difference occurs compared to the original content.

For this reason, when an image of an arbitrary size is input, the specific display device needs to perform image processing such as resizing the image while maintaining the original shape in order to realize an image suitable for the environment of the device. When changing the size of an image, it is necessary to not only retarget the aspect ratio of the input image, but also to prevent distortion in order to keep the original shape as natural as possible.

The present invention for solving the problems of the above-mentioned background art is to prevent structural twist when resizing an image in which a moving object and a static background coexist, while maintaining temporal consistency.

As a means for solving the above problems, the present invention provides a display device including a display panel, a driving unit, and a control unit. The display panel displays an image. The driver drives the display panel. The control unit controls the driving unit and image-processes an image input from the outside. The control unit calculates the first grid size corresponding to each point of the input image according to the target resolution in consideration of the salience value corresponding to a visually important part in the input image, and considers the second grid size in consideration of temporal consistency , calculates the final grid size based on the first and second grid sizes, generates a grid map, and expresses and generates a reassigned image to be finally output based on the grid map and the input image.

The controller may subtract the position of the grid point of the current frame from the position of the grid point of the previous frame in order to calculate the second grid size in consideration of temporal consistency.

The controller may calculate the final grid size for each axis by considering the ratio of the static background to the moving object of the first grid size and the second grid size.

In order to calculate the final grid size, the controller may assign a weight to a pixel having a large difference between a luminance value of a previous frame and a luminance value of the current frame (a large change in image).

In the control unit, the second grid size is calculated by the following formula,

In the above formula, Pos ^k _prev means the position of the k-th axis (time axis) in the grid map of the previous frame, Pos ^{k - 1} _cur means the position of the k-th axis in the grid map of the current frame, and GS _T ^k is A grid size in consideration of temporal coherence ( _T ) on the k-th axis may be represented.

In another aspect, the present invention provides an image processing method. The image processing method includes estimating a saliency value corresponding to a visually important part in an input image; calculating a first grid size corresponding to each point of the input image according to a target resolution in consideration of the saliency value and calculating a second grid size in consideration of temporal consistency; generating a grid map after calculating a final grid size based on the first and second grid sizes; and expressing and generating a redirected image to be finally output based on the grid map and the input image.

The calculating of the grid size may include subtracting the position of the grid point of the current frame from the position of the grid point of the previous frame in order to calculate the second grid size in consideration of temporal consistency.

In the calculating of the final grid size, the final grid size may be calculated by linearly combining the calculated first and second grid size values in consideration of the ratio of the moving object and the static background on each axis.

In the step of calculating the final grid size, a weight may be given to pixels having a large difference between the luminance value of the previous frame and the luminance value of the current frame (image change is large).

The second grid size is calculated by the following formula,

In the above formula, Pos ^k _prev means the position of the k-th axis (time axis) in the grid map of the previous frame, Pos ^{k - 1} _cur means the position of the k-th axis in the grid map of the current frame, and GS _T ^k is A grid size in consideration of temporal coherence ( _T ) on the k-th axis may be represented.

The present invention has the effect of providing an image processing method capable of maintaining temporal consistency while preventing a structural twist phenomenon when resizing an image. In addition, the present invention is an image processing method in consideration of temporal coherence when resizing an image in which a moving object and a static background coexist, while maintaining the original shape as natural as possible while minimizing the noise of the image due to the movement of the object. And there is an effect of providing a display device capable of displaying a high-quality image.

1 is a schematic block diagram of a display device;
2 is a schematic circuit configuration diagram of a sub-pixel;
3 is a cross-sectional view of a display panel;
4 is a diagram illustrating an example of changing the size of an image according to an image processing method;
5 is a view for explaining an image reassignment method according to an experimental example;
6 is a view for explaining an image reassignment method according to an embodiment of the present invention.
Fig. 7 is a view showing the canon stage of Fig. 6 in detail;
8 and 9 are first exemplary views showing the results of comparative experiments between the Experimental Example and the Example in order to verify the effect of a method considering time consistency according to the Example.
10 and 11 are second exemplary views showing the results of comparative experiments between the Experimental Example and the Example in order to verify the effect of a method considering time consistency according to the Example.
12 is a diagram illustrating in an easy-to-understand manner the aspect of a change appearing in the final output image according to the method of the experimental example and the method of the embodiment.

Hereinafter, specific details for carrying out the present invention will be described with reference to the accompanying drawings.

In the present invention, the display device is a light emitting display (LED), a liquid crystal display (LCD), a field emission display (FED), an electrophoretic display (Electro Phoretic Display: EPD), an electro-wetting display (EWD), and a quantum dot display (Quantum Dot display: QD) may be selected from among display devices such as, but not limited thereto.

1 is a schematic block diagram of a display device, FIG. 2 is a schematic circuit configuration diagram of a sub-pixel, FIG. 3 is a cross-sectional diagram of a display panel, and FIG. 4 is an example of changing the size of an image according to an image processing method is a diagram showing

1 , the display device according to an embodiment of the present invention includes an image processor 110 , a timing controller 120 , a data driver 130 , a scan driver 140 , and a display panel 150 . . The image processing unit 110 and the timing control unit 120 may be integrated in the form of an integrated circuit (IC) and implemented as one control unit, but is not limited thereto.

The image processing unit 110 performs image processing on the image data signal DATA supplied from the outside, and outputs a data enable signal DE and the like along with the image-processed data signal DATA. The image processing unit 110 may output one or more of a vertical synchronization signal, a horizontal synchronization signal, and a clock signal in addition to the data enable signal DE, but these signals are omitted for convenience of description.

The timing controller 120 receives the data signal DATA from the image processing unit 110 as well as a driving signal including a data enable signal DE or a vertical synchronization signal, a horizontal synchronization signal, and a clock signal. The timing controller 120 includes a gate timing control signal GDC for controlling the operation timing of the scan driver 140 and a data timing control signal DDC for controlling the operation timing of the data driver 130 based on the driving signal. to output

The data driver 130 samples and latches the data signal DATA supplied from the timing controller 120 in response to the data timing control signal DDC supplied from the timing controller 120 , converts it into a gamma reference voltage, and outputs it . The data driver 130 outputs the data signal DATA through the data lines DL1 to DLn. The data driver 130 may be formed in the form of an IC.

The scan driver 140 outputs a scan signal while shifting the level of the gate voltage in response to the gate timing control signal GDC supplied from the timing controller 120 . The scan driver 140 outputs a scan signal through the scan lines GL1 to GLm. The scan driver 140 is formed in the form of an IC or is formed in the display panel 150 in a gate-in-panel method.

The display panel 150 displays an image in response to the data signal DATA and the scan signal supplied from the data driver 130 and the scan driver 140 . The display panel 150 includes sub-pixels SP that operate to display an image. The sub-pixels SP include a red sub-pixel, a green sub-pixel, and a blue sub-pixel, or include a white sub-pixel, a red sub-pixel, a green sub-pixel, and a blue sub-pixel. The display panel 150 may be classified into a liquid crystal display panel, an electroluminescent display panel, an electrophoretic display panel, etc. according to the configuration of the pixel circuit PC included in the sub-pixels SP.

As shown in FIG. 2 , one sub-pixel SP is supplied in response to a scan signal supplied through the switching transistor SW connected to the scan line GL1 and the data line DL1 and the switching transistor SW. A pixel circuit (PC) operating in response to the data signal is included. For example, when the display panel is selected as the electroluminescent display panel, the pixel circuit PC of the sub-pixel SP may be configured to further include various types of compensation circuits in addition to a driving transistor, a storage capacitor, and an organic light emitting diode.

As shown in FIG. 3 , the display panel 150 includes a substrate (or thin film transistor substrate) 150a having a display area AA and a non-display area NA and a protective film (or protective substrate) 150b. include A pixel P is formed in the display area AA based on the circuit illustrated in FIG. 2 . The pixel P disposed on the display area AA includes red (R), white (W), blue (B), and green (G) sub-pixels. The red (R), white (W), blue (B), and green (G) sub-pixels may be horizontally or vertically disposed on the substrate 150a. However, the arrangement order of the sub-pixels described above is only an example and may be variously changed according to an implementation method of the display panel 150 . However, the pixel P may include only red (R), blue (B), and green (G) sub-pixels.

The display device described above may be implemented as a TV, an image player, a personal computer (PC), a home theater, a smart phone, an augmented/virtual reality device, and the like. The display device is implemented in various forms as in the previous example and has aspect ratios of various sizes.

Although display devices are produced in various forms as described above, video content is actually created to fit the aspect ratio of a specific target. For example, image content is standardized in a form of an image format suitable for a specific target such as a television, so that when implemented in a home theater, smart phone, augmented/virtual reality device, etc., an aspect ratio difference occurs compared to the original content.

For this reason, when an image of an arbitrary size is input, the specific display device needs to perform image processing such as resizing the image while maintaining the original shape in order to realize an image suitable for the environment of the device. The image processing may be performed by the image processing unit, the timing control unit, or the image processing unit and the timing control unit, but is not limited thereto.

As in the first example of Fig. 4 (a), the input image input in the size of horizontal (16) * vertical (9) (Video input) is horizontal (18) * vertical (9) as an output image (Video output) The size of the image may be changed. As in the second example of FIG. 4 (b), the input image input in the size of horizontal (18) * vertical (9) is a horizontal (16) * vertical (9) output image (Video output) The size of the image may be changed.

4, as shown in (a) and (b), changing the size of only one part (right side as in the example) of the video input is an example, but this is only an example and corresponds to at least one part should be understood as For example, the size of one of the top, bottom, left, and right of the input image may be changed, or the size of the input image may be changed up and down, left and right, or up and down, left and right.

As such, when changing the size of an image, it is necessary to not only retarget the aspect ratio of the input image, but also to prevent distortion in order to keep the original shape as natural as possible.

Hereinafter, after selecting one video retargeting method according to the experimental example and studying and considering its problems, an embodiment in which problems predicted and occurring from the experimental example are improved will be described.

<Experimental example>

5 is a diagram for explaining an image reassignment method according to an experimental example.

As shown in Fig. 5, the image reassignment method according to the experimental example includes an image input step (S110), a two-dimensional prominence estimation step for the input image (S120), a scene step (S130), an expression step (S140), and and outputting the redirected image ( S150 ).

The video input step S110 is a step of receiving an arbitrary video input from the outside.

The 2D saliency estimation step S120 is a step of estimating a visually important part (which may be defined as a focus of an eye or an area of concentration of interest, etc.) in the input image. The two-dimensional salience is divided into horizontal and vertical one-dimensional salience profiles to determine the importance of each row and column and protrude.

The two-dimensional saliency estimation step S120 includes a downsampling step S121, a quaternion Fourier transform step S122, and a Gaussian filtering step S123 to use a low-level saliency map. .

The down sampling step S121 is a step of extracting less data from the input image. The downsampling step can be defined as a preprocessing step to use a low-level saliency map. Downsampling can reduce the complexity that may occur in all calculations performed in subsequent steps.

The quaternion fourier transform saliency estimation (QFT saliency) step S122 is a step of obtaining a phase spectrum of an image sequence and obtaining an estimation result therefrom so as to estimate a visually salient part within the down-sampled image.

For example, RGB video information having motion is transformed into a quaternion representation (a complex number having four components) and then transformed into a frequency space according to a quaternion Fourier transform (hereinafter, QFT). Phase information in the frequency domain is extracted by normalizing each quaternion to a magnitude. As a result, prominent areas are gently blurred.

The Gaussian filtering step S123 is a step of removing high-frequency noise from a prominent portion obtained as a result of the estimation. In the experimental example, the Gaussian filtering method is used as an example to remove high-frequency noise, but the present invention is not limited thereto. Also, the filtering step may be omitted in some cases.

The warping step (S130) is a step of calculating a grid size corresponding to each point of the input image according to the target resolution in consideration of the previously obtained saliency value, and then generating a grid map to be.

In the canon step (S130), a one-dimensional prominence calculation step (S131), a non-linear one-dimensional scale step (S132), and a filtering step (S133) are performed to calculate the grid size for each axis and to give a grid size of the same size to them. , including an up-sampling step (S134) and a grid generation step (S135).

The 1D saliency calculation step S131 is a step of calculating the grid size for each axis for each of the horizontal and vertical 1D saliency profiles. For example, when calculating the 1D saliency, the saliency value may be calculated based on pixels of a predetermined unit (accumulated calculation in block units) in a manner that blocks predetermined areas of the horizontal 1D saliency profile and the vertical 1D salience profile, respectively, but limited to this. doesn't happen

Non-linear 1D scale step S132 is a step of calculating a non-linear scale vector for both the vertical and horizontal directions in order to maintain the aspect ratio of an important area, such as a visually prominent part. For example, when a scale vector is calculated, when a visually low or no important area is reduced, a scaling value lower than 1 may be obtained, and vice versa, a scaling value higher than 1 may have a scaling value higher than 1. In this case, the sum vector of all elements of the scaling must match the target size required to consider video retargeting.

The filtering step ( S133 ) is a step of directly performing normalization by applying a post filter after calculating the nonlinear one-dimensional scale. Since the input image undergoes high-frequency and low-frequency fluctuations within the prominent part through the process up to the nonlinear one-dimensional scale calculation, spatiotemporal fluctuations in the output image can be brought about. Therefore, if normalization is directly performed by applying a post filter, the problem caused by spatiotemporal fluctuations can be solved.

For example, in the filtering step S133, a non-causal finite impulse response (aFIR) filter may be used to remove the high frequency component, and an infinite impulse response (IIR) filter may be used to remove the low frequency component. may be, but is not limited thereto.

The up-sampling step S134 is a step of extracting more data from less data in the input image. In order to achieve the actual target resolution based on the sampling grid prepared through the above process, it is necessary to linearly upsample the horizontal and vertical scaling vectors and then generate a sampling grid again, so this step is performed.

In the grid generation step (S135), the grid size corresponding to each point of the input image is calculated according to the target resolution by considering the saliency value based on the re-prepared sampling grid, and then the grid map ( This is the step to create a grid map).

The rendering step ( S140 ) is a step of expressing and generating a retarget image to be finally output using the grid map and the input image generated through the previous step. The expression step S140 may create a retargeted image based on a framework of an elliptical weighted average (EWA splatting) method and the like. In this case, the EWA framework may use forward mapping and may resample an image under the maximum resolution input stream.

The retarget video output step S150 is a step of outputting a retargeted image whose aspect ratio is redefined with respect to the input image.

The experimental example calculates the importance of each area of the image, calculates the grid size of each point of the input image according to the target resolution by taking this into account, generates a grid map, and uses this to calculate the grid size It is a video retargeting method that expresses and generates

In particular, the experimental example calculates the grid size for each axis and gives the grid size of the same size to prevent distortion such as structure twist in which an object on the same axis in the input image is on a different axis in the output image. It is considered to have potential advantages.

However, the experimental example is considered to be difficult to apply to an image in which a moving object exists. For example, if priority is given to maintaining the grid size of a moving object, the grid size of the static background changes according to the movement of the object. As another example, if priority is given to maintaining the grid size of the static background, the grid size changes according to the movement of the object. As such, in the experimental example, since a moving object and a static background exist on one axis at the same time, if one is given priority, the other will be affected, and this is because the sound of the image is caused.

<Example>

6 is a diagram for explaining an image reassignment method according to an embodiment of the present invention, and FIG. 7 is a diagram specifically illustrating the scene stage of FIG. 6 .

As shown in FIG. 6 , the image reassignment method according to an embodiment of the present invention includes an image input step (S210), a two-dimensional prominence estimation step (S220) for the input image, a scene step (S230), an expression step ( S240) and a redirected image output step (S250).

The video input step S210 is a step of receiving an arbitrary video input from the outside.

The 2D saliency estimation step S220 is a step of estimating a visually important part (which may be defined as an eye focus or an area of concentration of interest, etc.) in the input image. The two-dimensional salience is divided into horizontal and vertical one-dimensional salience profiles to determine the importance of each row and column and protrude.

The two-dimensional saliency estimation step S220 includes a down-sampling step S221, a quaternion Fourier transform step S222, and a Gaussian filtering step S223 to use a low-level saliency map. .

The down sampling step S221 is a step of extracting less data from the input image. The downsampling step can be defined as a preprocessing step to use a low-level saliency map. Downsampling can reduce the complexity that may occur in all calculations performed in subsequent steps.

The quaternion fourier transform saliency estimation (QFT saliency) step S222 is a step of obtaining a phase spectrum of an image sequence and obtaining an estimation result therefrom so as to estimate a visually salient part in the down-sampled image.

For example, RGB video information having motion is transformed into a quaternion representation (a complex number having four components) and then transformed into a frequency space according to a quaternion Fourier transform (hereinafter, QFT). Phase information in the frequency domain is extracted by normalizing each quaternion to a magnitude. As a result, prominent areas are gently blurred.

The Gaussian filtering step S223 is a step of removing high-frequency noise from a prominent portion obtained as a result of the estimation. In the embodiment, the Gaussian filtering method is used as an example to remove high-frequency noise, but the present invention is not limited thereto. Also, the filtering step may be omitted in some cases.

In the warping step (S230), the first grid size corresponding to each point of the input image is calculated according to the target resolution in consideration of the previously obtained saliency value, and the second grid size in consideration of temporal consistency , and a final grid size is calculated based on the first and second grid sizes, and then a grid map is generated.

6 and 7, in the canon step S230, the first grid size value and the second grid size value calculated in consideration of the ratio of the moving object and the static background on each axis are linearly combined. In order to calculate the final grid size as , a first grid calculation step (S239), a second grid calculation step (S236), a final grid calculation step (S237), and a grid map generation step (S238) are included.

The first grid calculation step S239 is a step of calculating a grid size corresponding to each point of the input image according to the target resolution in consideration of the previously obtained saliency value. The higher the saliency value, the more similar the grid size to the original. In general, moving objects have a high saliency value.

The first grid calculation step (S239) is a one-dimensional prominence calculation step (S231), a non-linear one-dimensional scale step (S232), a filtering step in order to calculate the grid size for each axis and give a grid size of the same size to them. (S233) and an up-sampling step (S234).

The 1D saliency calculation step S231 is a step of calculating the first grid size for each axis for each of the horizontal and vertical 1D saliency profiles. For example, when calculating the 1D saliency, the saliency value may be calculated based on pixels of a predetermined unit (accumulated calculation in block units) in a manner that blocks predetermined areas of the horizontal 1D saliency profile and the vertical 1D salience profile, respectively, but limited to this. doesn't happen

The non-linear 1D scale step S232 is a step of calculating a non-linear scale vector in both the vertical direction and the horizontal direction in order to maintain the aspect ratio of an important area, such as a visually prominent part. When calculating the scale vector, when a visually low or absent important area is reduced, it may have a scaling value lower than 1, and vice versa, it may have a scaling value higher than 1 when it is enlarged. In this case, the sum vector of all elements of the scaling must match the target size required to consider video retargeting. The nonlinear one-dimensional scale may be calculated by the following equation.

In the above formula, GS _s ^Down (k) represents the grid size on the k- th axis, S _h represents the salience value,

W' _down means 1/2 of the width of the target resolution, R _max means the maximum value of the grid size, and R _min means the minimum value of the grid size.

The filtering step ( S233 ) is a step of directly performing normalization by applying a post filter after calculating the nonlinear one-dimensional scale. Since the input image undergoes high-frequency and low-frequency fluctuations within the prominent part through the process up to the nonlinear one-dimensional scale calculation, spatiotemporal fluctuations in the output image can be brought about. Therefore, if normalization is directly performed by applying a post filter, the problem caused by spatiotemporal fluctuations can be solved.

For example, in the filtering step S233, a non-causal finite impulse response (aFIR) filter may be used to remove the high frequency component, and an infinite impulse response (IIR) filter may be used to remove the low frequency component. may be, but is not limited thereto.

The up-sampling step S234 is a step of extracting more data from less data in the input image. In order to achieve the actual target resolution based on the sampling grid prepared through the above process, it is necessary to linearly upsample the horizontal and vertical scaling vectors and then generate a sampling grid again, so this step is performed.

The second grid calculation step S236 is a step of calculating the grid size so that the positions of the grid points of the previous frame and the grid points of the current frame are the same in order to calculate the grid size in consideration of temporal consistency. In this step, the grid point of the previous frame and the grid point of the current frame are to find a convergence value that is the same as or close to the maximum possible. In this case, the grid size for each axis may be calculated by the following equation.

In the above formula, Pos ^k _prev means the position of the k-th axis (time axis) in the grid map of the previous frame, Pos ^{k -1} _cur means the position of the k-th axis in the grid map of the current frame, and GS _T ^k is It represents the grid size considering the temporal coherence ( _T ) on the k-th axis. Due to the calculation of the grid size based on this formula, the position of the grid point of the static background is maintained, thereby preventing the background from rumble in the image.

The final grid size calculation step S237 is a step of calculating the final grid size for each axis by considering the ratio of the static background and the moving object of each axis to the previously obtained first and second grid sizes. In this case, the final grid size may be calculated by the following equation.

In the above formula, N ^k _{dist _pixel} is the number of pixels on the k-th axis where the difference between the luminance value of the previous frame and the luminance value of the current frame is large (large image change), and N ^k _tot represents the number of pixels on the k-th axis. . W _S ^k and W _T ^k represent weights for GS _S ^k and GS _T ^k on the k- th axis, respectively. GS ^k represents the final grid size on the k-th axis.

In the above formula, the number of pixels with a large difference between the luminance value of the previous frame and the current frame may be set to, for example, “the number of pixels greater than or equal to 20” as an example, but this is only an example and the number of pixels is an experimental result can be optimized based on

On the other hand, a change in the grid size in a static background is easier to recognize than a change in the grid size of a moving object. Therefore, in the embodiment, a boundary value for the change in the grid size is set in order to maintain the temporal consistency of the static background well. And when the set boundary value is exceeded, the grid size is reset. In this case, the reset of the grid size may be calculated by the following equation.

In the above formula, GS _T ^k and GS ^k represent the grid size and final grid size considering temporal consistency on the k- th axis, respectively, and λ is 0.1 for cutting when the final grid size is too large or smaller than the set boundary value. can be set, but this can be optimized based on the experimental results.

The grid map generating step S238 is a step of generating a grid map by accumulating the grid size of each grid point. In this case, the grid map may be calculated by the following equation.

In the above equation, Pos ^k _prev , Pos ^{k - 1} _cur denote the position of the k- th axis in the grid map of the previous frame and the current frame, respectively , and GS ^k denotes the final grid size.

The rendering step S240 is a step of expressing and generating a retarget image to be finally output using the grid map and the input image generated through the previous step. The expression step S240 may create a retargeted image based on a framework of an elliptical weighted average (EWA splatting) method or the like. In this case, the EWA framework may use forward mapping and may resample an image under the maximum resolution input stream.

The retarget video output step S250 is a step of outputting a retargeted video whose aspect ratio is re-specified with respect to the input video.

As described above, in the embodiment of the present invention, since the same grid size is given to the same axis, it is possible to prevent distortion such as a structural twist in which an object on the same axis in the input image exists on a different axis in the output image. have. In addition, since the embodiment of the present invention calculates the grid size in one dimension, the amount of calculation is small. In addition, since the embodiment of the present invention considers the ratio of the static background and the moving object of each axis, it is possible to maintain the temporal consistency of the object with high specific gravity.

In order to verify the effect of the method considering time consistency according to the embodiment of the present invention, the results of comparative experiments between the experimental examples and the examples will be described as follows.

<Example 1>

8 and 9 are first exemplary views showing results of comparative experiments between the Experimental Example and the Example in order to verify the effect of the method considering time consistency according to the Example.

In the first example, a 16:9 image with a horizontal and vertical size was converted into an 18:9 image using the method of the embodiment and the method of the experimental example. At this time, the temporal consistency of the method of the example was evaluated. For the experimental example, the Greisen [1] method described above was used.

And, as input images, 800 frames of "Discovery of Romance" and 500 frames of "Maritel (My Little Television)" were used as television programs with a resolution of 1280*720. In the first example, Jittery Metric (JM) [2] was used as an evaluation index to judge temporal consistency. Jittery Metric (JM) can be calculated by the following formula.

In the above formula, S _k,j ^cols represents the grid size of the k- th axis of the current frame , S _k,j-1 ^cols represents the grid size of the k- th axis of the previous frame , and DS _k,j ^cols represents the current frame and previous frame. It represents the difference in grid size on the k- th axis of the frame. N represents the width of the input image, and JM _j represents the jittery metric (JM) in the horizontal direction for the current frame and the previous frame. In this case, it can be said that the lower the DS and JM values, the better the temporal consistency is maintained.

The Greisen method [1] corresponding to the experimental example was proposed by P. Greisen, and was registered in ACM SIGGRAPH in 2012 like “Algorithm and VLSI architecture for real-time 1080p60 video retargeting,”.

Jittery Metric (JM)[2], which is an evaluation index, was proposed by Bo Yan, and has been registered in IEEE Transactions on multimedia in 2014 like “Effective video retargeting with jittery assessment,”.

In the graph of FIG. 8 , a horizontal axis is a column index, and a vertical axis is a grid size DS. In the graph of FIG. 9 , the horizontal axis is the frame number, and the vertical axis is the jittery metric (JM).

In the first example, the grid size distribution appearing in some frames of the same image is extracted for comparison between the experimental example and the example and shown as a graph in FIG. 8, so that the difference in temporal consistency between the experimental example and the example can be determined The Jittery Metric (JM) appearing in two television programs is excerpted and shown as a graph in FIG. 9 .

As can be seen from the experimental results shown in FIGS. 8 and 9 , it can be confirmed that the DS and JM values of the Example method (Proposed method) are generally lower than those of the Example method (Greisen's method).

For example, according to the method of the experimental example (Greisen's method), as shown in the 6th frame (<Frame #6>), the ninth frame (<Frame #9>), and the 12th frame (<Frame #12>) of FIG. Similarly, it can be seen that a part (PT) out of temporal coherence occurs in a specific image. However, the method of the embodiment (Proposed method) can maintain temporal consistency even in the same frame as the experimental example, it can be seen that a part such as "PT" does not occur.

<2nd example>

10 and 11 are second exemplary views showing results of comparative experiments between the Experimental Example and the Example in order to verify the effect of a method considering time consistency according to the Example.

In Example 2, the experiment was performed under the same conditions as in Example 1, but the evaluation index was changed to Jittery Metric-II (JM - II). This evaluation index is proposed considering the location of the grid point, not the size of the grid. Jittery Metric-II (JM - II) can be calculated by the following formula.

In the above formula, P _k,j ^cols indicates the position of the grid point on the k- th axis of the current frame , P _{k,j - 1} ^cols indicates the position of the grid point on the k- th axis of the previous frame , and PD _k,j ^cols is It represents the difference between the positions of the grid points on the k- th axis of the current frame and the previous frame. N represents the width of the input image, and JM - II _j represents the jittery metric (JM-II) in the horizontal direction for the current frame and the previous frame. In this case, it can be said that the lower the grid point position and the JM-II value, the better the temporal consistency is maintained in a static background.

In the graph of FIG. 10 , the horizontal axis is the pixel number, and the vertical axis is the position PD of the grid point. In the graph of FIG. 11 , the horizontal axis is frame number, and the vertical axis is Jittery Metric (JM-II).

The second example is a graph of FIG. 10 by extracting the location distribution of grid points appearing in some frames of the same image for comparison between the experimental example and the example, and the difference in temporal consistency between the experimental example and the example can be determined. Jittery Metric (JM-II) appearing in two television programs is extracted and shown as a graph in FIG.

As can be seen from the experimental results shown in FIGS. 10 and 11, the position value of the grid point is a value calculated by accumulating the grid size, so JM- It can be seen that the difference in II values is large.

For example, according to the method of the experimental example (Greisen's method), as shown in the 6th frame (<Frame #6>), the ninth frame (<Frame #9>), and the 12th frame (<Frame #12>) of FIG. Similarly, it can be seen that a part (PT) out of temporal coherence occurs in a specific image. However, the method of the embodiment (Proposed method) can maintain temporal consistency even in the same frame as the experimental example, it can be seen that a part such as "PT" does not occur.

When the position (PD) value of the grid point in the HD image has a value of 0.1 or more, the static background may be felt. As can be seen from FIG. 10, the Example method (Proposed method) has a grid point position value lower than 0.1, whereas the Experimental example method (Greisen's method) has a grid point position value much higher than this. Therefore, the Proposed method can prevent the static background from bouncing.

Hereinafter, when an image in which a moving image corresponding to a visually prominent part adjacent to a still image is different for each frame is processed by the method of the experimental example and the method of the embodiment, the change pattern appearing in the final image is easily illustrated as follows.

12 is a diagram illustrating in an easy-to-understand manner the aspect of a change appearing in the final output image according to the method of the experimental example and the method of the embodiment. In FIG. 12 , an ellipse having a solid pattern on the left corresponds to a still image (eg, a subtitle or a background, etc.), and an oval containing a diagonal pattern on the right corresponds to a moving image (eg, a moving object or person).

Ideally, even if a moving picture exists in a specific time period such as “t+1 and t+3” in a still image, the temporal position should remain the same as in “t and t+2” where the moving image did not exist. However, as found through the experimental example, when the size of the input image is changed, the temporal position is changed.

In the experimental example shown in (a) of FIG. 12 , it can be seen that when a moving image is displayed differently for each frame adjacent to a still image, as can be seen from position 5, the temporal position variation width is large.

On the other hand, in the embodiment shown in FIG. 12 (b), when a moving image is displayed differently for each frame adjacent to a still image, although the temporal position fluctuation width occurs at the 5th position as in the experimental example, the temporal position fluctuation width is higher than that of the experimental example. It can be seen that small

As described above, the present invention has an effect of providing an image processing method capable of maintaining temporal consistency while preventing a structural twist phenomenon when resizing an image. In addition, the present invention is an image processing method in consideration of temporal coherence when resizing an image in which a moving object and a static background coexist, while maintaining the original shape as natural as possible while minimizing the noise of the image due to the movement of the object. And there is an effect of providing a display device capable of displaying a high-quality image.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the technical configuration of the present invention can be changed to other specific forms by those skilled in the art to which the present invention pertains without changing the technical spirit or essential features of the present invention. It will be appreciated that this may be practiced. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. In addition, the scope of the present invention is indicated by the claims to be described later rather than the above detailed description. In addition, all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention.

110: image processing unit 120: timing control unit
130: data driver 140: scan driver
150: display panel SP: sub-pixel(s)
S110: image input step S120: two-dimensional prominence estimation step
S130: canon stage S140: expression stage
S150: Redirected video output stage

Claims (10)

a display panel for displaying an image;
a driver driving the display panel; and
A control unit for controlling the driving unit and processing an image input from the outside,
The controller calculates a first grid size corresponding to each point of the input image according to a target resolution in consideration of a saliency value corresponding to a visually important part in the input image, and also calculates a second grid size corresponding to each point of the input image in consideration of temporal consistency. After calculating the grid size, calculating the final grid size based on the first and second grid sizes, generating a grid map, and expressing a redesignated image to be finally output based on the grid map and the input image, and the display it creates. According to claim 1,
the control unit
A display device for subtracting a position of a grid point of a current frame from a position of a grid point of a previous frame in order to calculate the size of the second grid in consideration of the temporal consistency. According to claim 1,
the control unit
A display device for calculating a final grid size for each axis in consideration of a ratio of a static background to a moving object of the first grid size and the second grid size. 4. The method of claim 3,
the control unit
In order to calculate the final grid size, a weight is given to pixels having a large difference between a luminance value of a previous frame and a luminance value of a current frame (image change is large). According to claim 1,
the control unit
The second grid size is calculated by the following formula,

In the above formula, Pos ^k _prev means the position of the k-th axis (time axis) in the grid map of the previous frame, Pos ^{k - 1} _cur means the position of the k-th axis in the grid map of the current frame, and GS _T ^k is A display indicating the grid size considering the temporal coherence ( _T ) on the k-th axis. estimating a saliency value corresponding to a visually important part in the input image;
calculating a first grid size corresponding to each point of the input image according to a target resolution in consideration of the saliency value and calculating a second grid size in consideration of temporal consistency;
generating a grid map after calculating a final grid size based on the first and second grid sizes; and
and expressing and generating a redirected image to be finally output based on the grid map and the input image. 7. The method of claim 6,
The step of calculating the grid size is
An image processing method of subtracting a position of a grid point of a current frame from a position of a grid point of a previous frame in order to calculate the size of the second grid in consideration of the temporal consistency. 7. The method of claim 6,
The step of calculating the final grid size is
An image processing method of calculating the final grid size by linearly combining the first grid size value and the second grid size value calculated in consideration of the ratio of the moving object and the static background on each axis. 9. The method of claim 8,
The step of calculating the final grid size is
An image processing method in which a weight is given to pixels with a large difference between the luminance value of the previous frame and the luminance value of the current frame. 7. The method of claim 6,
The second grid size is calculated by the following formula,

In the above formula, Pos ^k _prev means the position of the k-th axis (time axis) in the grid map of the previous frame, Pos ^{k - 1} _cur means the position of the k-th axis in the grid map of the current frame, and GS _T ^k is An image processing method representing the grid size considering temporal coherence ( _T ) on the k-th axis.

Images