KR102430216B1 - Video Processing Apparatus, Video Receiving Terminal And Operating Method Of Thereof - Google Patents

Abstract

The embodiment relates to an image processing apparatus, an image receiving terminal, and an operating method thereof. A method of operating an image processing apparatus includes: dividing a frame of an original image into odd-numbered line frames and even-numbered line frames; selecting any one of the odd-numbered line frame and the even-numbered line frame; encoding the selected frame in a progressive manner; and transmitting the encoded frame to an image receiving terminal.

Description

Video Processing Apparatus, Video Receiving Terminal And Operating Method Of Thereof

The embodiment relates to an operating method of an image processing apparatus and an image receiving terminal.

There are two major types of video compression: a progressive method and an interlace method. In the progressive method, one frame of an image is sequentially scanned from (0,0) pixels and then compressed. A method in which the odd-numbered line frame and the even-numbered line frame are cross-outputted after the decoder deinterlaces the odd-numbered line frame and the even-numbered line frame after dividing it into an odd line frame and an even line frame and compressing it, respectively to be.

The advantage of the interlace method compared to the progressive method among existing video compression methods is that the same frame rate can be provided at half the bit rate, but the delay time increases due to deinterlace processing and the progressive method Contrast image quality degradation occurs.

In order to process interlaced encoding/decoding in real time, the codec must support it, but recent hardware codecs only support the progressive method, so the interlace method cannot be used, or if a software codec is used, decoding time increases due to deinterlacing processing and When there is a lot of movement in the screen, the image quality deteriorated.

The related prior art is as follows.

Korean Patent Registration No. 10-0974397 "Scalable encoding and decoding of interlaced digital video data"

Korean Patent No. 10-1049258 "Method and apparatus for encoding/decoding an interlace video signal using information of a lower layer whose types do not match"

The invention according to the embodiment reduces the network bandwidth usage when applying to a real-time video transmission service in a mobile communication network by compressing and transmitting an image by mixing an interlace method and a progressive method, and when an image is received from a video receiving terminal and output to a display An object of the present invention is to provide an image encoding and decoding method capable of recovering and outputting an image having the same resolution as the original image.

The invention according to an embodiment reduces network bandwidth usage when applying a real-time video transmission service in a mobile communication network by compressing and transmitting an image by mixing some of the interlace method and the progressive method, and receives the video from the video receiving terminal and then outputs it to the display In this case, it is possible to provide an image encoding and decoding method capable of recovering and outputting an image having the same resolution as that of the original image.

1 is a flowchart illustrating an operating method of an image processing apparatus according to an embodiment.
2 is a flowchart for explaining a method of operating an image receiving terminal according to an embodiment.
3 is a diagram for describing an encoding and decoding method according to an embodiment.
4 is a diagram for explaining a method of interpolating a frame that is not transmitted according to an embodiment.
5 is a block diagram illustrating the configuration of an image processing apparatus and an image receiving terminal according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but between each component another component It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, descriptions described in one embodiment may be applied to other embodiments as well, and detailed descriptions within the overlapping range will be omitted.

As a method for compressing an image, there are largely a progressive method and an interlace method.

The progressive method sequentially scans and compresses the image frame from the pixel at the (0,0) position, and the interlace method divides the image frame into odd line frames and even line frames. It is a method of separating, compressing and transmitting each, and deinterlacing odd-numbered line frames and even-numbered line frames in the decoder, and then outputting them.

Compared to the progressive method, the interlace method can provide the same frame rate with a half bit rate, but the delay time due to the deinterlacing process increases, and the image quality is deteriorated compared to the progressive method.

1 is a flowchart illustrating an operating method of an image processing apparatus according to an embodiment.

In operation 110, the image processing apparatus divides the frame of the original image into odd-numbered line frames and even-numbered line frames.

In an embodiment, it may be determined whether to encode the original image as it is, or to encode the frame separately through mode selection in the application. In the embodiment, an embodiment in which encoding is performed by dividing a frame will be described.

In an embodiment, the odd-numbered line frame and the even-numbered line frame may be separated from the original image by applying the above-described interlacing method.

In operation 120 , the image processing apparatus selects any one frame from among odd-numbered line frames and even-numbered line frames.

In the following description, an embodiment in which odd-numbered line frames are selected will be described. In an embodiment, the selected frame may be passed to the encoder.

The frame selected in step 130 is encoded in a progressive manner.

In an embodiment, it is possible to encode odd-numbered line frames in a progressive manner, and not perform encoding for unselected even-numbered line frames. By encoding only the selected frame, the vertical resolution of the image may be reduced by half compared to the original image.

Next, in step 140 , the image processing apparatus transmits the encoded frame to the image receiving terminal.

In the embodiment, since any one of the odd-numbered line frame and the even-numbered line frame is compressed and transmitted, the data transfer amount is reduced by half compared to the general progressive encoding.

In an embodiment, a command for restoring an image using the frame transmitted to the image receiving terminal may be transmitted together with the encoded frame, or may be transmitted in response to mode selection.

In an embodiment, the command for restoring an image may transmit a command for decoding an encoded frame or a command for interpolating an unselected frame using the transmitted frame. The interpolation method according to the embodiment will be described in detail later.

In an embodiment, the image processing apparatus may selectively separate frames and perform encoding on an original image to be transmitted.

For example, an encoding method may be set for an image stored and managed in the image processing apparatus in units of images. By analyzing the image through an image processing device or other analysis device, a section for dividing frames according to the complexity of the image can be set. encoding can be performed.

As another example, the image processing apparatus may set a rule for dividing frames in units of images. In general, an image with low complexity, such as animation, may be encoded by separating frames, and an image with high complexity, such as a movie, may be encoded without separating frames, depending on the type of image. Alternatively, based on the complexity of the image, when the complexity is higher than a predetermined criterion, the image may be encoded without dividing the frames.

In an embodiment, the decoding method may be transmitted to the image receiving terminal by including a command for image restoration in the protocol corresponding to the encoding method set in the image processing apparatus. As described above, a command for restoring an image may be transmitted together with the encoded frame to the image receiving terminal.

2 is a flowchart for explaining a method of operating an image receiving terminal according to an embodiment. The image receiving terminal according to the embodiment may include at least one of a display such as a smartphone, a tablet, a laptop, and/or a PC.

In step 210 , the video receiving terminal receives a frame in which one of an odd line frame or an even line frame of the original video is encoded.

In an embodiment, any one of odd-numbered line frames and even-numbered line frames separated from the original image may be selected through the method described in FIG. 1 to receive the progressively encoded image.

In an embodiment, the image receiving terminal may transmit a request for an image and receive a frame in which an image is compressed in response to the request.

In step 220, the video receiving terminal decodes the encoded frame in a progressive manner.

In an embodiment, the received frame is a frame encoded in a progressive manner, and the received frame may be decoded in a progressive manner.

In step 230, the video receiving terminal interpolates the decoded frame.

In an embodiment, the image receiving terminal may perform decoding and interpolation in response to a decoding command and/or an interpolation command received from the image processing apparatus. In an embodiment, the corresponding command may be received with the reception of the encoded frame, or the corresponding command may be received corresponding to the mode setting.

For example, when a command not to perform interpolation is received, the video receiving terminal may decode the frame in a progressive manner, and when a command to perform interpolation is received, the video receiving terminal decodes the frame in a progressive manner. Then, interpolation may be performed based on the decoded frame. If the decoded frame is an odd-numbered line frame, filtering by interpolation may be performed by generating an even-numbered line frame between frame columns.

In an embodiment, when receiving the encoded frame, the video receiving terminal may perform decoding on the received frame in a progressive manner based on a predetermined trigger. Alternatively, the video receiving terminal may perform interpolation based on the decoded frame after decoding the received frame in a progressive manner based on a predetermined trigger. In an embodiment, decoding and/or interpolation may be performed according to a command of an application for image reproduction of the image receiving terminal. The corresponding application may generate a command for decoding and/or interpolation by a predetermined setting or by a user setting.

A frame interpolation method according to an embodiment will be described in detail later.

In an embodiment, the image receiving terminal may restore the image by a decoding method corresponding to the encoding method set in the image processing apparatus.

For example, a frame may be divided into an odd-numbered line frame and an even-numbered line frame, and decoding may be performed on an encoded section of only one frame. To this end, the image receiving terminal may receive a protocol including information on a frame-separated section from the image processing apparatus.

As another example, the image receiving apparatus may apply a decoding method respectively corresponding to an encoded image and a general progressive-encoded image in which frames are separated according to the type of image. Alternatively, based on the complexity of the image, progressive decoding may be performed for a case where the complexity is less than a predetermined criterion, and then frames may be interpolated, and progressive decoding may be performed for a case where the complexity is greater than or equal to a predetermined criterion. To this end, the image receiving terminal may receive a protocol including information on whether a frame is separated and encoded image from the image processing apparatus.

In step 240, the video receiving terminal outputs the decoded frame and the interpolated frame according to the display of the video receiving terminal.

In an embodiment, an image recovered through decoding and/or interpolation may be output through an application within the image receiving terminal. The image may be output in full screen mode to fit the size of the display regardless of the resolution.

3 is a diagram for describing an encoding and decoding method according to an embodiment.

The original image is filled with odd-numbered line frames and even-numbered line frames intersecting in columns. In an embodiment, the image processing apparatus may divide the original image into odd-numbered line frames and even-numbered line frames, and may perform progressive encoding by selecting odd-numbered line frames among them. As shown, by performing progressive encoding, odd-numbered line frames can be provided in a tightly compressed form.

The encoded frame may be transmitted to an image receiving terminal through a wired/wireless network.

The video receiving terminal may perform progressive decoding on the encoded frame. In an embodiment, decoding may be performed in response to a decoding command received from the image processing apparatus, and an empty space of an even line frame may be generated in an odd line frame compressed through decoding. The corresponding space may be interpolated based on odd-numbered line frames. In an embodiment, an interpolation step may be performed in response to an interpolation command received from the image processing apparatus.

In an embodiment, an image obtained by decoding a frame received from the image receiving terminal in response to mode setting may be output to a display, or an image subjected to an interpolation step after decoding may be output to a display.

The image receiving terminal may output an image recovered through decoding and/or interpolation to a display. In an embodiment, it may be played in an application for playing the image of the image receiving terminal.

In an embodiment, the application may output an image in full-screen mode to fit the display of the image receiving terminal regardless of the resolution of the image, and then the size of the output image may be adjusted in response to a user's setting.

For example, an image reconstructed through decoding and interpolation may have a resolution with less error than the original image, and an image reconstructed through decoding only may have lower resolution compared to the original image.

4 is a diagram for explaining a method of interpolating a frame that is not transmitted according to an embodiment.

In an embodiment, a corresponding pixel value may be calculated by applying a weight to RGB values of adjacent pixels with respect to a specific pixel.

For example, it may be assumed that the pixel columns 401 and 403 correspond to pixel columns of odd-numbered line frames, and the pixel columns 402 correspond to pixel columns of even-numbered line frames. In an embodiment, when the odd-numbered line frame is encoded and transmitted, the pixel columns 401 and 403 in the image receiving apparatus may be restored through decoding. Accordingly, all pixels of the image may be recovered by performing interpolation of the pixel column 402 .

The RGB color of the pixel IL can be obtained using the RGB values of adjacent TL, BL, TC, and BC. Among them, the RGB color of IL can be obtained and interpolated by applying a 75% weight to the average of the RBG values of TL and BL above and below IL, and adding 25% of the average RGB values of TC and BC to the right of the diagonal line of IL. have. That is, it can be expressed by the following equation.

[Equation 1]

IL_R = ((TL_R + BL_R)/2)*0.75 + ((TC_R + BC_R)/2)*0.25

IL_G = (TL_G + BL_G)/2)*0.75 + ((TC_G + BC_G)/2)*0.25

IL_B = ((TL_B + BL_B)/2)*0.75 + ((TC_B + BC_B)/2)*0.25

In the same way, the RGB color of the pixel IC can be obtained using the RGB values of adjacent TL, BL, TC, BC, TR, and BR. The RGB color of the IC is calculated by applying a weight of 50% to the average of the RBG values of TC and BC above and below the IC, and adding 50% of the average RGB values of TL, BL, TR, and BR on the left and right sides of the IC diagonal. can be retrieved and interpolated. Expressed as a mathematical formula, it is as follows:

[Equation 2]

IC_R = ((TL_R + BL_R + TR_R + BR_R)/4)*0.5 + ((TC_R + BC_R)/2)*0.5

IC_G = ((TL_G + BL_G + TR_G + BR_G)/4)*0.5 + ((TC_G + BC_G)/2)*0.5

IC_B = ((TL_B + BL_B + TR_B + BR_B)/4)*0.5 + ((TC_B + BC_B)/2)*0.5

In an embodiment, the RGB color of the pixel IR may be obtained using RGB values of adjacent TR, BR, TC, and BC. Apply 75% weight to the average of the RBG values of TR and BR above and below IR, and add 25% to the average of the RGB values of TC and BC on the left side of the diagonal of IR to obtain the RGB color of IR and interpolate. can The formula is as follows.

[Equation 3]

IR_R = ((TR_R + BR_R)/2)*0.75 + ((TC_R + BC_R)/2)*0.25

IR_G = ((TR_G + BR_G)/2)*0.75 + ((TC_G + BC_G)/2)*0.25

IR_B = ((TR_B + BR_B)/2)*0.75 + ((TC_B + BC_B)/2)*0.25

5 is a block diagram illustrating the configuration of an image processing apparatus and an image receiving terminal according to an embodiment.

The image processing apparatus 510 according to the embodiment may include a frame separator 511 , an encoder 512 , and a communication unit 513 , and the image receiving terminal 520 includes a communication unit 521 and a decoder 522 . ) and a playback unit 523 .

The frame separator 511 divides the frame of the original image into odd-numbered line frames and even-numbered line frames.

In an embodiment, it may be determined whether to encode the original image as it is, or to encode the frame separately through mode selection in the application. In the embodiment, an embodiment in which encoding is performed by dividing a frame will be described.

In an embodiment, the odd-numbered line frame and the even-numbered line frame may be separated from the original image by applying the above-described interlacing method.

In an embodiment, it is possible to select which one of the odd-numbered line frame and the even-numbered line frame to perform encoding and transmit it to the encoder 512 .

The encoder 512 encodes the selected frame in a progressive manner.

In the following description, an embodiment in which odd-numbered line frames are selected will be described.

In an embodiment, it is possible to encode odd-numbered line frames in a progressive manner, and not perform encoding for unselected even-numbered line frames. By encoding only the selected frame, the vertical resolution of the image may be reduced by half compared to the original image.

Next, the communication unit 513 transmits the encoded frame to the video receiving terminal 520 .

In the embodiment, since any one of the odd-numbered line frame and the even-numbered line frame is compressed and transmitted, the data transfer amount is reduced by half compared to the general progressive encoding.

In an embodiment, the communication unit 513 may transmit a command for restoring an image using a frame transmitted to the image receiving terminal. For example, a command may be sent along with the frame or a command may be sent in response to mode selection.

In an embodiment, the command for restoring an image may transmit a command for decoding an encoded frame or a command for interpolating an unselected frame using the transmitted frame.

The communication unit 521 of the video receiving terminal 520 receives a frame in which one of an odd line frame or an even line frame of the original video is encoded.

In an embodiment, any one of odd-numbered line frames and even-numbered line frames separated from the original image may be selected to receive a progressively encoded image.

The decoder 522 decodes the encoded frame in a progressive manner.

In an embodiment, the received frame is a frame encoded in a progressive manner, and the received frame may be decoded in a progressive manner.

The decoder 522 interpolates the decoded frame.

In an embodiment, the decoder 522 may perform decoding and interpolation in response to a decoding command and/or an interpolation command received from the image processing apparatus 510 . In an embodiment, the corresponding command may be received with the reception of the encoded frame, or the corresponding command may be received corresponding to the mode setting.

For example, when a command not to perform interpolation is received, the decoder 522 may decode a frame in a progressive manner, and when a command to perform interpolation is received, the video receiving terminal decodes the frame in a progressive manner After that, interpolation may be performed based on the decoded frame. If the decoded frame is an odd-numbered line frame, filtering by interpolation may be performed by generating an even-numbered line frame between frame columns.

In an embodiment, decoding and/or interpolation may be performed according to a command of an application for image reproduction of the image receiving terminal. As described above, the corresponding application may generate a command for recovering a received frame by a predetermined trigger.

The interpolation method according to the embodiment may refer to the description of FIG. 4 .

The playback unit 523 outputs the decoded frame and the interpolated frame according to the display of the image receiving terminal 520 .

In an embodiment, an image recovered by decoding and/or interpolation may be output through an application for image reproduction of the image receiving terminal 520 . The image may be output in full screen mode to fit the size of the display regardless of the resolution.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

A method of operating an image processing apparatus, comprising:
analyzing the complexity of the original image;
determining an encoding mode for the original video based on the complexity;
dividing the frame of the original image into odd-numbered line frames and even-numbered line frames according to the encoding mode;
selecting any one of the odd-numbered line frame and the even-numbered line frame;
encoding the selected frame in a progressive manner; and
Transmitting the encoded frame to a video receiving terminal
including,
Determining an encoding mode for the original video based on the complexity includes:
determining the encoding mode as a mode for separating frames of the original image when the complexity is less than a predetermined criterion
containing,
A method of operating an image processing device.
According to claim 1,
Frames that are not selected among the odd-numbered line frame and the even-numbered line frame are not encoded or transmitted to the video receiving terminal,
A method of operating an image processing device.
According to claim 1,
transmitting a command for recovering an image using the transmitted frame to the image receiving terminal
further comprising,
A method of operating an image processing device.
4. The method of claim 3,
Sending the command includes:
sending a command to decode the encoded frame; and
transmitting a command for interpolating an unselected frame among the odd-numbered line frame and the even-numbered line frame using the decoded frame
comprising at least one of
A method of operating an image processing device.
According to claim 1,
The transmitted frame is decoded by the video receiving terminal and output as a full screen of the display of the video receiving terminal,
A method of operating an image processing device.
In the operation method of the video receiving terminal,
receiving, from an image processing apparatus, a frame in which any one frame of an odd-numbered line frame or an even-numbered line frame of an original image is encoded;
decoding the received frame in a progressive manner;
interpolating the decoded frame; and
outputting the decoded and interpolated frame according to the display of the video receiving terminal
including,
The encoded frame is
As a result of analyzing the complexity of the original image through the image processing device, any one of odd-numbered line frames or even-numbered line frames of the original image whose complexity is less than a predetermined criterion is an encoded frame;
A method of operating a video receiving terminal.
7. The method of claim 6,
a frame that is not transmitted among the odd-numbered line frame and the even-numbered line frame is not encoded by the image processing apparatus;
A method of operating a video receiving terminal.
7. The method of claim 6,
Receiving a command for restoring an image using the received frame from the image processing device
further comprising,
decoding the received frame in a progressive manner; and
Interpolating the decoded frame comprises:
corresponding to receiving a command to restore the image;
A method of operating a video receiving terminal.
7. The method of claim 6,
generating a command for recovering the received frame in response to receiving the encoded frame at the video receiving terminal
further comprising,
A method of operating a video receiving terminal.
10. The method of claim 9,
The step of generating the command is
generating instructions for decoding the received frame; and
generating a command for interpolating an unselected frame among the odd-numbered line frame and the even-numbered line frame using the decoded frame
comprising at least one of
A method of operating a video receiving terminal.
7. The method of claim 6,
Interpolating the decoded frame comprises:
separating the decoded frame in units of columns; and
interpolating between a first column and a second column of the decoded frame;
containing
A method of operating a video receiving terminal.
12. The method of claim 11,
Interpolating between the first column and the second column of the decoded frame comprises:
For interpolation of the first pixel between the first column and the second column, applying a weight to the adjacent pixels based on the distance between the first pixel and the adjacent pixels in the first column and the second column step; and
calculating each RGB value of the first pixel based on the applied weight
containing,
A method of operating a video receiving terminal.
A computer program stored on a medium in combination with hardware to execute the method of any one of claims 1 to 12.
In the image processing apparatus,
a frame separator for dividing the frame of the original image into odd-numbered line frames and even-numbered line frames;
an encoder for encoding one of the odd line frame and the even line frame; and
Communication unit for transmitting the encoded frame to the video receiving terminal
including,
The frame separation unit,
Analyze the complexity of the original image,
Determine the encoding mode for the original video based on the complexity, and when the complexity is less than a predetermined criterion, determine the encoding mode as a mode for separating frames of the original video,
Separating the frame of the original video according to the encoding mode,
image processing device.
In the video receiving terminal,
a communication unit configured to receive a frame in which one of odd-numbered line frames or even-numbered line frames of the original image is encoded from the image processing apparatus;
a decoder for decoding the encoded frame and interpolating the decoded frame; and
A playback unit for outputting the decoded and interpolated frame according to the display of the video receiving terminal
including,
The encoded frame is
As a result of analyzing the complexity of the original image through the image processing device, any one of odd-numbered line frames or even-numbered line frames of the original image whose complexity is less than a predetermined criterion is an encoded frame;
video receiving terminal.

Images