KR20210021630A - Method for transmiting contents and terminal apparatus using the same - Google Patents

Abstract

The present application relates to a method for transmitting content and a content creation terminal using the same. According to one embodiment of the present invention, the method for transmitting content of the content creation terminal may comprise the steps of: creating, by the content creation terminal, original content; generating compressed data obtained by encoding the original content and reference data obtained by sampling the original content; and transmitting the compressed data and reference data to a distribution server.

Description

Contents transmission method and content generation terminal using the same {Method for transmiting contents and terminal apparatus using the same}

The present application relates to a content transmission method capable of improving the quality, such as resolution of content provided through streaming, and a content generation terminal using the same.

Streaming technology refers to a technology that optimally uses network and CPU resources by transmitting/receiving content such as large-capacity video and audio in real time.

Streaming technology is becoming more and more important with the growth of the Internet because most users do not have fast enough access lines to download large multimedia files immediately.

In general, the method of receiving content service (for example, video content playback service) after downloading all the content from the receiving side requires a long waiting time from the user, but when using streaming technology, before downloading all the content Even though it is possible to provide a content service in real time by downloading only data on the current playback point through a client browser or plug-in. In this way, the streaming service has the advantage of being able to receive a content service as soon as a user selects a content.

The present application is to provide a content transmission method capable of improving the quality of content limited by a bandwidth or transmission speed of a transmission medium, and a content generation terminal using the same.

The present application is intended to provide a content transmission method capable of implementing content quality improvement at a receiving side by allowing a content generation terminal to further transmit reference data together with encoded compressed data, and a content generation terminal using the same.

The present application is to provide a content transmission method capable of performing quality improvement of received content using a deep learning-based quality improvement model, and a content generation terminal using the same.

According to an embodiment of the present invention, a method for transmitting content by a content generating terminal includes: generating, by the content generating terminal, original content; Generating compressed data obtained by encoding the original content and reference data obtained by sampling the original content; And transmitting the compressed data and reference data to a distribution server.

According to an embodiment of the present invention, a method for improving content quality of a distribution server includes: receiving compressed data obtained by encoding original content and reference data by sampling the original content from a content generation terminal; Decoding the compressed data to generate decoded data; And generating output data with improved quality of the decoded data by applying the decoded data and reference data to a quality improvement model.

A content generation terminal according to an embodiment of the present invention includes: a content generation unit that generates original content; An encoding unit generating compressed data by encoding the original content; A reference data generator configured to generate reference data by sampling the original content; And a transmission unit for transmitting the compressed data and reference data to a distribution server.

The distribution server according to an embodiment of the present invention includes: a receiver configured to receive compressed data obtained by encoding original content and reference data extracted by sampling the original content from a content generation terminal; A decoding unit for generating decoded data by decoding the compressed data; And a quality improvement unit that applies the decoded data and reference data to a quality improvement model to generate output data having improved quality of the decoded data.

According to another embodiment of the present invention, a method for improving content quality in a client terminal includes the steps of: receiving, from a content server, compressed data encoding original content and reference data sampling the original content; Decoding the compressed data to generate decoded data; And generating output data with improved quality of the decoded data by applying the decoded data and reference data to a quality improvement model.

According to another embodiment of the present invention, a content transmission method of a content server includes: receiving a request for streaming of original content from a client terminal; Setting an encoding quality according to a bit rate of a transmission medium connected to the client terminal and encoding the original content according to the encoding quality to generate compressed data; Generating reference data for the original content by sampling the original content every preset period or by sampling whenever a scene change occurs in the original content; And transmitting the compressed data and reference data to the client terminal through the transmission medium.

In addition, the solution to the above-described problem does not enumerate all the features of the present invention. Various features of the present invention and advantages and effects thereof may be understood in more detail with reference to the following specific embodiments.

According to a content transmission method according to an embodiment of the present invention and a content generation terminal using the same, since quality improvement can be performed at the distribution server side even when content of limited quality is received from the content generation terminal, high-quality content is provided. It is possible to provide to the user.

According to the content transmission method and the content generating terminal using the same according to an embodiment of the present invention, it is possible to implement quality improvement using reference data sampled of the original content instead of the entire original content.

However, the content transmission method according to the embodiments of the present invention and the effects that can be achieved by the content generating terminal using the same are not limited to those mentioned above, and other effects that are not mentioned are the present invention from the following description. It will be clearly understood by those of ordinary skill in the relevant technical field.

1 is a schematic diagram showing a content quality improvement system according to an embodiment of the present invention.
2 is a block diagram showing a content generation terminal and a distribution server of a content quality improvement system according to an embodiment of the present invention.
3 is a flow chart showing a method for transmitting content by a content generating terminal according to an embodiment of the present invention.
4 is a flow chart showing a method for improving content quality of a distribution server according to an embodiment of the present invention.
5 is a schematic diagram showing a content quality improvement system according to another embodiment of the present invention.
6 is a block diagram showing a client terminal and a content server of a content quality improvement system according to another embodiment of the present invention.
7 is a flow chart showing a method for improving content quality of a client terminal according to another embodiment of the present invention.
8 is a flow chart showing a content transmission method of a content server according to another embodiment of the present invention.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but identical or similar elements are denoted by the same reference numerals regardless of reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves. That is, the term'unit' used in the present invention means a hardware component such as software, FPGA or ASIC, and the'unit' performs certain roles. However,'part' is not limited to software or hardware. The'unit' may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Thus, as an example,'unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, Includes subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables. The functions provided in the components and'units' may be combined into a smaller number of components and'units', or may be further divided into additional components and'units'.

In addition, in describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all modifications included in the spirit and scope of the present invention It should be understood to include equivalents or substitutes.

1 is a schematic diagram showing a content quality improvement system according to an embodiment of the present invention.

Referring to FIG. 1, a content quality improvement system according to an embodiment of the present invention may include a content generating terminal 100, a distribution server 200, and a content receiving terminal 300.

Hereinafter, a content quality improvement system according to an embodiment of the present invention will be described with reference to FIG. 1.

The content generating terminal 100 may generate original content according to a user's manipulation, and the generated original content may be distributed to a plurality of content receiving terminals 300 through the distribution server 200.

For example, a user can directly shoot a video or image using a camera and a microphone provided in the content generation terminal 100, or record sound to create original content. Depending on the embodiment, a separate camcorder It is also possible to generate original contents using a camera, a recorder, etc. and then store them in the contents generation terminal 100. Thereafter, the content generation terminal 100 may support editing or modification of the generated original content.

The content generation terminal 100 can supply the generated original content to the distribution server 200 through a communication network, and the content generation terminal 100 encodes a multimedia signal such as a video signal and an audio signal corresponding to the original content. It can be converted into a multimedia stream.

Here, the encoded multimedia stream is a real-time streaming protocol (Real Time Streaming Protocol: RTSP, regulated in RFC 2326), a real-time transport protocol (RTP), a real-time messaging protocol. : RTMP) or the like can be transmitted to the distribution server 200.

The content generation terminal 100 may include a display unit for visually displaying the generated original contents, an input unit receiving user input, a communication unit for wired/wireless communication with the distribution server 200, and at least one program This may include a memory and a processor to be stored.

In addition, the content creation terminal 100 includes a mobile phone, a smart phone, a laptop computer, a desktop computer, a digital broadcasting terminal, a personal digital assistants (PDA), a portable multimedia player (PMP), Slate PC, tablet PC, ultrabook, wearable device (e.g., smartwatch, smart glass), head mounted display (HMD) )), etc.

The content generation terminal 100 may be connected to the distribution server 200 through a communication network, where the communication network may include a wired network and a wireless network. Specifically, it may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN), and may include a known World Wide Web (WWW: World). Wide Web). However, the communication network according to the present invention is not limited to the networks listed above, and may include a known wireless data network, a known telephone network, a known wired or wireless television network, and the like.

The content receiving terminal 300 may display the content provided from the distribution server 200 or the like in a visual or audible manner and provide it to a user. The content receiving terminal 300 may include a display unit for visually displaying contents, an input unit receiving user input, a communication unit for wired/wireless communication with the content server 200, and at least one program stored therein. It may include a memory and a processor. For example, the content receiving terminal 300 may be a smart phone or a tablet PC, and may be a portable device or a fixed device of the same type as the content generation terminal 100 described above. In addition, the content receiving terminal 300 may be connected to the distribution server 200 through a communication network, where the communication network may include a wired network and a wireless network.

The distribution server 200 may receive content from the content generating terminal 100 and may distribute the received content to a plurality of content receiving terminals 300. Here, the distribution server 200 may perform quality improvement on the content received from the content generation terminal 100 and then distribute it.

Specifically, when the content generation terminal 100 transmits the original content to the distribution server 200, it may go through a transmission medium such as an optical cable or a coaxial cable in a communication network. In this case, the resolution or sound quality of the original contents transmitted from the content generation terminal 100 to the distribution server 200 may be limited by the bandwidth (bandwith) or the transmission rate (bit rate) of the transmission medium. That is, when the content generation terminal 100 encodes the original content, the quality can be adjusted and encoded according to the transmission speed of the transmission medium, and accordingly, the distribution server 200 provides compressed data encoded with different qualities. I can receive it. Accordingly, the distribution server 200 may receive compressed data of low quality from the content generation terminal 100, and may generate decoded data by decoding it. In this case, a content whose resolution is relatively lower than that of the original content or the sound quality of a sound source is deteriorated may be provided.

However, since the distribution server 200 needs to provide high-quality content to each of the content receiving terminals 300, it is necessary to improve the quality of the content received from the content generating terminal 100. Here, according to the content quality improvement system according to an embodiment of the present invention, the content generation terminal 100 additionally provides high-quality reference data along with compressed data to the distribution server 200 to improve the quality of deteriorated content. It is possible to do.

Hereinafter, a content generation terminal 100 and a distribution server 200 according to an embodiment of the present invention will be described in detail.

2 is a block diagram showing a content generation terminal and a distribution server of a content quality improvement system according to an embodiment of the present invention.

Referring to FIG. 2, a content generation terminal 100 according to an embodiment of the present invention includes a content generation unit 110, an encoding unit 120, a reference data generation unit 130, and a transmission unit 140. I can.

The content generator 110 may generate original content. That is, the content generation unit 110 may capture a video or an image using a camera or a microphone included in the content generation terminal 100, or record sound to generate original content. For example, a celebrity, such as a celebrity, can create original content in such a way as to photograph their daily life as a video using their smart phone.

The encoding unit 120 may generate compressed data by encoding original content. Here, the encoding unit 120 may set the encoding quality according to the bit rate of the transmission medium connected to the distribution server 200, and may encode the original content according to the encoding quality. Here, the encoding unit 120 is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

The reference data generator 130 may generate reference data by sampling the original content. Since the encoding unit 120 generates compressed data by limiting the quality during encoding according to the transmission speed of the transmission medium, the distribution server 200 may receive compressed data of limited quality according to the transmission speed of the transmission medium. In this case, when only the received compressed data is decoded and the content is reproduced, a problem such as a relatively low quality such as a resolution of the content may occur.

In order to solve this problem, the reference data generation unit 130 may additionally generate reference data in addition to the compressed data. That is, reference data, which is metadata including information of the original content, may be further provided to the distribution server 200, and quality improvement may be performed by using the reference data thereafter.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. First, when the original content is a video, a high-quality image generated by capturing the original content may be used as reference data. That is, since the reference data includes high-definition image information corresponding to a specific frame included in the original content, it is possible to improve the quality of the frames before and after the special frame by utilizing this.

In addition, when the original content is a sound source, text information extracted from the sound source through speech recognition may be used as reference data. That is, when there is a problem with sound quality such as an incorrect voice in the compressed data of a sound source, it can be utilized for quality improvement, such as making the corresponding voice emit a sound corresponding to text information using text information.

Meanwhile, the reference data generation unit 130 may generate reference data by sampling the original content at every setting period, and according to an embodiment, it is also possible to generate reference data whenever a scene change occurs in the original content.

For example, if the original content is a video, the video may be captured once every N seconds to generate reference data. That is, it is possible to use reference data sampled from the original content, not the entire original content, for quality improvement. Here, since the reference data is a high-quality image, the capacity may be relatively large. Therefore, by including the reference data once every N seconds, the transmission capacity can be reduced and the quality can be improved. According to an embodiment, for a full-frame video of 640X360, an image having a resolution of 1920X1080 may be generated as reference data, one image per 10 seconds.

In addition, in the case of original content that is a video, there may be a number of cases in which the scene is switched after a similar scene in the video is maintained for a predetermined time. Here, since a sudden change occurs during a scene change, reference data may be generated at this time to enable quality improvement corresponding to the changed scene.

Additionally, the reference data generation unit 130 may generate reference data by further including a time stamp at the time when each reference data is generated.

The transmission unit 140 may transmit compressed data and reference data to the distribution server 200. Here, the transmission unit 140 may transmit compressed data and reference data according to a preset network protocol. For example, RTMP (Real Time Messaging Protocol), HTTP (Hypertext Transfer Protocol) can be used.

Meanwhile, referring to FIG. 2, the distribution server 200 according to an embodiment of the present invention may include a receiving unit 210, a decoding unit 220, and a quality improving unit 230.

The receiving unit 210 may receive compressed data obtained by encoding the original content from the content generation terminal 100 and reference data extracted by sampling the original content. That is, the content generation terminal 100 may request content distribution to the distribution server 200 in order to distribute its original content, and in this case, the distribution server 200 may transmit the content generation terminal 100 through the receiving unit 210. ) Transmitted compressed data and reference data can be received. Depending on the embodiment, the content generation terminal 100 may request a live broadcast to the distribution server 200, and the distribution server 200 uses the compressed data and reference data transmitted from the content generation terminal 100 to live live in real time. Streaming can be performed.

The decoding unit 220 may generate decoded data by decoding the compressed data. The decoding unit 220 may decode the compressed data using a codec used when encoding the compressed data. Here, the decoding unit 220 is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

The quality improvement unit 230 may apply the decoded data and the reference data to the quality improvement model to generate output data with improved quality of the decoded data. Here, the quality improvement model may be pre-trained based on deep learning, and if the quality improvement model is used, it is possible to estimate the original content corresponding to each of the decoded data from the decoded data and the reference data.

That is, instead of the entire original content, reference data sampled from the original content can be used, and the quality improvement model may be trained to generate output data having a quality corresponding to the original content by comparing the reference data and the decoded data. have.

Here, when the original content is a video, the quality improvement unit 230 may generate output data by restoring the resolution of the original content by using the decoded data and reference data, and when the original content is an audio, the sound quality is restored. Data can be created.

Depending on the embodiment, a super resolution model for restoring resolution as a quality improvement model, and SDR-HDR (Standard Dynamic Range-High Dynamic Range) image conversion for extending the dynamic range of the video Models can be used, and in addition, various models can be used such as image enhancement models and sound quality improvement models.

3 is a flowchart illustrating a method of transmitting content in a content generating terminal according to an embodiment of the present invention.

Referring to FIG. 3, the content generating terminal may generate original content (S110). That is, a video or image may be captured using a camera or a microphone included in the content generating terminal, or original content may be generated by recording sound or the like.

Thereafter, the content generation terminal may generate compressed data by encoding the original content, and generate reference data by sampling the original content (S120). That is, the content generation terminal may generate compressed data and reference data corresponding to the original content in order to transmit the original content to the distribution server.

Here, the content generation terminal may set the encoding quality according to the transmission speed of the transmission medium connected to the distribution server, and may generate compressed data by encoding the original content according to the encoding quality. Here, the content creation terminal is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

In addition, the content generation terminal may generate reference data for the original content by sampling the original content every setting period or whenever a scene change occurs in the original content. That is, in order to perform quality improvement in the distribution server, reference data including information on the original content may be additionally generated.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. Here, when the original content is a video, a high-quality image captured from the original content may be used as reference data, and when the original content is a sound source, text information extracted through voice recognition on the corresponding sound source may be used as reference data. In this case, the reference data may further include a timestamp of when each reference data is generated.

Meanwhile, when the original content is a video, the content server may generate reference data by capturing the video once every N seconds. At this time, since the reference data corresponds to a high-quality image, the capacity may be relatively large. Accordingly, the content server generates reference data once every N seconds, thereby reducing the required capacity and implementing quality improvement. For example, if the original content is a full-frame video of 640X360, the content server may generate an image having a resolution of 1920X1080 as reference data, one per 10 seconds.

In addition, there may be a number of cases in which a similar scene is maintained for a certain period of time, and then the scene is switched. Here, since a sudden change occurs during scene change, the content server may generate reference data at this time to enable quality improvement corresponding to the changed scene.

Thereafter, the content generation terminal may transmit the generated compressed data and reference data to the distribution server (S130). Here, the content generation terminal may transmit compressed data and reference data according to a preset network protocol, and may utilize, for example, Real Time Messaging Protocol (RTMP) or Hypertext Transfer Protocol (HTTP).

4 is a flow chart showing a method for improving content quality in a distribution server according to an embodiment of the present invention.

Referring to FIG. 4, the distribution server may receive compressed data encoding original content and reference data sampled original content from a content generation terminal (S210). The content creation terminal can request distribution of the original content to the distribution server, and can transmit compressed data and reference data to be distributed to the distribution server. Here, since the reference data includes information on the original content, it is possible to perform quality improvement by using it later.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. When the original content is a video, a high-resolution image captured of the original content may be used as reference data. In addition, when the original content is a sound source, text information extracted from the sound source through speech recognition may be used as reference data.

Meanwhile, the reference data may be generated by sampling the original content at every setting period, and according to embodiments, it may be generated whenever a scene change occurs in the original content.

Thereafter, the distribution server may generate decoded data by decoding the compressed data (S220). The distribution server can decode the compressed data by using the codec used when the content generation terminal encodes the compressed data. Here, the distribution server is MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE and other video codecs, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

When decoding is completed, the distribution server may apply the decoded data and the reference data to the quality improvement model to generate output data with improved quality of the decoded data (S230). Here, the quality improvement model is pre-trained based on deep learning, and if the quality improvement model is used, it is possible to estimate the original content corresponding to each of the decoded data from the decoded data and the reference data. That is, it is possible to generate output data having a quality corresponding to the original content from the decoded data by using the partial reference data sampled from the original content rather than the entire original content.

Depending on the embodiment, a super resolution model for restoring resolution as a quality improvement model, and SDR-HDR (Standard Dynamic Range-High Dynamic Range) image conversion for extending the dynamic range of the video Models can be used, and in addition, image enhancement models and sound quality improvement models can be used.

5 is a schematic diagram showing a content quality improvement system according to another embodiment of the present invention.

Referring to FIG. 5, a content quality improvement system according to another embodiment of the present invention may include a client terminal 600 and a content server 700.

Hereinafter, a content quality improvement system according to an embodiment of the present invention will be described with reference to FIG. 5.

The client terminal 600 may display the content provided from the content server 700 or the like in a visual or audible manner and provide it to a user. Here, the content provided from the content server 700 may be a video, audio, or image, and according to an embodiment, streaming of a video on-demand (VOD) or live broadcasting provided by the content server 700 Can be printed in a way.

The client terminal 600 may include a display unit for visually displaying contents, an input unit receiving user input, a communication unit for wired/wireless communication with the content server 700, and a memory in which at least one program is stored. And a processor.

Here, the client terminal 600 is a mobile phone, a smart phone, a laptop computer, a desktop computer, a digital broadcasting terminal, a personal digital assistants (PDA), a portable multimedia player (PMP), and a slate. PC (slate PC), tablet PC (tablet PC), ultrabook (ultrabook), wearable device (wearable device, e.g., smartwatch, glass type terminal (smart glass), HMD (head mounted display) ), etc.

The client terminal 600 may be connected to the content server 700 through a communication network, where the communication network may include a wired network and a wireless network. Specifically, it may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN), and may include a known World Wide Web (WWW: World). Wide Web). However, the communication network according to the present invention is not limited to the networks listed above, and may include a known wireless data network, a known telephone network, a known wired or wireless television network, and the like.

The content server 700 may provide content requested by the client terminal 100, and according to an embodiment, the content server 700 may provide content in real time using streaming or the like.

When a request for specific content is input from the client terminal 600, the content server 700 may search for the corresponding content in the content server 700 or the original content database D, and retrieve the searched content through a communication network. It can be provided to the client terminal 600 through. Here, the content server 700 may provide contents such as live broadcasting provided by channels operated by various broadcasting institutions or individuals.

The original content database D may store various types of video, such as VOD, and may include various types of content such as sound sources, images, and games in addition to the video.

When the content server 700 transmits the original content to the client terminal 600, it may go through a transmission medium such as an optical cable or a coaxial cable of a communication network. At this time, quality such as resolution of contents transmitted to the client terminal 600 may be lowered due to a bandwidth (bandwith) of a transmission medium or a transmission rate (bit rate). For example, when the content server 700 streams a video and transmits the video to the client terminal 600, the content server 700 may encode the original content and generate the compressed data. At this time, since the content server 700 adjusts and encodes the quality of the video according to the transmission speed of the transmission medium, the client terminal 600 may receive compressed data encoded with different qualities according to the transmission speed of the transmission medium. .

That is, after receiving the compressed data of low quality, the client terminal 600 may generate decoded data by decoding it, and in this case, the content of which the resolution is relatively low or the sound quality of the sound source is deteriorated compared to the original Can play.

Here, various methods for improving the quality of content executed in the client terminal 600 are proposed, and according to the content quality improvement system according to an embodiment of the present invention, the content server 700 refers to the compressed data together. By providing additional data to the client terminal 600, the quality of deteriorated content may be improved. Hereinafter, each of the client terminal 600 and the content server 700 according to an embodiment of the present invention will be described.

6 is a block diagram showing a client terminal and a content server of a content quality improvement system according to an embodiment of the present invention.

Referring to FIG. 6, a client terminal 600 according to an embodiment of the present invention may include a receiving unit 610, a decoding unit 620, and a quality improving unit 630.

The client terminal 600 may request the content server 700 to provide the original content. In this case, the receiving unit 610 encodes the original content from the content server 700 and the compressed data from the original content. The extracted reference data can be received.

Here, the content server 700 may generate compressed data by limiting the quality when encoding the original content according to the bit rate of the transmission medium, and the receiving unit 610 is limited according to the transmission speed of the transmission medium. You can receive quality compressed data. In this case, when only the received compressed data is decoded and the content is reproduced, a problem such as a relatively low quality such as a resolution of the content may occur.

To solve this, the receiving unit 610 may additionally receive reference data in addition to the compressed data. That is, reference data corresponding to metadata including information on the original content may be further provided from the content server 700, and quality improvement may be performed by using the reference data thereafter.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. First, when the original content is a video, a high-quality image generated by capturing the original content may be used as reference data. That is, since the reference data includes high-definition image information corresponding to a specific frame included in the original content, the quality of frames before and after the special frame can be improved by utilizing this.

In addition, when the original content is a sound source, text information extracted from the sound source through speech recognition may be used as reference data. That is, when there is a problem with sound quality such as an incorrect voice in the compressed data of a sound source, it can be utilized for quality improvement, such as making the corresponding voice emit a sound corresponding to text information using text information.

Meanwhile, the reference data may be generated by sampling the original content at every preset period, and according to embodiments, it may be generated whenever a scene change occurs in the original content.

For example, if the original content is a video, the video may be captured once every N seconds to generate reference data. That is, it is possible to generate reference data by sampling the original content rather than the entire original content. Here, since the reference data is a high-quality image, the capacity may be relatively large. Therefore, it is possible to implement quality improvement while reducing the required capacity by including reference data once every N seconds. According to an embodiment, for a full-frame video of 640X360, an image having a resolution of 1920X1080 may be generated as reference data, one image per 10 seconds.

In addition, in the case of original content that is a video, there may be a number of cases in which the scene is switched after a similar scene in the video is maintained for a predetermined time. Here, since a sudden change occurs during a scene change, reference data may be generated at this time to enable quality improvement corresponding to the changed scene.

Additionally, the reference data may further include a time stamp at the time when each reference data is generated, along with a high-quality image.

The decoding unit 620 may generate decoded data by decoding the compressed data. The decoding unit 620 may decode the compressed data using a codec used when encoding the compressed data. Here, the decoding unit 620 is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

The quality improvement unit 630 may apply the decoded data and the reference data to the quality improvement model, and may generate output data with improved quality of the decoded data from the quality improvement model. Here, the quality improvement model may be pre-trained based on deep learning, and if the quality improvement model is used, it is possible to estimate the original content corresponding to each of the decoded data from the decoded data and the reference data.

That is, instead of the entire original content, reference data sampled from the original content can be used, and the quality improvement model can be trained to generate output data having a quality corresponding to the original content by comparing the reference data and the decoded data. .

Here, the quality improvement unit 630 may generate output data that restores the resolution of the original content by using the decoded data and reference data when the original content is a video, and when the original content is audio, the sound quality is restored. Data can be created.

Depending on the embodiment, a super resolution model for restoring resolution as a quality improvement model, and SDR-HDR (Standard Dynamic Range-High Dynamic Range) image conversion for extending the dynamic range of the video Models can be used, and in addition, various models can be used such as image enhancement models and sound quality improvement models.

Meanwhile, referring to FIG. 6, a content server 700 according to an embodiment of the present invention may include an encoding unit 710, a reference data generation unit 720, and a transmission unit 730.

The content server 700 may receive a request for streaming of the original content from the client terminal 600, and in this case, the encoding unit 710 may encode the original content in response to the streaming request.

Specifically, the encoding unit 710 may set the encoding quality according to the bit rate of the transmission medium connected to the client terminal 600, and may generate compressed data by encoding the original content according to the encoding quality. . Here, the encoding unit 710 is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

In addition, the reference data generator 720 may sample the original content at every preset period or sample whenever a scene change occurs in the original content to generate reference data for the original content.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. Here, when the original content is a video, a high-quality image captured of the original content may be used as reference data, and when the original content is a sound source, text information extracted through voice recognition on the corresponding sound source may be used as reference data. In addition, the reference data may further include a timestamp of a time point at which each reference data is generated, along with a high-quality image acquired through sampling.

Meanwhile, when the original content is a video, the reference data generator 720 may generate reference data by capturing the video once every N seconds. For example, when the original content is a full-frame video of 640X360, the reference data generator 720 may generate an image having a resolution of 1920X1080 as reference data one by one every 10 seconds.

In addition, there may be a number of cases in which a similar scene is maintained for a certain period of time, and then the scene is switched. Here, since a sudden change occurs during a scene change, the reference data generator 720 may generate reference data each time a scene is changed to enable quality improvement corresponding to the changed scene.

The transmission unit 730 may transmit compressed data and reference data to the client terminal 600 through a transmission medium. Here, the transmission unit 730 may transmit compressed data and reference data according to a preset network protocol. For example, RTMP (Real Time Messaging Protocol), HTTP (Hypertext Transfer Protocol) can be used.

7 is a flow chart showing a method for improving content quality in a client terminal according to an embodiment of the present invention.

Referring to FIG. 7, the client terminal may receive compressed data obtained by encoding original content from a content server and reference data extracted from the original content (S710). The client terminal may request the content server to provide original content, and in this case, may receive compressed data and reference data from the content server. Here, since the reference data includes information on the original content, it is possible to perform quality improvement by using it later.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. When the original content is a video, a high-resolution image captured of the original content may be used as reference data. In addition, when the original content is a sound source, text information extracted from the sound source through speech recognition may be used as reference data.

Meanwhile, the reference data may be generated by sampling the original content at every preset period, and according to embodiments, it may be generated whenever a scene change occurs in the original content.

Thereafter, the client terminal may generate decoded data by decoding the compressed data (S720). The client terminal may decode the compressed data using the codec used when the content server encodes the compressed data. Here, the client terminal is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA, FLAC. Audio codecs such as DTS can be used.

When decoding is completed, the client terminal may apply the decoded data and the reference data to the quality improvement model to generate output data with improved quality of the decoded data (S730). Here, the quality improvement model is pre-trained based on deep learning, and if the quality improvement model is used, it is possible to estimate the original content corresponding to each of the decoded data from the decoded data and the reference data. That is, it is possible to generate output data having a quality corresponding to the original content from the decoded data by using the reference data sampled from the original content rather than the entire original content.

Depending on the embodiment, a super resolution model for restoring resolution as a quality improvement model, and SDR-HDR (Standard Dynamic Range-High Dynamic Range) image conversion for extending the dynamic range of the video Models can be used, and in addition, image enhancement models and sound quality improvement models can be used.

8 is a flowchart illustrating a method of transmitting content in a content server according to an embodiment of the present invention.

Referring to FIG. 8, the content server may receive a request for streaming of original content from a client terminal (S810). In this case, the content server may search for the original content from the content server or the original content database.

Thereafter, when the requested original content is retrieved, the content server may set the encoding quality according to the bit rate of the transmission medium connected to the client terminal, and may generate compressed data by encoding the original content according to the encoding quality. (S820). Specifically, the encoding quality may be set according to the transmission speed of the transmission medium connected to the client terminal, and compressed data may be generated by encoding the original content according to the encoding quality. Here, the content server is a video codec such as MPEG, Indeo, DivX, Xvid, H.264, WMV, RM, Cinepak, MOV, ASF, RA, XDM, RLE, MP3, AC3, AAC, OGG, WMA , FLAC. Audio codecs such as DTS can be used.

In addition, the content server may generate reference data for the original content by sampling the original content at every preset period or whenever a scene change occurs in the original content (S830). That is, the content server may additionally generate reference data including information on the original content so that the client terminal can perform quality improvement.

Specifically, the reference data may be a high-definition image captured of the original content or text information generated by voice recognition of the original content. Here, when the original content is a video, a high-quality image captured of the original content may be used as reference data, and when the original content is a sound source, text information extracted through voice recognition on the corresponding sound source may be used as reference data. In this case, the reference data may further include a timestamp of a time point at which each reference data is generated, along with a high-quality image obtained through sampling.

Meanwhile, when the original content is a video, the content server may generate reference data by capturing the video once every N seconds. At this time, since the reference data corresponds to a high-quality image, the capacity may be relatively large. Accordingly, the content server generates reference data once every N seconds, thereby reducing the required capacity and implementing quality improvement. For example, if the original content is a full-frame video of 640X360, the content server may generate an image having a resolution of 1920X1080 as reference data, one per 10 seconds.

In addition, there may be a number of cases in which a similar scene is maintained for a certain period of time, and then the scene is switched. Here, since a sudden change occurs during scene change, the content server may generate reference data at this time to enable quality improvement corresponding to the changed scene.

Thereafter, the content server may transmit the generated compressed data and reference data to the client terminal through a transmission medium (S840). Here, the content server may transmit compressed data and reference data according to a preset network protocol, and may utilize, for example, Real Time Messaging Protocol (RTMP) or Hypertext Transfer Protocol (HTTP).

The present invention described above can be implemented as a computer-readable code on a medium on which a program is recorded. The computer-readable medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single piece of hardware or several pieces of hardware are combined. The medium is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And ROM, RAM, flash memory, and the like may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server. Therefore, the detailed description above should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not limited by the above-described embodiments and the accompanying drawings. It will be apparent to those of ordinary skill in the art to which the present invention pertains, that components according to the present invention can be substituted, modified, and changed within the scope of the technical spirit of the present invention.

100: content generation terminal 110: content generation unit
120: encoding unit 130: reference data generation unit
140: transmission unit 200: distribution server
210: receiving unit 220: decoding unit
230: quality improvement unit 300: content receiving terminal
600: client terminal 610: receiver
620: decoding unit 630: quality improvement unit
700: content server 710: encoding unit
720: reference data generation unit 730: transmission unit

Claims (12)

In the content transmission method of the content generation terminal,
Generating, by the content generation terminal, original content;
Generating compressed data obtained by encoding the original content and reference data obtained by sampling the original content; And
And transmitting the compressed data and reference data to a distribution server.
The method of claim 1, wherein the original content
And a live broadcast content generated by the content generating terminal in real time or a recorded broadcast content stored in the content generating terminal.
The method of claim 1, wherein the generating step
And generating compressed data of limited quality according to a bit rate of a transmission medium between the content generating terminal and the distribution server.
The method of claim 1, wherein the reference data is
And a reference image of an original resolution generated by capturing the original content or a reference text generated by voice recognition of the original content.
The method of claim 1, wherein the reference data is
And generating the original content by sampling at each setting period or whenever a scene change occurs in the original content.
In the content quality improvement method of the distribution server,
Receiving compressed data obtained by encoding original contents and reference data obtained by sampling the original contents from a contents generating terminal;
Generating decoded data by decoding the compressed data; And
And generating output data having improved quality of the decoded data by applying the decoded data and reference data to a quality improvement model.
The method of claim 6, wherein the quality improvement model
The content quality improvement method, which is learned based on deep learning, and restores the decoded data to output data having a resolution or sound quality of the original content using the reference data.
The method of claim 6, wherein the quality improvement model
Content quality improvement method, characterized in that it is any one of a super resolution model, a standard dynamic range-high dynamic range (SDR-HDR) image conversion model, an image enhancement model, and a sound quality improvement model. .
A content generator for generating original content;
An encoding unit generating compressed data by encoding the original content;
A reference data generator configured to generate reference data by sampling the original content; And
Content generation terminal comprising a transmission unit for transmitting the compressed data and reference data to the distribution server.
A receiver configured to receive compressed data obtained by encoding original content and reference data extracted by sampling the original content from a content generation terminal;
A decoding unit for generating decoded data by decoding the compressed data; And
Distribution server comprising a quality improvement unit for applying the decoded data and reference data to a quality improvement model to generate output data having improved quality of the decoded data.
In the content quality improvement method in the client terminal,
Receiving compressed data obtained by encoding original contents and reference data obtained by sampling the original contents from a contents server;
Generating decoded data by decoding the compressed data; And
And generating output data having improved quality of the decoded data by applying the decoded data and reference data to a quality improvement model.
In the content transmission method of the content server,
Receiving a request for streaming of original content from a client terminal;
Setting an encoding quality according to a bit rate of a transmission medium connected to the client terminal and encoding the original content according to the encoding quality to generate compressed data;
Generating reference data for the original content by sampling the original content every preset period or by sampling whenever a scene change occurs in the original content; And
And transmitting the compressed data and reference data to the client terminal through the transmission medium.

Images