KR20230140266A - Method and system for optimizing video encoding using double buffering in single encoding structure - Google Patents

Abstract

Disclosed is a video encoding optimization method and system using double buffering in a single encoding structure. An encoding optimization method according to an embodiment includes the steps of storing frame images corresponding to the first segment of an input image in a first segment buffer, and when storage of the frame images corresponding to the first segment in the first segment buffer is completed. In response, changing the storage buffer to a second segment buffer to store frame images corresponding to a second segment in the second segment buffer, predicting encoding options for frame images stored in the first segment buffer, and It may include transmitting the predicted encoding option and frame images stored in the first segment buffer to an encoder.

Description

Video encoding optimization method and system using double buffering in a single encoding structure {METHOD AND SYSTEM FOR OPTIMIZING VIDEO ENCODING USING DOUBLE BUFFERING IN SINGLE ENCODING STRUCTURE}

The explanation below is about video encoding technology.

Recently, video and sound have been combined with communications and computers to provide multimedia information that has been integrated into new media. For example, with the provision of high-speed data transmission networks, users can watch stereoscopic sound and high-definition video, and make face-to-face calls between users through video phones. Additionally, you can purchase products by viewing product information in real time through a computer or TV, and watch music or movies through a website. Additionally, it is possible to take video lectures through a computer.

Such multimedia information has been developed based on video compression (i.e. encoding) technology. Data that carries information can be compressed by removing redundant elements (elements that are not essential to accurately reconstruct the data) from the data. In the case of lossy compression, the data restored by the decoder is not identical to the original data, but subjective redundant elements are removed to achieve high compression efficiency. In image or video compression, subjective redundant elements are elements that can be removed without significantly affecting the image quality that the viewer can intuitively feel.

[Prior document number]

Korean Patent No. 10-1136858

Using artificial intelligence (AI) technology, we provide encoding optimization technology that can improve compression efficiency by finding the optimal encoding parameters for each section of the video.

Provides encoding optimization technology that can improve throughput by using double buffering in a single encoding structure.

An encoding optimization method executed on a computer device including at least one processor, comprising: storing frame images corresponding to a first segment of an input image in a first segment buffer, by the at least one processor; In response to completion of storage of frame images corresponding to the first segment in the first segment buffer, the at least one processor changes the storage buffer to a second segment buffer to store frame images corresponding to the second segment. storing in a second segment buffer; predicting, by the at least one processor, encoding options for frame images stored in the first segment buffer; and transmitting, by the at least one processor, the predicted encoding option and the frame images stored in the first segment buffer to an encoder.

According to one side, the encoding optimization method is such that the storage of frame images corresponding to the second segment in the second segment buffer is completed by the at least one processor, and the frame images stored in the first segment buffer are stored in the encoder. The step of changing the storage buffer to the first segment buffer and storing frame images corresponding to the third segment in the first segment buffer may be further included in response to being transmitted to the first segment buffer.

According to another aspect, the step of storing frame images corresponding to the third segment in the first segment buffer includes, when frame images to be transmitted to the encoder remain in the first segment buffer, the frame images corresponding to the third segment are stored in the first segment buffer. After waiting until all stored frame images are delivered to the encoder, frame images corresponding to the third segment may be stored in the first segment buffer.

According to another aspect, the encoding optimization method is such that, by the at least one processor, frame images corresponding to the second segment are completed in the second segment buffer, and frame images stored in the first segment buffer are The method may further include changing the storage buffer to a third segment buffer in response to transmission to the encoder and storing frame images corresponding to the third segment in the third segment buffer.

According to another aspect, the encoding optimization method includes predicting, by the at least one processor, encoding options for frame images stored in the second segment buffer; and transmitting, by the at least one processor, the predicted encoding option and the frame images stored in the second segment buffer to an encoder.

According to another aspect, the step of predicting the encoding option may be characterized by predicting the encoding option corresponding to the feature of the input image using an artificial intelligence model.

According to another aspect, the artificial intelligence model is a convolution neural network (CNN) model for extracting frame features for each frame image, and a recurrent neural network (RNN) model for extracting video features based on relationships between the frame images. An encoding option prediction model including a model and a classifier that classifies encoding options corresponding to the video feature, wherein the CNN model, the RNN model, and the classifier are ) can be characterized as being learned in a way.

According to another aspect, the step of predicting the encoding option may be characterized by predicting a constant rate factor (CRF) that satisfies a target VMAF (Video Multi-method Assessment Fusion) score as the encoding option.

According to another aspect, predicting the encoding option includes predicting a first CRF that satisfies a target VMAF score as the encoding option; predicting a second CRF that satisfies a target bit rate as the encoding option; and determining a third CRF to be actually applied to encoding the input image using the first CRF and the second CRF.

A computer program stored on a computer-readable recording medium is provided in conjunction with a computer device to execute the method on the computer device.

Provided is a computer-readable recording medium on which a program for executing the above method on a computer device is recorded.

A computer device, comprising at least one processor configured to execute instructions readable by the computer device, wherein frame images corresponding to a first segment of an input image are stored in a first segment buffer by the at least one processor. In response to completion of storing frame images corresponding to the first segment in the first segment buffer, the storage buffer is changed to the second segment buffer to store the frame images corresponding to the second segment in the second segment buffer. A computer device is provided that stores, predicts encoding options for frame images stored in the first segment buffer, and transmits the predicted encoding options and the frame images stored in the first segment buffer to an encoder. .

Using artificial intelligence (AI) technology, compression efficiency can be improved by finding the optimal encoding parameters for each section of the video.

Throughput can be improved by using double buffering in a single encoding structure.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
Figure 3 is a diagram showing an example of components of a distributed encoding system in one embodiment of the present invention.
Figure 4 is a diagram for explaining the basic concept of an encoding option prediction model for encoding optimization in an embodiment of the present invention.
Figure 5 shows a video encoding optimization method in one embodiment of the present invention.
Figures 6 and 7 are example diagrams for explaining image quality measurement indices for each resolution in one embodiment of the present invention.
Figure 8 is an example diagram for explaining the process of generating label data for model learning in one embodiment of the present invention.
Figure 9 is an example diagram illustrating a process for predicting additional encoding options considering service constraints in one embodiment of the present invention.
Figure 10 is a diagram illustrating an example of an image analysis process on a segment basis, according to an embodiment of the present invention.
Figure 11 is a diagram illustrating an example of a segment-level image analysis process using two buffers, according to an embodiment of the present invention.
Figure 12 is a flowchart showing an example of an encoding optimization method according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The video encoding optimization system according to embodiments of the present invention may be implemented by at least one computer device. At this time, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the video encoding optimization method according to the embodiment of the present invention under the control of the driven computer program. there is. The above-described computer program can be combined with a computer device and stored in a computer-readable recording medium to execute the video encoding optimization method on the computer.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment in FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, and 140, a plurality of servers 150 and 160, and a network 170. Figure 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited as in Figure 1.

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented with a computer system. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, Personal Digital Assistants (PDAs), and Portable Multimedia Players (PMPs). ), tablet PCs, game consoles, wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, etc. For example, in FIG. 1, the shape of a smartphone is shown as an example of the electronic device 110. However, in embodiments of the present invention, the electronic device 110 actually communicates with other devices through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer systems capable of communicating with electronic devices 120, 130, 140 and/or servers 150, 160.

The communication method is not limited, and may include not only a communication method utilizing communication networks that the network 170 may include (e.g., mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.), but also short-range wireless communication between devices. You can. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 170 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

Each of the servers 150 and 160 is a computer device or a plurality of computers that communicate with a plurality of electronic devices 110, 120, 130, 140 and a network 170 to provide commands, codes, files, content, services, etc. It can be implemented with devices. For example, the server 150 may be a system that provides a first service to a plurality of electronic devices 110, 120, 130, and 140 connected through the network 170, and the server 160 also provides a network ( It may be a system that provides a second service to a plurality of electronic devices 110, 120, 130, and 140 connected through 170). As a more specific example, the server 150 provides a service (for example, a video service, etc.) targeted by the application through an application as a computer program installed and running on a plurality of electronic devices 110, 120, 130, and 140. As a first service, it can be provided through a plurality of electronic devices 110, 120, 130, and 140. As another example, the server 160 may provide a service for distributing files for installing and running the above-described application to a plurality of electronic devices 110, 120, 130, and 140 as a second service.

Figure 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices 110, 120, 130, and 140 described above or each of the servers 150 and 160 may be implemented by the computer device 200 shown in FIG. 2.

As shown in FIG. 2, this computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 200 as a separate permanent storage device that is distinct from the memory 210. Additionally, an operating system and at least one program code may be stored in the memory 210. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on computer programs installed by files received over network 170.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 170. For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as memory 210 is transmitted to the network ( 170) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170. Signals, commands, data, etc. received through the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc. may be stored in a storage medium (as described above) that the computer device 200 may further include. It can be stored as a permanent storage device).

The input/output interface 240 may be a means for interfacing with the input/output device 250. For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 240 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. The input/output device 250 may be configured as a single device with the computer device 200.

Additionally, in other embodiments, computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most prior art components. For example, the computer device 200 may be implemented to include at least some of the input/output devices 250 described above, or may further include other components such as a transceiver, a database, etc.

One of the video compression algorithms takes human visual characteristics into account and uses a method of increasing the compression rate by adding a level of loss to the original video that is difficult for humans to perceive. You can adjust the video compression rate by adjusting the degree of loss.

Commercial video compressors (codecs) provide a bitrate control function to maintain the bit rate (video size per second) at a level that does not cause problems for streaming without significantly deteriorating video quality through lossy compression. .

Commonly used video codecs maintain a fixed target bit rate or provide a bit rate control function to maintain constant image quality. Here, maintaining a constant image quality means that the image quality of the encoding result is uniform, and the level of image quality cannot be known before encoding.

However, because the complexity and characteristics of each image are different, compressing all images with the same target bit rate or image quality results in compression with a higher quality than human perception and compression into a size that is larger than necessary.

Additionally, since videos often consist of multiple scenes with different complexity and characteristics, the above problem may occur depending on the section of the video.

These embodiments seek to optimize the compression rate of a video by determining an appropriate bit rate or appropriate image quality option according to the characteristics of each section of the video through AI technology.

Figure 3 is a diagram showing an example of components of a distributed encoding system in one embodiment of the present invention.

Referring to FIG. 3, the distributed encoding system 300 is an optimized encoder component for application to the video platform 330, and includes a distributed encoder 310 and an AI serving module 320. It can be included.

Each of the distributed encoder 310 and the AI serving module 320 may be implemented by the computer device 200 described with reference to FIG. 2 .

In the video service, the video to be served can be uploaded to the video platform 330 and encoded through the distributed encoding system 300. Whether or not to optimize the encoding option is determined by the video platform 330, and when optimization is applied, parameters such as model information may be added to the encoding request.

The distributed encoder 310 may include a distributed processor 311 and a worker 312.

The distributed processing unit 311 divides the original image into several segments and assigns them to a plurality of workers 312. As encoding progresses in the worker 312, a compressed bitstream is received, and the received bitstream is temporarily stored in local storage or managed as loaded in memory. The distributed processing unit 311 may generate a final video file by merging the encoding results of the workers 312 for each segment.

The worker 312 is a unit transcoder for performing encoding tasks and serves to encode video segments allocated from the distributed processing unit 311. At this time, the worker 312 may operate in direct connection with the AI serving module 320 to optimize encoding options. If encoding option optimization is activated, some frame images required for encoding option prediction can be prepared and the prepared frame images can be transmitted to the AI serving module 320 along with model information to request the optimal encoding option. The worker 312 performs encoding by applying the prediction result of the AI serving module 320 and checks the encoding result image quality and transmission bit rate. If the image quality or transmission bit rate as a result of encoding is outside the appropriate range, the worker 312 interpolates the prediction error based on experimental values and performs encoding again. Simultaneously with the encoding process, the compressed bitstream is transmitted to the distributed processing unit 311, and the model prediction result and encoding result data are stored in the video platform 330.

The AI serving module 320 operates in conjunction with the distributed encoder 310 and includes an encoding option prediction model 321. When receiving an encoding option request for encoding a video segment from each worker 312, the AI serving module 320 predicts the optimal encoding option through the encoding option prediction model 321 specified in the received request, resulting in a prediction result. can be returned to the worker (312).

The video platform 330 may collect logs about the prediction accuracy of the encoding option prediction model 321 based on cloud search such as ELK (Elastic Logstash Kibana). In other words, the video platform 330 records the encoding results of the worker 312 as a log to monitor the accuracy of the encoding option prediction model 321 and monitors the monitoring results and the user (e.g., administrator, etc.) 340. Based on the input value, it can support additional model learning and improvement of prediction accuracy through new images.

In this embodiment, the encoding option prediction model 321 is an AI-based prediction model that determines the characteristics of each segment of the video and predicts the optimal encoding option accordingly.

Encoding options refer to parameters that can control the encoding rate, that is, the video compression rate. For example, CRF (Constant Rate Factor) can be used as an optimized encoding option.

CRF guarantees visually more uniform image quality compared to CQP (constant quantization parameter) and can reflect human perceptual recognition characteristics.

The basic concept of the encoding option prediction model 321 is shown in FIG. 4.

Referring to FIG. 4, the encoding option prediction model 321 can receive the images 401 of the frames constituting the video and extract features of the video through a deep learning model. At this time, the deep learning model may include a convolution neural network (CNN) model and a recurrent neural network (RNN) model. The CNN model is responsible for extracting features of frame images, and the RNN model is responsible for learning relationships between sequences of data (features of frame images).

The encoding option prediction model 321 is learned to classify the optimal CRF class for each feature extracted through a deep learning model through supervised learning.

When a frame image of a new video is input to the encoding option prediction model 321 learned in this way, the optimal CRF category suitable for the characteristics of the image can be predicted.

Figure 5 shows a video encoding optimization method in one embodiment of the present invention.

Referring to FIG. 5, the distributed encoding system 300 includes a learning process (S510) and an inference process (S520) for the encoding option prediction model 321.

The encoding option prediction model 321 includes a CNN model, a RNN model, and a classifier.

First, the distributed encoding system 300 performs a pre-processing process. The preprocessing process includes extracting frame images from video segments. The distributed encoding system 300 can optimize frame image size, number of frame images, etc. to prevent memory problems and increase accuracy in the learning process (S510).

The data set for learning the encoding option prediction model 321 can be a video segment data set created using videos uploaded to a video platform.

For example, the encoding optimization option uses CRF and can determine the CRF range to be learned by considering the distribution of the data set and the effective range during actual encoding.

The frame image size can be increased from the initial 224×224 to 336×336, 448×448, 560×560, and beyond. In sizes below 336×336, the original video can be resized after center-cropping 80%. Resize so that the original information is maintained as much as possible, but the aspect ratio of the original may be changed.

When the frame image (1920×1080) of the original video is resized to the above size, information loss is bound to occur during the resizing process. To reduce this loss, increasing the image size as much as possible helps improve accuracy.

In the case of the number of frame images, 15 images at 250ms intervals from a 4-second segment video can be used as model input to improve accuracy. To improve accuracy, the distributed encoding system 300 can select 10 images at 500 ms intervals from a 5-second segment video and then resize the selected images to a size of 336×336 that can be processed by CNN. In other words, for one segment image, 10 pieces × (336 × 336 × 3) data can be provided as input to the encoding option prediction model 321.

This is just an example, and the selection of images to use as model input can be changed at any time. For example, it is also possible to select 20 images at 200ms intervals from a 4-second segment video.

Through repeated encoding of each video, CRF options that satisfy the appropriate image quality of VMAF (Video Multi-method Assessment Fusion) standards are explored, and the collected 4-second segment video and GT (ground truth) data set are used to create an encoding option prediction model ( 321) can be used in the learning process (S510).

Next, the distributed encoding system 300 performs a video feature extraction process. In CNN, frame features for each frame image given as input to the encoding option prediction model 321 can be extracted, and frame features extracted from CNN are input into RNN (LSTM (Long Short-Term Memory)) in image order. Video features can be extracted based on relationship information between feature sequences.

Lastly, the distributed encoding system 300 performs a classification process and, for example, can classify video features extracted from the RNN using a softmax classifier.

Therefore, the distributed encoding system 300 can fine-tune the CNN model and learn the entire model, including the CNN model, for one loss function in an end-to-end (E2E) manner.

For image classification performance, the weights of the CNN learned in the learning process (S510) can be optimized to better distinguish image characteristics. With the supervised learning method, you can learn the characteristics of the video, that is, the relationship between video features and encoding options (CRF), and build a model that classifies it as the optimal option suitable for the characteristics of the video.

When a frame image of a new video is input to the encoding option prediction model 321 learned through the above-described learning process (S510), the optimal CRF category suitable for the characteristics of the image can be predicted as an inference process (S520).

In particular, these embodiments can predict target image quality achievement parameters for multiple resolution images by building the encoding option prediction model 321 as a single model.

Streaming services serve one video in multiple resolutions, and each resolution must be encoded to the target quality. Encoding parameters for encoding to the target image quality can be derived through image analysis using AI, and the derived results may be different for each resolution. In these cases, it is common to operate a separate model for each resolution.

For example, when encoding 1080p and 720p videos, a model for 1080p and a model for 720p are created respectively, and each video is inferred with the corresponding model. As the number of models increases in a service environment that supports multiple resolutions, it becomes difficult to develop, manage, and maintain models.

In general, VMAF is used as a picture quality measurement indicator for streaming services. VMAF is an image quality indicator that considers both compression artifacts and scaling artifacts.

Figure 6 is a general VMAF measurement method, and shows the results of resizing all encoded resolutions to 1080p and comparing them with the original 1080p.

By obtaining the convex-hull of the overlapping VMAF curve and selecting a point on the convex-hull, it is possible to select the optimal resolution for the corresponding bitrate. However, because the VMAF range of each resolution is different, the VMAF standard for each resolution cannot be set the same.

Figure 7 compares the encoding results at each resolution with the original image before encoding at the same resolution, and shows the results considering only compression artifacts and excluding scaling artifacts.

Although it is not possible to create a convex-hull, the VMAF standard can be used equally for all resolutions. In other words, when obtaining the label used for learning the encoding option prediction model 321, the same image quality score (e.g., VMAF 93) for the entire resolution can be taken as the GT value.

In this way, if the values of picture quality encoding parameters (fixed picture quality encoding parameters such as CRF, QP (quantization parameter), etc.) that satisfy the same standard (VMAF score) for all resolutions are used as labels, CRFs can be grouped into the same category.

In other words, if CRF values that satisfy the same VMAF standard for each resolution are obtained as labels, images with the same label can be grouped as data with the same label even if the resolutions are different. For example, 1080p CRF 23 and 720p CRF 23 can be grouped into the same category, so when constructing a data set for learning, the data set in the CRF 23 category can be composed by mixing 1080p and 720p videos.

If each label category is learned by constructing a data set that mixes multiple resolutions, one model can be used for encoding of multiple resolutions. CRF for 1080p and 720p resolution encoding can be predicted with one model.

The method for generating label data is as follows.

The optimal CRF value of each segment image is used as a label.

Referring to FIG. 8, the distributed encoding system 300 repeats encoding by varying the CRF value for the segment image 801 (S81), and at this time, measures the image quality of the encoding result (S83) to determine whether a certain level of image quality is satisfied. Determine whether or not (S83).

The distributed encoding system 300 can obtain the label 802 by finding a CRF value that satisfies a certain level of image quality through the above-described processes (S81 to S83).

The standard for picture quality is VMAF, which best reflects human perception. For example, VMAF 93, which is suitable for streaming services, can be used as the standard. Each segment image 801 is encoded by changing the CRF until the VMAF value becomes 93, and the CRF value at the point when the VMAF value becomes 93 is selected as the label 802.

For example, the label creation process for 1080p and 720p resolution encoding is as follows.

The 1080p resolution encoding label obtains the label CRF based on the original 1080p based on VMAF 93.

In the case of a 720p resolution encoding label, the original 1080p is first resized to 720p, and the label CRF is obtained by repeating encoding with different CRFs based on the resized 720p. The quality standard is VMAF 93, the same as 1080p. At this time, when obtaining VMAF, the original image quality comparison is a video resized from 1080p to 720p. In other words, the VMAF is measured by considering only the compression artifact, without considering the resize artifact, and the corresponding CRF is obtained. A CRF that satisfies a specific image quality (VMAF) considering only compression artifacts is obtained for each resolution. At this time, the CRF category can be grouped into multiple resolutions, so 1080p and 720p resolutions can be grouped to create a data set for each category.

Therefore, it is possible to build a single model that supports both 1080p and 720p resolutions with a mixed 1080p and 720p data set, and through this, optimal CRF results can be obtained as inference results for 1080p segmented images and 720p segmented images. .

In the process of generating label data, for example, when encoding is performed while changing the CRF in increments of 1, the CRF closest to VMAF 93 is selected, so there is a problem of reduced label accuracy.

To solve this, the accuracy of the label can be improved by applying a label class with the CRF in units smaller than 1, for example, 0.5, through testing on the CRF unit.

Therefore, by using the encoding option prediction model 321 built for CRF prediction, encoding can be optimized and repetitive processes can be minimized.

Furthermore, these embodiments can predict efficient target picture quality achievement parameters under service constraints.

When compressing video for streaming services, bitrate restrictions are necessary to minimize video quality degradation and ensure uninterrupted viewing. Therefore, it is necessary to generate the resulting image within the bitrate constraint, and the bitrate constraint and importance of the constraint are different for each service. It is also important to produce results with the highest quality possible at the same bit rate.

Figure 9 shows an example of a process for deriving picture quality parameters to be applied to actual encoding considering service constraints in one embodiment of the present invention.

Referring to FIG. 9, the distributed encoding system 300 may extract features of the input image 901 (S900) and then predict an encoding option optimized for the extracted features (S901 to S904).

In the process of predicting the encoding option, the remaining steps (S901, S902, S903) except step (S903) use the AI-based encoding option prediction model (321).

In detail, the distributed encoding system 300 can predict the first CRF (CRF1), which is an image quality encoding parameter that satisfies the target image quality (e.g., VMAF 93) as an encoding option corresponding to the features of the input image 901. (S901). Since the method for predicting the first CRF is the same as described above, detailed description is omitted.

The distributed encoding system 300 may predict a second CRF (CRF2), which is an image quality encoding parameter that satisfies the target bitrate, as an encoding option corresponding to the features of the input image 901 (S902). As a video quality control parameter, the CRF option, which provides uniform picture quality for each section, is generally used, but when encoding with CRF, the bitrate of the resulting video is unknown. In order to find a CRF that satisfies the target bitrate, the CRF value must be adjusted, repeated encoding must be performed, and the resulting bitrate must be checked, which incurs a very large encoding cost. In order to satisfy the target image quality and service bitrate limitations with a low encoding cost, a second CRF prediction that satisfies the target bitrate is required. By predicting a second CRF that satisfies the target bit rate along with a first CRF that satisfies the target image quality, it is possible to reduce costs and improve service quality.

For the second CRF prediction, the CRF value that satisfies the specific bitrate of each segment image is used as a label. By encoding the CRF value differently for each segment image, you can find the CRF value that satisfies the target bitrate and select that CRF value as a label. The target bit rate may be different for each application, and for example, label data may be generated as 2nd CRF 23 for category 1, 2nd CRF 23.5 for category 2, 2nd CRF 24 for category 3, etc.

Depending on the embodiment, it is also possible to generate label data by combining the first CRF and the second CRF. For example, for category 1, label data is generated as 1st CRF 23 and 2nd CRF 24, for category 2, 1st CRF 23 and 2nd CRF 25, for category 3, 1st CRF 23 and 2nd CRF 26, etc. can do. When using label data in which the first CRF and the second CRF are combined, steps S902 and S903 are omitted in FIG. 9 and the CRF predicted in step S901 is used as the final CRF to apply to encoding. You can.

The encoding option prediction model 321 receives the images of the frames constituting the input image 901, extracts the features of the image using deep learning (CNN and RNN), and classifies them into classes corresponding to each CRF using the extracted features. It is learned through supervised learning. At this time, the correct answer label of the CRF class becomes the second CRF generated from the label data above. When label data is generated by combining the first CRF and the second CRF, the correct answer label of the CRF class consists of a combination of the first CRF and the second CRF. By inputting frames of the video to be analyzed into the encoding option prediction model 321 learned with this label data set, a class corresponding to a CRF that satisfies the bit rate can be output.

The distributed encoding system 300 may determine the third CRF (CRF3), which is the final CRF to be applied to actual encoding, by considering the first CRF that satisfies the target image quality and the second CRF that satisfies the target bit rate (S903). For example, the distributed encoding system 300 may determine the first CRF to be the third CRF when the first CRF is greater than or equal to the second CRF. Meanwhile, if the first CRF is smaller than the second CRF, the distributed encoding system 300 may determine the third CRF by considering the bitrate limit compliance weight, which is a user input value related to model learning (S903). In applications where bitrate constraints must be strict, if the first CRF is smaller than the second CRF, the second CRF may be determined as the third CRF. In applications where bitrate constraints are relatively less stringent, a value closer to the first CRF can be selected for image quality gain. At this time, the distributed encoding system 300 may determine the third CRF by correcting the CRF value using the bitrate limit compliance weight (w), which is a user input value (Equation 1).

[Equation 1]

CRF3 = CRF1×(1-w)+CRF2×w

The bitrate limit compliance weight can be entered as a value between 0 and 1.0. If the weight is 0, the first CRF may be determined as the third CRF without complying with the bit rate limit, and if the weight is 1, the second CRF may be determined as the third CRF to strictly comply with the bit rate limit.

The distributed encoding system 300 can predict the QP limitation parameter as an additional encoding option corresponding to the features of the input image (S904).

CRF, one of the picture quality encoding parameters, determines the frame QP. The encoder is equipped with various block-level QP decision algorithms such as AQ (adaptive quantization) and MB (macroblock)-tree to improve perceived picture quality, and it is common to use a combination of CRF and the corresponding encoding options. The QP determination algorithm determines the block QP by applying an offset value for each block to the frame QP determined from the CRF. At this time, blocks that are assigned an excessively low or high QP are created. The encoder provides options such as minimum QP (minqp) and maximum QP (maxqp) for QP limitation, and by using these to limit the QP value of blocks to an appropriate level, additional bitrate reduction can be achieved while maintaining perceived image quality. there is.

When applied together with the third CRF, the distributed encoding system 300 can derive the minimum QP value and maximum QP value that can obtain the optimal bitrate-to-image quality. The minimum QP prevents excessively high image quality, and the maximum QP prevents excessively low image quality, and can be applied in blocks to achieve visually more even image quality. For example, the distributed encoding system 300 may derive the minimum QP value and maximum QP value by applying an offset to the third CRF (minqp=CRF3-offset, maxqp=CRF3+offset). For example, when the CRF offset is 2, the value obtained by subtracting 2 from CRF3 can be set as the minimum QP value, and the value obtained by adding 2 to CRF3 can be set as the maximum QP value.

The distributed encoding system 300 may apply the minimum QP value and maximum QP value as encoding options along with the third CRF, and depending on the embodiment, the process of predicting the QP limit parameter (S904) is omitted to reduce the inference time. And it is also possible to apply only the third CRF to actual encoding.

To predict the QP limitation parameter, the CRF offset that satisfies the optimal cost when combined with the encoding CRF (CRF3) determined for each segment is used as a label. Each segment image is encoded by setting different CRF values and setting different minimum QP and maximum QP (or offset) for each CRF value. After obtaining the bitrate and image quality (VMAF) of each encoding result, the cost can be obtained from this, and the offset value of the image with the lowest cost can be selected as the label.

Cost can be defined as Equation 2:

[Equation 2]

Cost = rate+λ×d

Here, the λ (lambda) value is determined as a user input value when learning the model depending on whether to prioritize bit rate reduction or image quality preservation.

The features of the image and the third CRF determined in the previous step are input values, and label data can be generated, for example, CRF offset 2 for category 1, CRF offset 4 for category 2, and CRF offset 6 for category 3. You can. Finally, the encoding application parameters are the third CRF, minimum QP (CRF3-offset), and maximum QP (CRF3+offset).

In general, the lower the QP, the lower the image quality gain compared to the bit rate, so the offset can be applied asymmetrically. In other words, the minimum QP can be determined as (CRF3-offset+1), and the maximum QP can be applied as (CRF3+offset). For example, when the third CRF value is 26 and the offset is 4, the minimum QP value may be 23 and the maximum QP value may be 30.

The encoding option prediction model 321 receives the images of the frames that make up the input video, extracts the features of the image using deep learning (CNN and RNN), and learns through supervised learning to classify the features and the optimal cost class for each encoding CRF. do. At this time, the correct answer label of the CRF class consists of the CRF offset generated from the label data above. By inputting the frame image of the video to be analyzed and the third CRF value into the encoding option prediction model 321 learned with this label data set, the CRF offset value can be obtained as a QP limitation parameter corresponding to the classification result category.

Therefore, in this embodiment, in predicting the optimal encoding option to be applied to encoding using the AI-based encoding option prediction model 321, it is necessary to predict image quality encoding parameters that satisfy the target image quality, as well as predict image quality encoding parameters under service constraints. A picture quality encoding parameter that satisfies the target bitrate and/or a QP limit parameter that satisfies the picture quality balance condition can be additionally predicted.

Meanwhile, in the above embodiments, it was explained that in the case of the number of frame images, 15 images at 250 ms intervals in analysis segment units (for example, 4-second segment images) can be used as model input to improve accuracy.

Figure 10 is a diagram illustrating an example of an image analysis process on a segment basis, according to an embodiment of the present invention. The segment analyzer (1010) can correspond to the AI serving module (320) described above, and the segment analyzing module (1012) included in the segment analyzer (1010) is an encoding option prediction model (321). can respond.

The segment analysis unit 1010 may receive a video frame (decoded video frame, 1020) and store frame images for the current segment in a segment buffer (1011). Thereafter, the segment analysis module 1012 can take the frame images stored in the segment buffer 1011 and predict the optimal encoding option for the frame images of the corresponding segment. In Figure 10, encoding options are indicated as quality parameters (CRF). At this time, the segment analysis module 1012 can transmit the predicted encoding option to the segment buffer 1011, and the segment buffer 1011 can transmit the buffered frame images along with the predicted encoding option to the encoder 1030. .

At this time, when applying encoding optimization technology through segment-level video analysis to single encoding, system utilization may decrease because the encoder must wait during the analysis (inference) time. The encoding process is a very intensive process, and a decrease in the encoder's system utilization can have a significant impact on the encoding performance time.

Therefore, in order to prevent system utilization from decreasing when performing inference, the encoding pipeline uses two or more buffers to continuously store decoded video frames in the other buffer when one buffer is inferred. Movement can be performed without interruption.

Figure 11 is a diagram illustrating an example of a segment-level image analysis process using two buffers, according to an embodiment of the present invention. The segment analyzer (1110) can correspond to the AI serving module (320) described above, and the segment analyzing module (1113) included in the segment analyzer (1110) is an encoding option prediction model (321). can respond. At this time, one segment analysis module 1113 can process frame images stored in each of the two segment buffers 1111 and 1112.

First, storage of frame images of the decoded video frame may begin in segment buffer 1 (1111). At this time, when frame images for the first segment are stored in segment buffer 1 (1111), the segment analysis unit 1110 changes the storage buffer to segment buffer 2 (1112) and performs an analysis process (inference). . For example, the segment analysis module 1113 can predict encoding options using frame images stored in segment buffer 1 (1111). During this analysis process, frame images for the second segment may be continuously stored in segment buffer 2 1112. Thereafter, when the prediction of the encoding options for the frame images stored in segment buffer 1 (1111) is completed, the segment analysis module 1113 may transfer the predicted encoding options to segment buffer 1 (1111), and segment buffer 1 ( 1111) may transmit predicted encoding options and stored frame images to the encoder 1130.

Thereafter, when frame images for the second segment are stored in segment buffer 2 (1112), the segment analysis module 1113 may perform inference on the frame images stored in segment buffer 2 (1112). At this time, since the frame images of segment buffer 1 (1111) have been transferred to the encoder 1130, storage of frame images for the third segment may begin in segment buffer 1 (1111). If frame images for the first segment remain in segment buffer 1 (1111) due to inference delay, the analysis process must wait until all frame images for the first segment are delivered to the encoder (1130). You can. However, this situation rarely occurs. In this way, by alternately storing frame images using the two segment buffers 1111 and 1112, the transcoding pipeline can be prevented from being interrupted during inference.

Through the above explanation, it will be easy to understand that three or more segment buffers may be used depending on the situation. The operation method is the same as when using two segment buffers, and three or more segment buffers can sequentially store frame images for each segment. The analysis process using two or more segment buffers is suitable for non-real-time encoding that must maintain high throughput. In addition, the analysis process using two or more segment buffers is applied to a single encoding method that encodes the entire image with one encoding process. During single encoding, the encoder uses the encoding option (value of CRF) transmitted by the segment analysis unit 1110. ) can be adjusted to handle encoding.

Figure 12 is a flowchart showing an example of an encoding optimization method according to an embodiment of the present invention. The encoding optimization method according to this embodiment may be performed by the computer device 200. At this time, the processor 220 of the computer device 200 may be implemented to execute control instructions according to the code of an operating system included in the memory 210 or the code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform steps 1210 to 1270 included in the method of FIG. 12 according to control instructions provided by code stored in the computer device 200. can be controlled.

In step 1210, the computer device 200 may store frame images corresponding to the first segment of the input image in the first segment buffer. For example, assuming that the segment length is 4 seconds and the inference interval is 250 ms, the computer device 200 can store 15 frame images extracted at 250 ms intervals from the first segment in the first segment buffer. .

In step 1220, in response to completion of storing frame images corresponding to the first segment in the first segment buffer, the computer device 200 changes the storage buffer to the second segment buffer to store frame images corresponding to the second segment. It can be stored in the second segment buffer. For example, the computer device 200 may store 15 frame images extracted at 250 ms intervals corresponding to the second segment in the second segment buffer. Saving the frame images to the second segment buffer may also occur during the process of the computer device 200 predicting the encoding option in step 1230.

In step 1230, the computer device 200 may predict encoding options for frame images stored in the first segment buffer. For example, the computer device 200 may predict encoding options corresponding to features of the input image using an artificial intelligence model. Here, the artificial intelligence model can correspond to the encoding option prediction model 321 described above. Since the encoding option prediction model 321 and the method of predicting the encoding option using the encoding option prediction model 321 have been described in detail previously, repeated explanations will be omitted.

In step 1240, the computer device 200 may transmit the predicted encoding option and frame images stored in the first segment buffer to the encoder. The encoder can process encoding by applying these predicted encoding options.

In step 1250, the computer device 200 stores the frame images corresponding to the second segment in the second segment buffer, and in response to the frame images stored in the first segment buffer being transferred to the encoder, the storage buffer is stored in the first segment buffer. By changing to a segment buffer, frame images corresponding to the third segment can be stored in the first segment buffer. At this time, if there are still frame images to be transmitted to the encoder in the first segment buffer, the computer device 200 waits until all frame images stored in the first segment buffer are transmitted to the encoder, and then frames corresponding to the third segment. Images may be stored in the first segment buffer.

Meanwhile, as described above, three or more segment buffers may be used. In this case, the computer device 200 changes the storage buffer from the second segment buffer to the third segment buffer in step 1250, thereby storing frame images corresponding to the third segment in the first segment buffer and storing them in the third segment buffer. It can be stored in a buffer. If three segment buffers are used, the storage buffer can then be changed from the third segment buffer back to the first segment buffer, and if four segment buffers are used, the storage buffer can be changed from the third segment buffer to the fourth segment buffer. So, it can be easily understood that after that, it will sequentially change from the fourth segment buffer to the first segment buffer.

In step 1260, the computer device 200 may predict encoding options for frame images stored in the second segment buffer. Since the frame images of the next segment are stored in the second segment buffer while predicting the encoding option for the frame images stored in the first segment buffer, the encoding option prediction model 321 is used until the frame images are stored in the segment buffer again. You can immediately process the analysis of frame images of the next segment without having to wait.

In step 1270, the computer device 200 may transmit the predicted encoding option and frame images stored in the second segment buffer to the encoder. Likewise, the encoder can process encoding of frame images according to the passed encoding options.

As such, according to embodiments of the present invention, compression efficiency can be improved by finding optimal encoding parameters for each section of the image using artificial intelligence (AI) technology. Additionally, throughput can be improved by using double buffering in a single encoding structure.

The system or device described above may be implemented with hardware components or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments and equivalents of the claims also fall within the scope of the following claims.

Claims (15)

An encoding optimization method executed on a computer device comprising at least one processor, comprising:
storing frame images corresponding to a first segment of an input image in a first segment buffer, by the at least one processor;
In response to completion of storage of frame images corresponding to the first segment in the first segment buffer, the at least one processor changes the storage buffer to a second segment buffer to store frame images corresponding to the second segment. storing in a second segment buffer;
predicting, by the at least one processor, encoding options for frame images stored in the first segment buffer; and
Transferring, by the at least one processor, the predicted encoding option and frame images stored in the first segment buffer to an encoder.
Encoding optimization method including. According to paragraph 1,
Storage of frame images corresponding to the second segment in the second segment buffer is completed by the at least one processor, and the storage buffer is opened in response to the frame images stored in the first segment buffer being transferred to the encoder. Changing to the first segment buffer and storing frame images corresponding to the third segment in the first segment buffer.
An encoding optimization method further comprising: According to paragraph 2,
The step of storing frame images corresponding to the third segment in the first segment buffer includes:
If there are still frame images to be transmitted to the encoder in the first segment buffer, wait until all frame images stored in the first segment buffer are transmitted to the encoder, and then send the frame images corresponding to the third segment to the encoder. An encoding optimization method characterized by storing in a first segment buffer. According to paragraph 1,
When the storage of frame images corresponding to the second segment in the second segment buffer is completed by the at least one processor and the frame images stored in the first segment buffer are transmitted to the encoder, a storage buffer is generated. Changing to a 3-segment buffer and storing frame images corresponding to the third segment in the third segment buffer.
An encoding optimization method further comprising: According to paragraph 1,
predicting, by the at least one processor, encoding options for frame images stored in the second segment buffer; and
Transferring, by the at least one processor, the predicted encoding option and the frame images stored in the second segment buffer to an encoder.
An encoding optimization method further comprising: According to paragraph 1,
The step of predicting the encoding option is,
An encoding optimization method characterized by predicting encoding options corresponding to features of an input image using an artificial intelligence model. According to clause 6,
The artificial intelligence model includes a convolution neural network (CNN) model for extracting frame features for each frame image, a recurrent neural network (RNN) model for extracting video features based on relationships between the frame images, and the video features. An encoding option prediction model that includes a classifier that classifies the encoding option corresponding to the CNN model, the RNN model, and the classifier are learned in an E2E (end-to-end) manner for one loss function.
An encoding optimization method characterized by According to paragraph 1,
The step of predicting the encoding option is,
An encoding optimization method characterized by predicting a CRF (Constant Rate Factor) that satisfies a target VMAF (Video Multi-method Assessment Fusion) score as the encoding option. According to paragraph 1,
The step of predicting the encoding option is,
predicting a first CRF that satisfies a target VMAF score as the encoding option;
predicting a second CRF that satisfies a target bit rate as the encoding option; and
Determining a third CRF to be actually applied to encoding the input image using the first CRF and the second CRF
An encoding optimization method comprising: A computer program stored in a computer-readable recording medium to cause the computer device to execute the encoding optimization method of any one of claims 1 to 9. In computer devices,
At least one processor implemented to execute readable instructions on the computer device
Including,
By the at least one processor,
Store frame images corresponding to the first segment of the input image in a first segment buffer,
In response to completion of storing frame images corresponding to the first segment in the first segment buffer, change the storage buffer to a second segment buffer to store frame images corresponding to the second segment in the second segment buffer,
Predict encoding options for frame images stored in the first segment buffer,
Transferring the predicted encoding option and frame images stored in the first segment buffer to the encoder.
A computer device characterized by a. According to clause 11,
By the at least one processor,
In response to the completion of storage of frame images corresponding to the second segment in the second segment buffer and the frame images stored in the first segment buffer being transmitted to the encoder, the storage buffer is changed to the first segment buffer to three Storing frame images corresponding to the th segment in the first segment buffer
A computer device characterized by a. According to clause 11,
By the at least one processor,
In response to the storage of frame images corresponding to the second segment in the second segment buffer being completed and the frame images stored in the first segment buffer being transferred to the encoder, the storage buffer is changed to the third segment buffer to create the third segment buffer. Storing frame images corresponding to segments in the third segment buffer
A computer device characterized by a. According to clause 11,
By the at least one processor,
Predict encoding options for frame images stored in the second segment buffer,
Transferring the predicted encoding option and frame images stored in the second segment buffer to the encoder.
A computer device characterized by a. According to clause 11,
to predict the encoding option by the at least one processor,
Predicting encoding options corresponding to features of the input image using an artificial intelligence model
A computer device characterized by a.