KR102472971B1 - Method, system, and computer program to optimize video encoding using artificial intelligence model - Google Patents

Abstract

A video encoding optimization method, system, and computer program using an artificial intelligence model are disclosed. An encoding optimization method may include predicting an encoding option corresponding to a feature of an input image using an artificial intelligence model; and performing encoding on the input video according to the encoding option.

Description

Video encoding optimization method, system, and computer program using artificial intelligence model {METHOD, SYSTEM, AND COMPUTER PROGRAM TO OPTIMIZE VIDEO ENCODING USING ARTIFICIAL INTELLIGENCE MODEL}

The description below relates to video encoding technology.

Recently, multimedia information in which images and sounds are combined with communication and computers to become a new medium is provided. For example, as a high-speed data transmission network is supplied, stereophonic sound and high-definition images can be viewed, and users can make face-to-face conversations through video calls. In addition, products can be purchased while viewing product information in real time through a computer or TV, and music or movies can be enjoyed through a website. In addition, it is possible to attend video lectures through a computer.

Such multimedia information has been developed based on video compression (ie, encoding) technology. Data conveying information can be compressed by removing redundant elements from the data (elements that are not essential to accurately recover the data). In the case of lossy compression, the data recovered by the decoder is not the same as the original data, but subjective redundant elements are removed to obtain high compression efficiency. In image or video compression, the subjective redundancy factor is a factor that can be removed that does not significantly affect the picture quality intuitively felt by the viewer.

As an example of video compression technology, Korean Patent Registration No. 10-1136858 (registered on April 9, 2012) discloses an encoding technology in a video compression standard.

Provides encoding optimization technology that can improve compression efficiency by finding optimal encoding parameters for each section of video using artificial intelligence (AI) technology.

An encoding optimization technology capable of maintaining high image quality while reducing data costs borne by a user is provided.

It provides encoding optimization technology to improve bitrate along with user experience.

A method for optimizing encoding executed in a computer device, the computer device comprising at least one processor configured to execute computer readable instructions contained in a memory, wherein the method for optimizing encoding is performed by the at least one processor, predicting an encoding option corresponding to a feature of an input image using an intelligent model; and performing encoding on the input video according to the encoding option by the at least one processor.

According to one aspect, in the predicting of the encoding option, an encoding option corresponding to a feature of a corresponding section may be predicted for each section divided from the input image.

According to another aspect, the artificial intelligence model is a convolution neural network (CNN) model for extracting frame features for each frame image, and a recurrent neural network (RNN) model for extracting video features based on a relationship between the frame images. , and an encoding option prediction model including a classifier for classifying an encoding option corresponding to the video feature, wherein the CNN model, the RNN model, and the classifier are end-to-end (E2E) for one loss function. can be learned in this way.

According to another aspect, the artificial intelligence model may be configured as a single model using a data set in which encoding options satisfying the same standard for serviceable resolution are grouped with labels of the same category.

According to another aspect, predicting the encoding option may include predicting a constant rate factor (CRF) that satisfies a target video multi-method assessment fusion (VMAF) score as the encoding option.

According to another aspect, predicting the encoding option may include predicting a first CRF that satisfies a target VMAF score as the encoding option; predicting a second CRF that satisfies a target bit rate as the encoding option; and determining a third CRF to be actually applied to encoding of the input image using the first CRF and the second CRF.

According to another aspect, the determining of the third CRF may include determining the third CRF in consideration of a weight related to a bit rate limit, which is a user input value.

According to another aspect, the determining of the third CRF may include determining the first CRF as the third CRF if the first CRF is greater than or equal to the second CRF; and determining the third CRF using a weight related to bitrate limitation when the first CRF is smaller than the second CRF.

According to another aspect, the predicting of the encoding option may include predicting a QP restriction parameter for limiting a quantization parameter (QP) value of a block of a frame image to a target level as the encoding option based on the third CRF. may further include.

According to another aspect, the step of estimating the QP limiting parameter may include calculating a minimum QP value and a maximum QP value by applying an offset corresponding to the QP limiting parameter to the third CRF. have.

A computer program stored in a computer readable recording medium is provided to execute the encoding optimization method on the computer device.

A computer device comprising at least one processor configured to execute computer readable instructions included in a memory, wherein the at least one processor predicts an encoding option corresponding to a feature of an input image using an artificial intelligence model and , It provides a computer device characterized in that performing encoding on the input image according to the encoding option.

According to embodiments of the present invention, video compression efficiency can be improved by finding and applying optimal encoding parameters for each section of a video based on artificial intelligence (AI) technology to encoding.

According to the embodiments of the present invention, by securing video compression efficiency according to an encoding option optimized through an AI model, user experience and bitrate improvement can be guaranteed, storage space for video services is secured, and user of network cost can be reduced.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining the internal configuration of the electronic device and the server in one embodiment of the present invention.
3 is a diagram illustrating an example of components of a distributed encoding system according to an embodiment of the present invention.
4 is a diagram for explaining the basic concept of an encoding option prediction model for encoding optimization according to an embodiment of the present invention.
5 illustrates a video encoding optimization method according to an embodiment of the present invention.
6 and 7 are exemplary diagrams for explaining an image quality measurement index for each resolution in an embodiment of the present invention.
8 is an exemplary diagram for explaining a process of generating label data for model learning in one embodiment of the present invention.
9 is an exemplary diagram for explaining a process of predicting an additional encoding option considering service constraints according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to video encoding optimization technology using an artificial intelligence model.

Embodiments including those specifically disclosed in this specification can predict optimal encoding options using artificial intelligence models, thereby achieving significant advantages in various aspects such as compression efficiency, cost reduction, resource conservation, and service quality. can do.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , and 140 , a plurality of servers 150 and 160 , and a network 170 . 1 is an example for explanation of the invention, and the number of electronic devices or servers is not limited as shown in FIG. 1 .

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals implemented as computer systems or mobile terminals. Examples of the plurality of electronic devices 110, 120, 130, and 140 include a smart phone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcast terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), tablet PC, game console, wearable device, internet of things (IoT) device, virtual reality (VR) device, augmented reality (AR) device, and the like. As an example, FIG. 1 shows the shape of a smartphone as an example of the electronic device 110, but in the embodiments of the present invention, the electronic device 110 substantially uses a wireless or wired communication method to transmit other information via the network 170. It may refer to one of various physical computer systems capable of communicating with the electronic devices 120 , 130 , and 140 and/or the servers 150 and 160 .

The communication method is not limited, and includes not only a communication method utilizing a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 170 may include, but also short-range wireless communication between devices. can For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network 170 may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110, 120, 130, and 140 through the network 170 to provide commands, codes, files, contents, services, and the like, or a computer device or a plurality of computers. Can be implemented in devices. For example, the server 150 may be a system that provides a first service to a plurality of electronic devices 110, 120, 130, and 140 accessed through the network 170, and the server 160 may also include a network ( It may be a system that provides a second service to a plurality of electronic devices 110, 120, 130, and 140 accessed through 170). As a more specific example, the server 150 provides a service (eg, a video service, etc.) for which the application is intended through an application as a computer program that is installed and driven in the plurality of electronic devices 110, 120, 130, and 140. As the first service, it may be provided to a plurality of electronic devices 110, 120, 130, and 140. As another example, the server 160 may provide a service for distributing files for installing and running the above-described application to the plurality of electronic devices 110, 120, 130, and 140 as a second service.

2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention. Each of the plurality of electronic devices 110 , 120 , 130 , and 140 or each of the servers 150 and 160 described above may be implemented by the computer device 200 shown in FIG. 2 .

As shown in FIG. 2 , the computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210 . Also, an operating system and at least one program code may be stored in the memory 210 . These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on a computer program installed by files received over network 170 .

The processor 220 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 220 by memory 210 or communication interface 230 . For example, processor 220 may be configured to execute received instructions according to program codes stored in a recording device such as memory 210 .

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, storage devices described above) through the network 170 . For example, a request, command, data, file, etc. generated according to a program code stored in a recording device such as the memory 210 by the processor 220 of the computer device 200 is controlled by the communication interface 230 to the network ( 170) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170 . Signals, commands, data, etc. received through the communication interface 230 may be transferred to the processor 220 or the memory 210, and files, etc. may be stored as storage media that the computer device 200 may further include (described above). permanent storage).

The input/output interface 240 may be a means for interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 240 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 250 and the computer device 200 may be configured as one device.

Also, in other embodiments, computer device 200 may include fewer or more elements than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer device 200 may be implemented to include at least some of the aforementioned input/output devices 250 or may further include other components such as a transceiver and a database.

Hereinafter, specific embodiments of a method and system for optimizing video encoding using an artificial intelligence model will be described.

One of the video compression algorithms uses a method of increasing the compression rate by giving an original video a loss that is difficult for humans to perceive in consideration of human visual characteristics. The image compression rate can be adjusted by adjusting the degree of loss.

Commercial video compressors (codecs) provide a bitrate control function to maintain the bitrate (video size per second) that does not cause a problem in streaming without significantly degrading the video quality through lossy compression. .

Commonly used video codecs provide a bit rate control function that maintains a fixed target bit rate or maintains a constant picture quality. Here, maintaining a constant picture quality means that the picture quality of the encoding result is uniform, and the level of the picture quality cannot be known before encoding.

However, since complexity and characteristics are different for each image, if all images are compressed with the same target bit rate or quality, the result is that the image is compressed to a higher quality than human perception and to a larger size than necessary.

In addition, since a video often consists of several scenes with different complexity and characteristics, the above problem may occur depending on the section of the video.

The present embodiments are intended to optimize the compression rate of a video by determining an appropriate bit rate or an appropriate quality option according to the characteristics of each section of the video through AI technology.

3 is a diagram illustrating an example of components of a distributed encoding system according to an embodiment of the present invention.

Referring to FIG. 3, the distributed encoding system 300 is an optimized encoder component for application to the video platform 330, and includes a distributed encoder 310 and an AI serving module 320. can include

Each of the distributed encoder 310 or AI serving module 320 may be implemented by the computer device 200 described with reference to FIG. 2 .

In a video service, a video to be served can be uploaded to the video platform 330 and encoded through the distributed encoding system 300 . Whether to optimize encoding options is determined by the video platform 330, and parameters such as model information may be added to the encoding request when optimization is applied.

The distributed encoder 310 may include a distributor 311 and a worker 312 .

The distributed processing unit 311 divides the original video into several segments and allocates them to a plurality of workers 312 . While encoding is progressing in the worker 312, a compressed bitstream is received, and the received bitstream is temporarily stored in a local storage or managed while being loaded into a memory. The distributed processing unit 311 may generate a final image file by merging encoding results of the worker 312 for each segment.

The worker 312, as a unit transcoder for performing the encoding task, serves to encode the video segment allocated from the distributed processing unit 311. At this time, the worker 312 may operate in direct association with the AI serving module 320 for optimizing the encoding option. If encoding option optimization is activated, some frame images necessary for encoding option prediction may be prepared, and the prepared frame image along with model information may be transmitted to the AI serving module 320 to request an optimal encoding option. The worker 312 performs encoding by applying the prediction result of the AI serving module 320 and checks the encoding result quality and transmission bit rate. The worker 312 interpolates a prediction error based on an experimental value when the encoding result quality or transmission bit rate is out of an appropriate range, and then performs encoding again. Simultaneously with encoding, the compressed bitstream is transmitted to the distribution processing unit 311, and the prediction result of the model and encoding result data are stored in the video platform 330.

The AI serving module 320 operates in conjunction with the distributed encoder 310 and includes an encoding option prediction model 321. When receiving an encoding option request for encoding a video segment from each worker 312, the AI serving module 320 predicts the optimal encoding option through the encoding option prediction model 321 specified in the received request and predicts the predicted result. may be returned to the worker 312.

The video platform 330 may collect logs about prediction accuracy of the encoding option prediction model 321 based on a cloud search such as ELK (Elastic Logstash Kibana). In other words, the video platform 330 monitors the accuracy of the encoding option prediction model 321 by recording the encoding result of the worker 312 as a log, and monitors the accuracy of the monitoring result and the user (eg, administrator, etc.) 340. Based on the input value, it can support model additional learning and prediction accuracy improvement through new images.

In this embodiment, the encoding option prediction model 321 is an AI-based prediction model that determines characteristics of each segment of an image and predicts an optimal encoding option accordingly.

An encoding option means a parameter capable of adjusting an encoding rate, that is, an image compression rate, and, for example, a constant rate factor (CRF) can be used as an optimization encoding option.

Compared to CQP (constant quantization parameter), CRF guarantees a visually more uniform picture quality and can reflect human perceptual characteristics.

The basic concept of the encoding option prediction model 321 is as shown in FIG. 4 .

Referring to FIG. 4 , the encoding option prediction model 321 may receive images 401 of frames constituting an image and extract features of the image through a deep learning model. In this case, the deep learning model may include a convolution neural network (CNN) model and a recurrent neural network (RNN) model. The CNN model serves to extract features of frame images, and the RNN model serves to learn relationships between data (features of frame images) sequences.

The encoding option prediction model 321 is trained to classify an optimal CRF class for each feature extracted through a deep learning model by supervised learning.

When a frame image of a new video is input to the encoding option prediction model 321 learned in this way, an optimal CRF category suitable for the characteristics of the corresponding image can be predicted.

5 illustrates a video encoding optimization method according to an embodiment of the present invention.

Referring to FIG. 5 , the distributed encoding system 300 includes a learning process S510 and an inference process S520 for the encoding option prediction model 321 .

The encoding option prediction model 321 includes a CNN model, an RNN model, and a classifier.

First, the distributed encoding system 300 performs a pre-processing process. The pre-processing process includes a process of extracting frame images from video segments. The distributed encoding system 300 may optimize the frame image size, the number of frame images, and the like in order to prevent a memory problem in the learning process (S510) and increase accuracy.

A data set for learning the encoding option prediction model 321 may utilize a video segment data set created using a video uploaded to a video platform.

For example, the encoding optimization option may determine a range of CRFs to be learned by using CRFs and considering the distribution of data sets and effective ranges during actual encoding.

The frame image size can be increased from the initial 224×224 to 336×336, 448×448, 560×560, or more. In sizes below 336×336, the original video can be resized after 80% center-crop. It is resized so that the original information can be maintained as much as possible, but the aspect ratio of the original can be changed.

When the frame image (1920×1080) of the original video is resized to the above size, information loss inevitably occurs during the resizing process. In order to reduce this loss, increasing the image size as much as possible helps improve accuracy.

In the case of the number of frame images, 15 images at 250 msec intervals in 4 sec segment images can be used as model inputs to improve accuracy. The distributed encoding system 300 selects 10 images at intervals of 500 msec from a 5-second segment image to improve accuracy, and then resizes the selected images to a size of 336×336 that can be processed by CNN. In other words, 10 × (336 × 336 × 3) data for one segment image may be provided as an input to the encoding option prediction model 321 .

This is just an example, and the selection of an image to be used as a model input can be changed as much as you like. For example, it is also possible to select 20 images at intervals of 200 msec from a 4 second segment image.

Through repetitive encoding of each image, CRF options that satisfy the appropriate quality of VMAF (Video Multi-method Assessment Fusion) criteria are explored, and the collected 4-second segment images and GT (ground truth) data sets are used as an encoding option prediction model ( 321) can be used in the learning process (S510).

Next, the distributed encoding system 300 performs a video feature extraction process. Frame features for each frame image given as input to the encoding option prediction model 321 can be extracted from the CNN, and the frame features extracted from the CNN are input to the RNN (LSTM) in image order based on the relationship information between feature sequences. to extract video features.

Finally, the distributed encoding system 300 performs a classification process, and may classify the video features extracted from the RNN using, for example, a softmax classifier.

Accordingly, the distributed encoding system 300 may perform fine-tuning up to the CNN model and learn the entire model including the CNN model for one loss function in an end-to-end (E2E) manner.

For image classification performance, weights of the CNN learned in the learning process (S510) may be optimized to distinguish image characteristics well. A supervised learning method can build a model that classifies the optimal option according to the characteristics of an image by learning the characteristics of an image, that is, the relationship between video features and encoding options (CRFs).

When a frame image of a new video is input to the encoding option prediction model 321 learned through the learning process (S510), an optimal CRF category suitable for the characteristics of the image can be predicted as an inference process (S520).

In particular, in the present embodiments, the encoding option prediction model 321 may be built as a single model to predict target image quality achievement parameters of images of various resolutions.

In a streaming service, one video is provided in multiple resolutions, and each resolution must be encoded with a target quality. Encoding parameters for encoding with a target image quality can be derived through image analysis using AI, and the derived result may be different for each resolution. In this case, it is common to separately operate a model for each resolution.

For example, when encoding 1080p and 720p videos, a model for 1080p and a model for 720p are created respectively, and each video is inferred as a corresponding model. In this way, as the number of models increases in a service environment supporting various resolutions, it is difficult to develop, manage, and maintain models.

In general, VMAF is used as a quality measurement index for streaming services. VMAF is an image quality index considering both compression artifacts and scaling artifacts.

FIG. 6 is a general VMAF measurement method, which is a result of resizing all encoded resolutions to 1080p and comparing them with the original 1080p.

By obtaining the convex-hull of the overlapped VMAF curves and selecting a point on the convex-hull, it is possible to select the optimal resolution for the corresponding bit rate. However, since the VMAF range of each resolution is different, it is not possible to obtain the same VMAF criteria for each resolution.

7 compares the encoding result of each resolution with the original video before encoding of the same resolution, and is a result of considering only compression artifacts excluding scaling artifacts.

It cannot make convex-hull, but it can take the same VMAF criterion for full resolution. In other words, when obtaining a label used for learning of the encoding option prediction model 321, the same quality score (eg, VMAF 93) for the entire resolution may be taken as a GT value.

In this way, if a quality encoding parameter (fixed quality encoding parameter such as CRF, QP (quantization parameter), etc.) value that satisfies the same standard (VMAF score) for all resolutions is used as a label, CRFs can be grouped into the same category.

In other words, if CRF values satisfying the same VMAF criterion are obtained as labels for each resolution, images having the same label can be bundled with data of the same label even if the resolutions are different. For example, since 1080p CRF 23 and 720p CRF 23 can be grouped into the same category, when configuring a data set for learning, a data set of the CRF 23 category can be configured by mixing 1080p and 720p images.

If each label category is configured and trained with a data set that mixes multiple resolutions, a single model can be used for encoding of multiple resolutions. CRFs for 1080p and 720p resolution encoding can be predicted with one model.

The method to create label data is as follows.

The optimal CRF value of each segment image is used as a label.

Referring to FIG. 8 , the distributed encoding system 300 repeats encoding with different CRF values for the segment image 801 (S81), and measures the quality of the encoding result (S83) to determine whether a certain level of quality is satisfied. It is determined whether or not (S83).

The distributed encoding system 300 may obtain a label 802 by finding a CRF value that satisfies a certain level of image quality through the above-described steps S81 to S83.

The criterion for image quality uses VMAF that best reflects human perception, and may be based on, for example, VMAF 93 suitable for streaming services. Each segment image 801 is encoded while changing the CRF until the VMAF value reaches 93, and the CRF value at the time when the VMAF value reaches 93 is selected as the label 802 .

For example, label generation process for 1080p and 720p resolution encoding is as follows.

For the 1080p resolution encoding label, a label CRF is obtained based on the original 1080p based on VMAF 93.

In the case of a 720p resolution encoding label, first, the original 1080p is resized to 720p, and the label CRF is obtained while repeating encoding by changing the CRF based on the resized 720p. The picture quality standard is VMAF 93, the same as 1080p. At this time, when VMAF is obtained, the image quality comparison original is an image obtained by resizing 1080p to 720p. That is, the VMAF is measured by considering only the compression artifact without considering the resize artifact, and a corresponding CRF is obtained. A CRF that satisfies a specific image quality (VMAF) considering only compression artifacts is obtained for each resolution. At this time, since CRF categories can be grouped into several resolutions, a data set of each category can be created by grouping 1080p and 720p resolutions.

Therefore, a single model supporting both 1080p and 720p resolutions can be built with a mixed data set of 1080p and 720p, and through this, optimal CRF results can be obtained as inference results for 1080p segment images and 720p segment images. have.

In the process of generating label data, for example, when encoding is performed while changing the CRF by 1 unit, since the CRF closest to VMAF 93 is selected, there is a problem in that the accuracy of the label is lowered.

In order to solve this problem, label accuracy may be improved by applying a label class with a CRF of a unit smaller than 1, for example, a unit of 0.5 through a test for the CRF unit.

Therefore, by using the encoding option prediction model 321 built for CRF prediction, encoding can be optimized and an iterative process can be minimized.

Furthermore, the present embodiments can predict an effective target picture quality achievement parameter under service constraints.

When compressing video for streaming services, it is necessary to limit the bit rate to ensure uninterrupted viewing along with minimization of video quality degradation. Therefore, it is necessary to generate a resulting image within the bitrate constraint, and the bitrate constraint and the importance of the constraint are different for each service. It is also important to produce high-quality results at the same bit rate.

9 illustrates an example of a process of deriving picture quality parameters to be applied to actual encoding in consideration of service constraints in an embodiment of the present invention.

Referring to FIG. 9 , the distributed encoding system 300 extracts features of an input image 901 (S900) and then predicts encoding options optimized for the extracted features (S901 to S904).

In the process of predicting encoding options, the remaining steps (S901, S902, and S903) except for step S903 use the AI-based encoding option prediction model 321.

In detail, the distributed encoding system 300 may predict a first CRF (CRF1), which is a picture quality encoding parameter that satisfies a target picture quality (eg, VMAF 93), as an encoding option corresponding to a feature of the input image 901. (S901). Since the method of predicting the first CRF is the same as described above, a detailed description thereof will be omitted.

The distributed encoding system 300 may predict a second CRF (CRF2), which is a picture quality encoding parameter that satisfies a target bit rate, as an encoding option corresponding to a feature of the input image 901 (S902). As an image quality control parameter, a CRF option that shows a uniform image quality for each section is generally used, but when encoding with CRF, the bit rate of the encoded video is unknown. In order to find a CRF that satisfies the target bit rate, it is necessary to check the resulting bit rate after performing iterative encoding while adjusting the CRF value, which incurs a very large encoding cost. In order to satisfy the target picture quality and service bit rate limitations with low encoding cost, a second CRF prediction satisfying the target bit rate is required. If a second CRF that satisfies the target bitrate is predicted together with the first CRF that satisfies the target picture quality, it is possible to reduce costs and improve service quality.

For the second CRF prediction, a CRF value satisfying a specific bit rate of each segment image is used as a label. A CRF value satisfying a target bit rate may be found by encoding the CRF value differently for each segment image, and the corresponding CRF value may be selected as a label. The target bitrate may be different for each application, and for example, label data such as 2nd CRF 23 for category 1, 2nd CRF 23.5 for category 2, and 2nd CRF 24 for category 3 can be generated.

Depending on embodiments, it is also possible to generate label data by combining the first CRF and the second CRF. For example, label data is generated such as 1st CRF 23 and 2nd CRF 24 in case of category 1, 1st CRF 23 and 2nd CRF 25 in case of category 2, 1st CRF 23 and 2nd CRF 26 in case of category 3, etc. can do. When label data in which the first CRF and the second CRF are combined are used, steps S902 and S903 are omitted in FIG. 9 and the predicted CRF in step S901 is used as the final CRF to be applied to encoding. can

The encoding option prediction model 321 receives the images of the frames constituting the input image 901, extracts image features through deep learning (CNN and RNN), and classifies them into classes corresponding to each CRF using the extracted features. It is learned through supervised learning to At this time, the correct answer label of the CRF class becomes the second CRF generated from the label data above. When label data is generated by combining the first CRF and the second CRF, the correct answer label of the CRF class is composed of a combination of the first CRF and the second CRF. When frames of an image to be analyzed are input to the encoding option prediction model 321 learned with the label data set, a class corresponding to a CRF satisfying a bit rate may be output.

The distributed encoding system 300 may determine a third CRF (CRF3) as a final CRF to be applied to actual encoding by considering the first CRF that satisfies the target picture quality and the second CRF that satisfies the target bit rate (S903). For example, the distributed encoding system 300 may determine the first CRF as the third CRF when the first CRF is greater than or equal to the second CRF. Meanwhile, when the first CRF is smaller than the second CRF, the distributed encoding system 300 may determine a third CRF in consideration of a bitrate restriction compliance weight, which is a user input value related to model learning (S903). In an application requiring strict bit rate constraints, when the first CRF is smaller than the second CRF, the second CRF may be determined as the third CRF. In applications with relatively less stringent bit rate constraints, a value closer to the first CRF may be selected for picture quality gain. At this time, the distributed encoding system 300 may determine the third CRF by correcting the CRF value using the bit rate restriction compliance weight w, which is a user input value (Equation 1).

[Equation 1]

CRF3 = CRF1×(1-w)+CRF2×w

The bitrate limit compliance weight may be input as a value within 0 to 1.0. When the weight is 0, the first CRF is determined as the third CRF without complying with the bitrate limit, and when the weight is 1, the second CRF is determined as the third CRF to strictly comply with the bitrate limit.

The distributed encoding system 300 may predict a QP restriction parameter as an additional encoding option corresponding to a feature of an input image (S904).

CRF, one of the quality encoding parameters, determines the frame QP. The encoder is equipped with various block-level QP decision algorithms such as AQ (adaptive quantization) and MB (macroblock)-tree for improving perceived picture quality, and it is common to use a combination of CRF and corresponding encoding options. The QP determination algorithm determines the block QP by applying an offset value for each block to the frame QP determined from the CRF. At this time, a block to which an excessively low or high QP is allocated occurs. The encoder provides options such as minimum QP (minqp) and maximum QP (maxqp) to limit QP, and by using these options to limit the QP values of blocks to an appropriate level, additional bitrate reduction can be achieved while maintaining perceived picture quality. have.

When applied together with the third CRF, the distributed encoding system 300 may derive a minimum QP value and a maximum QP value capable of obtaining an optimal bit rate-to-image quality. The minimum QP serves to prevent excessively high picture quality and the maximum QP serves to prevent excessively low picture quality, and can achieve a visually more even picture quality by applying it in units of blocks. As an example, the distributed encoding system 300 may derive a minimum QP value and a maximum QP value by applying an offset to the third CRF (minqp=CRF3-offset, maxqp=CRF3+offset). For example, when the CRF offset is 2, a value obtained by subtracting 2 from CRF3 may be set as the minimum QP value, and a value obtained by adding 2 to CRF3 may be set as the maximum QP value.

The distributed encoding system 300 may apply the minimum QP value and the maximum QP value as encoding options together with the third CRF, and depending on the embodiment, the process of estimating the QP limiting parameter (S904) is omitted in order to reduce inference time. It is also possible to apply only the third CRF to actual encoding.

To predict the QP limiting parameter, a CRF offset that satisfies the optimal cost when combined with the encoding CRF (CRF3) determined for each segment is used as a label. Different CRF values are set for each segment image, and encoding is performed by setting different minimum QP and maximum QP (or offset) for each CRF value. After obtaining the bit rate and image quality (VMAF) of each encoding result, a cost is obtained from this, and an offset value of an image having the lowest cost may be selected as a label.

The cost can be defined as Equation 2

[Equation 2]

Cost = rate+λ×d

Here, the λ (lambda) value is determined as a user input value during model training depending on whether bitrate reduction or image quality preservation is prioritized.

The features of the image and the third CRF determined in the previous step are input values. For example, label data such as CRF offset 2 for category 1, CRF offset 4 for category 2, and CRF offset 6 for category 3 are generated. can Finally, the encoding application parameters are the third CRF, the minimum QP (CRF3-offset), and the maximum QP (CRF3 + offset).

In general, the lower the QP, the lower the image quality gain relative to the bit rate, so the offset can be applied asymmetrically. In other words, the minimum QP may be determined as (CRF3-offset + 1), and the maximum QP may be applied as (CRF3 + offset). For example, when the third CRF value is 26 and the offset is 4, the minimum QP value may be 23 and the maximum QP value may be 30.

The encoding option prediction model 321 receives the images of the frames constituting the input image, extracts features of the image through deep learning (CNN and RNN), and learns to classify the feature into an optimal cost class for each encoding CRF through supervised learning. do. At this time, the correct answer label of the CRF class is composed of the CRF offset generated as the label data. When the frame image of the video to be analyzed and the third CRF value are input to the encoding option prediction model 321 learned with the label data set, a CRF offset value can be obtained as a QP limiting parameter corresponding to the classification result category.

Therefore, in the present embodiment, in predicting the optimal encoding option to be applied to encoding using the AI-based encoding option prediction model 321, not only predicting the picture quality encoding parameter that satisfies the target picture quality, but also A picture quality encoding parameter satisfying a target bit rate and/or a QP limiting parameter satisfying a picture quality balance condition may be additionally predicted.

As described above, according to embodiments of the present invention, video compression efficiency can be improved by finding and applying optimal encoding parameters for each segment video based on AI technology to encoding. According to embodiments of the present invention, by securing video compression efficiency according to an encoding option optimized through an AI-based learning model, it is possible to guarantee improvement in user experience and bit rate, as well as securing storage space for video services. , can reduce the user's network cost.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable PLU (programmable logic unit). logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. have. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. In this case, the medium may continuously store a program executable by a computer or temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

An encoding optimization method executed on a computer device,
The computer device includes at least one processor configured to execute computer readable instructions contained in a memory;
The encoding optimization method,
predicting, by the at least one processor, an encoding option corresponding to a feature of an input image using an artificial intelligence model; and
Encoding, by the at least one processor, the input image according to the encoding option.
including,
Predicting the encoding option,
predicting a first constant rate factor (CRF) satisfying a target video multi-method assessment fusion (VMAF) score and a second CRF satisfying a target bit rate as the encoding option through the artificial intelligence model;
if the first CRF is greater than or equal to the second CRF, determining the first CRF as a third CRF to be actually applied to encoding of the input image; and
If the first CRF is smaller than the second CRF, determining the third CRF using a bitrate limit compliance weight, which is a user input value related to model learning, and the first CRF and the second CRF.
An encoding optimization method comprising a. According to claim 1,
Predicting the encoding option,
Predicting an encoding option corresponding to a feature of the corresponding section for each section divided from the input image
Encoding optimization method characterized by. According to claim 1,
The artificial intelligence model includes a convolution neural network (CNN) model for extracting frame features for each frame image, a recurrent neural network (RNN) model for extracting video features based on relationships between the frame images, and the video features. An encoding option prediction model including a classifier for classifying encoding options corresponding to , wherein the CNN model, the RNN model, and the classifier are trained in an end-to-end (E2E) method for one loss function
Encoding optimization method characterized by. According to claim 1,
The artificial intelligence model is composed of a single model using a data set in which encoding options satisfying the same standard for serviceable resolution are bundled with labels of the same category.
Encoding optimization method characterized by. delete delete delete delete According to claim 1,
Predicting the encoding option,
Predicting a QP limiting parameter for limiting a QP (quantization parameter) value of a block of a frame image as the encoding option based on the third CRF to a target level
Encoding optimization method further comprising. According to claim 9,
Predicting the QP limiting parameter,
Calculating a minimum QP value and a maximum QP value by applying an offset corresponding to the QP limiting parameter to the third CRF
An encoding optimization method comprising a. A computer program stored in a computer readable recording medium in order to execute the encoding optimization method of any one of claims 1 to 4, 9, and 10 in the computer device. In a computer device,
at least one processor configured to execute computer readable instructions contained in memory;
including,
The at least one processor,
Predicting encoding options corresponding to features of the input image using an artificial intelligence model,
Encoding the input video according to the encoding option;
The at least one processor,
Predicting a first CRF that satisfies a target VMAF score and a second CRF that satisfies a target bitrate as the encoding option through the artificial intelligence model,
If the first CRF is greater than or equal to the second CRF, determining the first CRF as a third CRF to be actually applied to encoding of the input video;
If the first CRF is smaller than the second CRF, determining the third CRF using a bitrate limit compliance weight, which is a user input value related to model learning, and the first CRF and the second CRF
Characterized by a computer device. According to claim 12,
The at least one processor,
Predicting an encoding option corresponding to a feature of the corresponding section for each section divided from the input image
Characterized by a computer device. According to claim 12,
The artificial intelligence model includes a CNN model for extracting frame features for each frame image, an RNN model for extracting video features based on the relationship between the frame images, and a classifier for classifying encoding options corresponding to the video features. As an encoding option prediction model that includes, the CNN model, the RNN model, and the classifier are learned in an E2E manner for one loss function
Characterized by a computer device. According to claim 12,
The artificial intelligence model is composed of a single model using a data set in which encoding options satisfying the same standard for serviceable resolution are bundled with labels of the same category.
Characterized by a computer device. delete delete delete According to claim 12,
The at least one processor,
Predicting a QP limiting parameter limiting a QP value of a block of a frame image as the encoding option based on the third CRF to a target level
Characterized by a computer device. According to claim 19,
The at least one processor,
Calculating a minimum QP value and a maximum QP value by applying an offset corresponding to the QP limiting parameter to the third CRF
Characterized by a computer device.

Images