KR102169073B1 - Real-time Streaming Blocking System and Method Based on the Results of the Video Obscenity Analysis - Google Patents

Abstract

Disclosed are a system for blocking streaming in real-time based on the result of video harmfulness analysis and a method thereof, wherein the system comprises: a video server in which a video is encoded and stored in the server in the shape of a segment having a fixed reproduction length and bitrates different from each other for HTTP adaptive streaming, and which provides real-time streaming video data; and a user terminal in which an HTTP adaptive streaming client is installed and which compares similarity among video frames by using a deep learning-based harmfulness analysis engine to calculate length information of the scene of each frame and harmfulness probability for each frame to determine whether to block streaming depending on the final harmfulness rating so as to block harmful videos. Therefore, harmful videos may be blocked during real-time streaming.

Description

Real-time Streaming Blocking System and Method Based on the Results of the Video Obscenity Analysis}

The present invention relates to a real-time streaming blocking system and method based on a video harmfulness analysis result. More specifically, in a VoD and real-time streaming service system to which HTTP adaptive streaming technology is applied, video frames are extracted from a video segment being streamed in real time, and Using a running-based hazard analysis engine, it compares the similarity between video frames with the original harmful image, calculates the length information of the scene of each frame and the harmful probability for each frame, and blocks harmful videos by deciding to block streaming according to the final harmful rating. , To a real-time streaming blocking system and method based on the result of harmfulness analysis of video.

In recent years, as the spread of computers and smart devices increases and decreases, harmful sites and videos that can be accessed by PC and smart devices are flooding the wired and wireless Internet, and guardians who need protection, such as children and teenagers, access harmful sites or adult videos. Or illegal videos are downloaded or exposed to harmful videos.

To solve this problem, it is necessary to block harmful sites for young children and adolescents and to block harmful videos.

As prior art 1 related to this, in Patent Publication No. 10-2016-0116585, "a method and apparatus for blocking a harmful area of a video" is disclosed.

The video harmful area blocking device determines whether the current frame of the input video is a harmful frame, detects a face and at least a part of a person other than the face with respect to the harmful frame, and then detects the detected area among the face and at least a part of the person. Masking is performed, and at least a portion of a face and a person is tracked by sequentially using information on at least a portion of a face and a person detected in a frame immediately preceding the current frame as an initial value to mask the tracked area.

As a related prior art 2, Patent Publication No. 10-2011-0120545 discloses a "computer harmful video blocking device and method" that allows the guardian to fundamentally and efficiently block access to harmful videos when using a computer. .

An apparatus for blocking a harmful video when a guardian uses a computer includes: a database storing permission file information for a video file permitted to be executed on the computer; An input module for receiving identification information of the computer user; A search module for comparing and searching the request file information for the video file requested for execution with the permission file information stored in the database when a request for execution of the video file stored in the computer is received from the guardian; And when permission file information corresponding to the requested file information is searched in the database by the search module, the video file requested for execution is executed, and permission file information corresponding to the requested file information is not searched in the database. In this case, it includes a control module to block execution of the video file requested for execution.

However, the existing streaming system calculates the final harmfulness level by considering the harmfulness analysis result of the video frame extracted from the real-time streaming video and the scene length information in which the video frame is repeated, and based on the calculated final harmfulness level, No harmful video blocking control was provided for the video.

Patent Publication No. 10-2016-0116585 (Publication date October 10, 2016), "Method and device for blocking harmful areas of video", Electronics and Telecommunications Research Institute Patent Publication No. 10-2011-0120545 (Publication date November 04, 2011), "A device and method for blocking harmful computer videos", Seungsoo Lee

An object of the present invention for solving the above problem is to calculate a final harmful rating by considering the scene length information in which a video frame is repeated similar to the result of a harmfulness analysis of a video frame extracted from a real-time streaming video, and calculate the calculated final harmful rating. It provides a real-time streaming blocking system based on the result of analyzing harmfulness of a video, which blocks harmful videos with respect to a video being streamed in real time.

Another object of the present invention is to provide a method for blocking real-time streaming based on a result of analyzing harmfulness of a video.

In order to achieve the object of the present invention, a real-time streaming blocking system based on a video harmfulness analysis result encodes video data in the form of segments having a fixed playback length and different bitrates for HTTP adaptive streaming, A video server that is stored in and provides real-time streaming video data; And HTTP adaptive streaming client is installed, and the similarity between video frames is compared with the original image using a deep learning-based harmfulness analysis engine, and the length information of the scene of each frame and the harmful probability for each frame are calculated and streamed according to the final harmfulness level. It includes a user terminal that blocks harmful images by determining blocking.

In order to achieve another object of the present invention, a method for blocking real-time streaming based on a result of a video harmfulness analysis includes: (a) a format of a segment having a fixed playback length and different bitrates for video data for HTTP adaptive streaming. Encoding and storing in a video server, and providing real-time streaming video data from the video server to a user terminal; And (b) an HTTP adaptive streaming client is installed in the user terminal, and the similarity between video frames is compared with the original image using a deep learning-based harmfulness analysis engine, and the length information of the scene of each frame and the harmful probability for each frame are calculated. Blocking harmful videos by calculating and determining streaming blocking according to the final harmful rating,
The user terminal is installed with the HTTP adaptive streaming client, downloads a Media Presentation Description (MPD) file describing information on a video to be requested from the video server, and a specific quality according to the measured network bandwidth and the state of the terminal ( bitrate) of the video segment,
The deep learning-based harmfulness analysis engine uses a CNN (Convolutional Neural Network) model, and compares the harmful original image with the comparison image by learning the characteristics of the input images for a given data set of the video segment, that is, the similarity between video frames. By comparing the harmful original image, the length information of the scene of each frame and the harmful probability for each frame are calculated, and the harmfulness of the video is analyzed and the result of the video harmfulness analysis in the form of a harmful probability for each video frame is blocked from video streaming. Provided as a module.

The real-time streaming blocking system and method based on the results of the video harmfulness analysis of the present invention analyzes the harmfulness of the video using the video frame extracted from the video being streamed in real time, and the length of the scene in which the harmfulness analysis result of the video and the extracted video frame are repeated. It is possible to perform meaningful blocking control in a real-time streaming service by controlling the streaming blocking of harmful videos by reflecting.

In a service platform that provides VoD and real-time streaming services applied with adaptive streaming technology, personal broadcasting and real-time streaming, the harmfulness of the video is analyzed by analyzing the harmfulness of the video using the video frame extracted based on the similarity comparison from the video being streamed. It is possible to perform meaningful blocking control in a real-time streaming service by minimizing the load that may occur and controlling the blocking of streaming by reflecting the result of the harmfulness analysis and the repeated scene length of the extracted video frame.

1 is a diagram showing the structure of a real-time streaming blocking system based on a video harmfulness analysis result.
2 is a diagram illustrating an example of a scene length analysis based on a similarity comparison between frames.
FIG. 3 is a diagram illustrating an example of performing a similarity comparison by calculating a Structural Similarity (SSIM) value for analyzing a scene length based on a similarity comparison between video frames in FIG. 2.
4 is a diagram showing an example of a process of calculating a final hazard level by using a hazard analysis result and scene length information.
FIG. 5 shows a process of reflecting the harmful rating for each video frame calculated by the harmfulness analysis and the repeated length of the scene corresponding to each video frame when calculating the final harmfulness rating.
6 shows the definition of each variable used to calculate the hazard level of FIG. 4.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

A system and method for controlling streaming blocking based on a result of analysis of harmfulness of a video being streamed in real time are provided.

The system extracts a video frame from a video segment transmitted from a video server to a user terminal for harmfulness analysis. The video frame to be extracted is determined to minimize the load that may occur in the hazard analysis through adaptive extraction based on similarity comparison. In addition, in order to determine an appropriate blocking time for a real-time streaming video, the system calculates a final harmfulness rating by considering the length information of a scene in which video frames similar to the result of the harmfulness analysis of the extracted video frame are repeated. The system performs blocking control on the video being streamed according to the calculated final harmful rating.

The real-time streaming blocking system based on the video harmfulness analysis result of the present invention

In VoD and real-time streaming service platforms with HTTP adaptive streaming technology,

A video server that encodes video data in the form of segments having a fixed playback length and different qualities for HTTP adaptive streaming and stores them in the server, and provides real-time streaming video data; And

An HTTP adaptive streaming client is installed, it downloads an MPD (Media Presentation Description) file describing information on the video to be requested from the video server, and generates a video segment of an appropriate bitrate according to the measured network bandwidth and the state of the terminal. Requested, and in the case of video harmfulness analysis, a deep learning-based harmfulness analysis engine is used, and the deep learning-based harmfulness analysis engine is basically composed of a CNN (Convolutional Neural Network) model and learns the characteristics of input images for a given data set. By doing so, it includes a user terminal that receives and plays the real-time streaming video data by classifying harmful images and non-hazardous images,
The user terminal is installed with the HTTP adaptive streaming client, downloads a Media Presentation Description (MPD) file describing information on a video to be requested from the video server, and a specific quality according to the measured network bandwidth and the state of the terminal ( bitrate) of the video segment,
The deep learning-based harmfulness analysis engine uses a CNN (Convolutional Neural Network) model, and compares the similarity between video frames with the original harmful image by learning the characteristics of input images for a given data set of a video segment. The length information and the harmful probability of each frame are calculated, the harmfulness of the video is analyzed, and the video harmfulness analysis result is provided to the video streaming blocking control module in the form of the harmfulness probability for each video frame.

The user terminal extracts a video frame from the video segment among the real-time streaming video, calculates the final harmfulness rating by considering the length information of the repeated video frames similar to the result of harmfulness analysis on the video frame, and calculates the final harmfulness rating. Accordingly, blocking control is performed on the video being streamed.

The real-time streaming blocking system based on the result of video harmfulness analysis operates based on HTTP (Hyper Text Transfer Protocol) adaptive streaming, a streaming technology mainly used when transmitting video over the Internet. HTTP adaptive streaming encodes and stores a video on a server in the form of segments having a fixed playback length and different quality. When streaming starts, the user terminal (HTTP adaptive streaming client) downloads an MPD (Media Presentation Description) file describing information on the video to be requested from the server, and a video of appropriate quality according to the measured network status and terminal status. Request a segment. In the case of video hazard analysis, a deep learning-based hazard analysis engine is used. The deep learning-based harmfulness analysis engine is basically composed of a CNN (Convolutional Neural Network) model, and operates in a manner that distinguishes harmful and non-hazardous images by learning the characteristics of input images for a given data set.

In the present invention, it is assumed that a harmful video refers to a video including violence or obscene that is inappropriate for minors or children, and the harmfulness analysis result can be obtained by transmitting a video frame extracted from a video segment as an input image to a harmfulness analysis engine.

In order to minimize the load of harmfulness analysis, the present invention relates to a technology that extracts only necessary information from the transmitted video segment and transmits it to the harmfulness analysis engine, and a method of determining real-time streaming blocking using the result of the harmfulness analysis. The image judgment mechanism is not described.

1 shows the structure of a real-time streaming blocking system based on a video harmfulness analysis result.

In this system, the user terminal extracts a video frame from the streamed video segment, learns the characteristics of the input images of each video frame, classifies a harmful image from a non-harmful image, analyzes the harmfulness of the video, and adds it to each video frame. A deep learning-based harmfulness analysis engine that provides results of harmfulness analysis on the disease; A video frame extraction module for extracting video frames excluding duplicated video frames to increase the efficiency of video harmfulness analysis; And a video streaming blocking control module that calculates a final harmful rating by using the video harmfulness analysis result and scene length information, and controls streaming blocking of the harmful video determined as the final harmful rating.

The deep learning-based harmfulness analysis engine uses a CNN (Convolutional Neural Network) model, and by learning the characteristics of input images for a given data set of a video segment, it classifies harmful and non-hazardous images to analyze the results of the video harmfulness analysis. It is provided as a video streaming blocking control module.

When a video segment is transmitted from the video server to the user terminal, the proposed system of the user terminal adaptively extracts the video frame so as to minimize the load of harmfulness analysis through similarity comparison between video frames. The extracted video frames are delivered to a deep learning-based harmfulness analysis engine.

The harmfulness analysis engine delivers the result of the harmfulness analysis in the form of a harmful probability for each frame through the harmfulness analysis of the input video frame to the video streaming blocking control module.

The video streaming blocking control module compares the similarity between video frames, calculates length information of each frame, and a harmful probability for each frame, and determines streaming blocking according to the calculated final harmfulness level.

2 shows an example of a scene length analysis based on a similarity comparison between frames. This system assumes an HTTP adaptive streaming environment in which a video is stored in a video server in the form of segments having a predetermined playback length of various qualities in order to adaptively adjust the requested video quality according to the network situation. The video segment includes a fixed number of video frames according to the frame rate of the video. The system analyzes the scene length of a frame by comparing the similarity between adjacent video frames. The embodiment shows a case where there are 4 video frames having a high similarity to Frame 1, and 3, 6, and 3 are respectively in Frames 2, 3, and 4. A video frame with a varying degree of similarity between frames serves as a criterion for dividing a scene, and if the scene is different, the result of harmfulness analysis is highly likely to be different. Therefore, the system determines the timing of blocking streaming in consideration of the scene length.

은 4이며,

(두 번째 추출된 동영상 프레임이 포함된 장면의 길이),

(세 번째 추출된 동영상 프레임이 포함된 장면의 길이), 그리고

(네 번째 추출된 동영상 프레임이 포함된 장면의 길이)의 값은 각각 3, 6, 3이다.A different number of video frames are included in the video segment according to the frame rate of the video. In the embodiment of the present invention, it is assumed that the frame rate is 30 and the playback length per video segment is fixed to 2 seconds. (That is, the total number of video frames included in the video segment is 60) The video segment may include one or two or more scenes in units of small video fragments having a playback length of about several seconds. The real-time streaming blocking system based on the result of the harmfulness analysis of the video extracts the video frames that are the basis of the scene change from the scenes included in the video segment in order to extract the video frame to minimize the load of the harmfulness analysis, and the previously saved comparison target image The harmful images are classified by comparing the similarity between video frames through and learning. Accordingly, the result of comparing the similarity between the video frames shown in the exemplary embodiment of the figure indicates the result of classifying the scenes including each video frame. Length of the scene containing the first extracted movie frame

Is 4,

(The length of the scene containing the second extracted movie frame),

(The length of the scene containing the third extracted video frame), and

The values of (the length of the scene including the fourth extracted video frame) are 3, 6, and 3, respectively.

FIG. 3 is a diagram illustrating an example of performing a similarity comparison by calculating a Structural Similarity (SSIM) value for analyzing a scene length based on a similarity comparison between video frames in FIG. 2.

For the scene length analysis based on the similarity comparison between video frames, a Structural Similarity (SSIM) value is calculated and the similarity of the original image containing the learned harmful scene and the comparison image of the video frame is compared.

SSIM is a representative metric used in the perception quality evaluation of 2D images (movie frames, etc.). Since the human visual system is specialized in deriving structural information from images, the degree of distortion of structural information has the greatest effect on the perceived quality. Note is a calculated value based on assumptions. Accordingly, the SSIM value is calculated using brightness information and distortion information for each pixel of two images to be subjected to similarity comparison.

As can be seen in the example of FIG. 3, the real-time streaming blocking system based on the result of analyzing the harmfulness of the video compares the similarity between the extracted video frames. In the embodiment, the SSIM values for the 10th video frame and the 11th video frame have a small value of 0.4795, which can be considered to mean different scenes, whereas the SSIM values for the 11th and 12th video frames are 0.8258. Because it has a large value, it can be considered that the two frames are included in the same scene. The embodiment of FIG. 3 is merely to show how video frame extraction based on similarity comparison works. In the proposed real-time streaming blocking system based on the result of harmfulness analysis of a video, several scenes included in a video segment are classified and each frame is There is a difference with the mechanism for extracting a video frame representing a scene. In the case of the proposed system, a part in which the SSIM value changes significantly while comparing the similarity between adjacent video frames is determined as a criterion for scene change. For example, in the case of a scene including the first extracted video frame in the example of FIG. 2, it may be viewed as a scene in which video frames similar to the extracted video frame are gathered. On the other hand, in the case of a scene including the second extracted video frame, it may be viewed as another scene starting from a video frame in which the SSIM value changes significantly in the scene including the first extracted video frame.

는 원본 이미지 x의 평균 밝기,

는 비교 이미지

의 평균 밝기, 그리고

은 이미지 밝기 상수를 의미한다. 해당 수식에 의해 계산되는 값은 원본 이미지와 비교 이미지의 평균 밝기를 종합하여 두 이미지의 평균 밝기를 나타낼 수 있도록 정규화된 값을 의미한다.When the original image is x and the comparison image is y, to calculate the SSIM value, first, the average brightness I(x,y) of the pixels of the two images is calculated by the following equation.

Is the average brightness of the original image x,

The comparison image

The average brightness of, and

Means the image brightness constant. The value calculated by the corresponding formula refers to a value normalized to represent the average brightness of the two images by synthesizing the average brightness of the original image and the comparison image.

After calculating the average brightness, the standard deviation value of each image is defined as the contrast value c(x,y) of the image, and the standard deviation value of the two images is calculated as follows.

는 원본 이미지 x의 밝기 표준편차,

는 비교 이미지 y의 밝기 표준 편차, 그리고

는 이미지 콘트라스트 상수를 의미한다. 두 이미지의 평균 밝기를 계산하는 수식과 마찬가지로, 해당 수식에 의해 계산되는 값은 원본 이미지와 비교 이미지의 밝기 표준편차를 종합하여 두 이미지의 콘트라스트를 나타낼 수 있도록 정규화된 값을 의미한다.

Is the standard deviation of the brightness of the original image x,

Is the standard deviation of the brightness of the comparison image y, and

Denotes the image contrast constant. Like the equation for calculating the average brightness of two images, the value calculated by the equation means a value normalized to represent the contrast of the two images by synthesizing the standard deviation of brightness of the original image and the comparison image.

는 각 이미지의 밝기 정보가 서로 얼마나 연관성이 있는지를 나타내기 위한 값이며,

는 이미지 구조 상수를 의미한다. 해당 수식에 의해 계산되는 값은 두 이미지 각각의 밝기 표준편차 대비 두 이미지 사이의 연관된 밝기 표준 편차의 값의 상대적 비율을 의미한다.After calculating the image contrast, the value of the image structure index s(x,y) is calculated using the brightness covariance and brightness standard deviation between the two images as shown in the following equation. Brightness covariance

Is a value to indicate how much the brightness information of each image is related to each other,

Means the image structure constant. The value calculated by the corresponding equation refers to a relative ratio of the standard deviation of the brightness of each of the two images to the standard deviation of the brightness of the two images.

The image structure constant is calculated using the image contrast constant as follows. Since the process of calculating the image contrast includes the standard deviation of the brightness of the image, it is related to the process of calculating the image structure index. However, since brightness covariance is used to calculate the image structure index, Likewise, the image structure constant is calculated.

Finally, the SSIM value is calculated by the following equation by synthesizing the average brightness of the two images, the contrast value of the image, and the image structure index. At this time, the values of the weights a, b, and c used are all set to 1. The meaning of the formula is to compare the similarity between the original image and the comparison image by reflecting all factors that affect the perceived quality of a person. Since the weights are all set to 1, in the SSIM calculation, each factor is It will have the same influence.

In the present invention, SSIM values for two adjacent video frames are calculated, and similarity is determined by comparing the calculated SSIM values with a specific threshold. In the example of FIG. 2, a threshold value of 0.5 for classifying each scene based on the similarity comparison between the original image and the comparison image was used. In addition, using the similarity comparison result between the video frames, as in the embodiment of FIG. 2, video frames included in the video segment are classified into scene units, and the length of each scene is analyzed and used to calculate the final harmful rating.

4 shows an example of a process of calculating a final hazard level by using the hazard analysis result and scene length information.

은 0.76 (76%), 두 번째로 추출된 동영상 프레임에 대한 유해 확률

는 0.32 (32%)이다. 추출된

번째 동영상 프레임에 대한 유해 확률로부터 해당 동영상 프레임의 유해 등급

는 다음 수식과 같이 계산된다.It shows the process of calculating the final hazard rating for determining the hazard rating for each scene and blocking streaming by using the hazard analysis result of the video frame extracted from the video segment and the scene length information. The result of the hazard analysis of the video frame used as an input of the hazard analysis engine is expressed as a hazard probability for each frame. Harmful probability for the first extracted video frame in the example in the figure

Is 0.76 (76%), the probability of harm for the second extracted video frame

Is 0.32 (32%). Extracted

The harmfulness rating of the video frame from the harmful probability for the second video frame

Is calculated as follows.

번째 동영상 프레임이 포함된 장면의 길이 정보

를 이용하여 최종 유해 등급

을 다음 수식과 같이 계산한다.

는 최대 유해 등급,

는 유해 등급

에 해당하는 동영상 프레임이 포함된 장면의 길이, 그리고

는 최종 유해 등급을 계산할 때 유해 등급

에 반영되는 가중치를 의미한다. 유해 등급 가중치는 스트리밍 차단을 실제로 결정하게 되는 사업자 등에 따라 다른 값을 가질 수 있으며, 어떻게 설정하느냐에 따라 최종 유해 등급에 반영되는 각 장면이 가지는 유해 등급의 중요도가 변한다. 도 4의 예시에서 사용되는 유해 등급 가중치는 모두 1로 설정되었다. 예를 들어 높은 유해 등급에 대한 가중치가 큰 값으로 설정되어 있을 경우, 최종 유해 등급을 계산할 때 장면별 유해 등급의 변화가 빠르게 반영된다. 반대로 낮은 유해 등급에 대한 가중치가 큰 값으로 설정되어 있을 경우, 최종 유해 등급을 계산할 때 장면별 유해 등급의 변화는 상대적으로 천천히 반영된다. 해당 수식에서 계산되는 값은 추출된 각 동영상 프레임이 포함된 장면 길이와 유해 등급을 이용하여 가중 이동 평균을 계산함으로써 현재 스트리밍 중인 동영상의 유해성을 나타내기 위한 누적 유해 등급을 의미하게 된다.Through the above equation, the hazard probability can be converted into a hazard class having discrete values ranging from 1 to 10. After the hazard class has been calculated, the extracted

Length information of the scene containing the first video frame

Final hazard rating using

Is calculated as the following formula.

Is the maximum hazard class,

Is a hazardous rating

The length of the scene containing the video frame corresponding to, and

Is the hazard class when calculating the final hazard class.

It means the weight reflected in. The hazard rating weight may have a different value depending on the operator who actually decides to block streaming, and the importance of the hazard rating of each scene reflected in the final hazard rating changes depending on how it is set. All harmful grade weights used in the example of FIG. 4 are set to 1. For example, if the weight for a high hazard level is set to a large value, the change in the hazard grade for each scene is quickly reflected when calculating the final hazard grade. Conversely, when the weight for a low hazard level is set to a large value, the change in the hazard grade for each scene is relatively slowly reflected when calculating the final hazard grade. The value calculated in the equation refers to a cumulative hazard rating to indicate the harmfulness of the currently streaming video by calculating a weighted moving average using the scene length including each extracted video frame and the hazard rating.

As can be seen in the example of FIG. 4, the final hazard level is calculated using the length information of the scenes including the frames extracted so far and the result of the hazard analysis of each frame, and the hazard grades 8 and 4, and the scene lengths 4 and 3, respectively. The final hazard class calculated by reflecting is 7. Subsequently, when a scene having a different hazard level appears, the calculated value of the final hazard level varies, and the final hazard level may be larger or smaller than the previous value according to the change of the hazard level.

If the result of the hazard analysis and the length of each scene for the video frames extracted so far are the same as the table in the figure, the system first calculates the hazard level using the hazard probability for each frame derived from the hazard analysis result. If the hazard rating is high, it means that the frame's hazard is high. The system calculates the final harmful rating by reflecting the harmful rating for each frame, the number of repetitions of the scenes corresponding to each frame, and the scene length information in order to determine the blocking time of the currently streaming video. For each frame, as the scene length of the frame increases, the weight reflected when calculating the final harmful level increases. This system does not determine the streaming blocking time using only the harmfulness analysis result at the current time, but considers the harmfulness analysis result of previously extracted frames and information on how long each frame has been repeated. Can perform streaming blocking control.

은

번째로 추출된 동영상 프레임이 반복된 개수

를 고려하여 최종 유해 등급을 계산하는 수식으로, 도 4의 예시를 통해 설명한 수식과 동일하다. 도 5의 예시에서는 유해 등급이 1부터 5까지 총 5개 존재하는 상황을 가정하였으며, 따라서 최종 유해 등급을 계산하기 위한 수식을 각 유해 등급에 대한 함수

로 표현하는 것이 가능하다. 먼저 도 5의 ①에서 ②에 해당하는 타임라인에서 본 발명에서 제시하는 동영상 유해성 분석 결과 기반의 실시간 스트리밍 차단 시스템은 해당 타임라인에서 유해성 분석 엔진 모듈의 입력으로 제공된 동영상 프레임에 대한 유해 등급을 결정한다. ②에서 ③에 해당하는 타임라인에서는 이전 타임라인에서 추출된 동영상 프레임의 유해 등급과 다른 유해 등급을 가지는 동영상 프레임들이 반복되는 장면이기 때문에 해당 동영상 프레임들에 대해 유해 등급을 새로 계산한다. 최종적으로 ③에서 ④에 해당하는 타임라인에서는 첫 번째 타임라인에서 추출된 동영상 프레임의 유해 등급과 같은 유해 등급을 가지는 동영상 프레임들이 반복되기 때문에 최종 유해 등급을 산출할 때 이를 반영한다. 최종 유해 등급은 타임라인에 따라 반복된 장면, 혹은 같은 유해 등급을 가지는 장면의 길이 정보를 이용하여 계산한다. 그림의 예시에서도 볼 수 있듯이

의 유해 등급을 가지는 장면은 타임라인 ①에서 ②와 타임라인 ③에서 ④, 두 경우가 존재한다. 처음에는 총 7개의 동영상 프레임들이 타임라인 ①에서 ②에 해당하는 장면에 포함되었으므로 최종 유해 등급은 장면 길이 7과

유해 등급을 이용하여 계산되며, 이후에 타임라인 ③에서 ④를 거친 후 유해 등급

에 해당하는 동영상 프레임이 포함된 장면의 길이가 30이 증가하였으므로 최종 유해 등급을 계산할 때 이를 반영하게 된다. 타임라인 ②에서 ③의 경우 유해 등급

를 가지며 최종 유해 등급을 계산할 때 높은 유해 등급을 가지지만 유해 등급

에 비해 장면 길이가 짧기 때문에 최종 유해 등급에 반영되는 비율이 줄어들게 된다. 최종 유해 등급 산출에 사용되는 유해 등급별 가중치

는 장면 별로 누적된 동영상 프레임이 30개 이상일 경우 업데이트하게 되며 새로운 장면이 등장할 때마다 그림의 예시에서와 같이 모든 유해 등급에 대한 가중치 값이 1로 초기화된다. 최종 유해 등급은 어느 한 시점에서의 유해 등급만을 고려하는 것이 아니라 시간에 따라, 그리고 반복되는 장면의 길이에 따라 누적된 유해 등급을 고려하여 계산되기 때문에 실시간으로 유해한 동영상을 차단하는 데 있어서 적시성을 보장할 수 있다.FIG. 5 shows a process of reflecting the harmful rating for each video frame calculated by the harmfulness analysis and the repeated length of the scene corresponding to each video frame when calculating the final harmfulness rating. G refers to the maximum harm rating set by a provider that performs streaming blocking,

silver

The number of times the extracted video frame is repeated

This is an equation for calculating the final hazard level in consideration of and is the same as the equation described through the example of FIG. 4. In the example of FIG. 5, it is assumed that there are a total of five hazard grades from 1 to 5, and therefore, the formula for calculating the final hazard grade is a function for each hazard grade.

It is possible to express it as First, in the timeline corresponding to ① to ② of FIG. 5, the real-time streaming blocking system based on the result of the harmfulness analysis of the video presented by the present invention determines the harmfulness level of the video frame provided as an input of the harmfulness analysis engine module in the timeline. . In the timeline corresponding to ② to ③, since video frames having a harmful rating different from that of the video frame extracted from the previous timeline are repeated, the harmful rating is newly calculated for the corresponding video frames. Finally, in the timeline corresponding to ③ to ④, the video frames having the same harmfulness rating as that of the video frame extracted from the first timeline are repeated, so this is reflected when calculating the final harmfulness rating. The final hazard level is calculated using length information of a scene repeated according to a timeline or a scene having the same hazard level. As you can see in the example in the picture

There are two cases of scenes with a hazard rating of: ② in the timeline ① and ④ in the timeline ③. Initially, a total of 7 video frames were included in the scene corresponding to the timeline ① to ②, so the final hazard rating was 7 and

It is calculated using the hazard class, and after going through ④ in the timeline ③, the hazard class

Since the length of the scene including the video frame corresponding to is increased by 30, this is reflected when calculating the final harm rating. Hazard grade in case of ③ in timeline ②

And has a high hazard class when calculating the final hazard class, but

Since the scene length is shorter than that, the ratio reflected in the final hazard rating decreases. Weight for each hazard class used to calculate the final hazard class

Is updated when there are more than 30 video frames accumulated for each scene, and each time a new scene appears, the weight values for all harmful ratings are initialized to 1 as in the example of the figure. As the final hazard rating is calculated not only by considering the hazard rating at any one point in time, but by taking into account the hazard rating accumulated over time and according to the length of the repeated scene, timeliness is guaranteed in blocking harmful videos in real time. can do.

6 shows the definition of each variable used to calculate the final hazard level of FIG. 4.

는

번째로 추출된 동영상 프레임에 대한 유해성 분석 결과로 해당 프레임의 장면이 유해 장면일 확률을 나타낸다.

는 i 번째로 추출된 동영상 프레임이 반복된 개수로 장면 길이 정보를 나타낸다. g는 추출된 동영상 프레임의 유해 등급으로 프레임 별 유해 확률을 이용하여 결정된다. G는 서비스 제공자가 설정 가능한 최대 유해 등급으로 도 3의 예에서는 최소 1등급부터 최대 10등급까지 유해 등급을 가지도록 설정하였다.

는

와 동일한 값으로 장면 길이를 나타내며,

는 최종 유해 등급을 산출할 때 사용되는 가중치로

와 마찬가지로 추출된 동영상 프레임의 유해 등급이 높을수록 최종 유해 등급 산출에 더 많은 가중치를 주기 위해 설정된 변수이다. 본 시스템에서는 최종 유해 등급이 각 프레임의 장면 길이에 너무 민감하게 반응하지 않도록 모든 유해 등급에 대한 가중치

의 값을 1로 설정하였다.

의 값은 사용자에게 제공되는 서비스 또는 동영상의 특성에 따라 다르게 설정될 수 있는 값이다. g'은 최종 유해 등급이며, 동영상 프레임의 유해 등급이 가질 수 있는 값의 범위 내에서 결정된다.

Is

As a result of analyzing the harmfulness of the second extracted video frame, it indicates the probability that the scene of the corresponding frame is a harmful scene.

Denotes scene length information as the number of repetitions of the i- th extracted video frame. g is the harmfulness level of the extracted video frame, and is determined using a harmful probability for each frame. G is the maximum harmful rating that can be set by the service provider, and in the example of FIG. 3, it is set to have a harmful rating ranging from a minimum of 1 to a maximum of 10.

Is

The scene length is represented by the same value as

Is the weight used when calculating the final hazard rating.

Likewise, the higher the harmfulness level of the extracted video frame, the more weight is set to calculate the final harmfulness level. In this system, all hazard ratings are weighted so that the final hazard rating is not too sensitive to the scene length of each frame.

The value of was set to 1.

The value of is a value that can be set differently according to the characteristics of a service or video provided to a user. g'is the final harmful rating, and is determined within the range of values that can have the harmful rating of the video frame.

The method for blocking real-time streaming based on the result of the video harmfulness analysis of the present invention

(a) For HTTP adaptive streaming, video data is encoded in the form of segments having a fixed playback length and different bitrates and stored in a video server, and real-time streaming video data is provided to the user terminal from the video server. Step to do; And

(b) An HTTP adaptive streaming client is installed in the user terminal, and the similarity between video frames is compared with the original image using a deep learning-based harmfulness analysis engine to calculate the length information of the scene of each frame and the harmful probability for each frame. And determining to block streaming according to the final harmful rating to block harmful videos,

The user terminal is installed with the HTTP adaptive streaming client, downloads a Media Presentation Description (MPD) file describing information on a video to be requested from the video server, and a specific quality according to the measured network bandwidth and the state of the terminal ( bitrate) of the video segment,

The deep learning-based harmfulness analysis engine uses a CNN (Convolutional Neural Network) model, and compares the harmful original image with the comparison image by learning the characteristics of the input images for a given data set of the video segment, that is, the similarity between video frames. By comparing the harmful original image, the length information of the scene of each frame and the harmful probability for each frame are calculated, and the harmfulness of the video is analyzed and the result of the video harmfulness analysis in the form of a harmful probability for each video frame is blocked from video streaming. Provided as a module.

When the video segments are transmitted from the video server to the user terminal in step (b), the user terminal adaptively extracts the video frame to minimize the load of harmfulness analysis through similarity comparison between video frames, and the extracted video frame Is delivered to a deep learning-based hazard analysis engine,

The deep learning-based harmfulness analysis engine of the user terminal uses a Convolutional Neural Network (CNN) model, and compares the harmful original image and the comparison image by learning the characteristics of input images for a given data set of a video segment, that is, The similarity between video frames is compared with the harmful original image, the length information of the scene of each frame and the harmful probability of each frame are calculated, and the harmfulness of the video is analyzed and the result of the video harmfulness analysis in the form of a harmful probability for each video frame Providing to a video streaming blocking control module;

The video streaming blocking control module comparing the similarity between video frames, calculating length information of each frame scene and a harmful probability for each frame, determining streaming blocking according to the calculated final harmful rating, and performing blocking control of the harmful video; And

The deep learning-based hazard analysis engine includes the step of calculating a final hazard rating for determining a hazard rating for each scene and blocking streaming using the hazard analysis result of the video frame extracted from the video segment and the scene length information. do.

The deep learning-based harmfulness analysis engine calculates a SSIM (Structural Similarity; Structural Similarity Index) value for the scene length analysis based on the similarity comparison between the video frames, and compares the original image containing the learned harmful scene with the comparison image of the frame. Compare similarities.

The SSIM value is calculated as follows by synthesizing the average brightness I(x,y) of the pixels of the two images, the contrast value c(x,y) of the image, and the image structure index s(x,y),

At this time, the values of the weights a, b, and c are all set to 1, comparing the similarity between the original image and the comparison image.Since the weights are all set to 1, the calculated SSIM value is a specific threshold. It compares with the value to determine whether it is similar.

The deep learning-based hazard analysis engine classifies each video frame extracted from the video segment into a scene unit, and uses the hazard analysis result and scene length information for each video frame to classify and stream the hazard rating for each scene. Calculate the final hazard class to determine blocking,

The result of the hazard analysis of the video frame used as an input of the hazard analysis engine is expressed as a hazard probability for each frame.

The real-time streaming blocking system based on the result of the harmfulness analysis of a video proposed by the present invention extracts a video frame through a similarity comparison between video frames in order to minimize the load that may occur in the harmfulness analysis, and the harmfulness analysis result of the extracted video frame and each Significant blocking control for a real-time streaming service is performed using information on the length of a scene including a video frame. If the method of calculating the accumulated hazard ratings so far using the hazard analysis result of the extracted video frame and the scene length information, a method different from the formula for calculating the final hazard grade used in the present invention (accumulated value other than a weighted moving average) It is also possible to apply, for example, using other functions to calculate. In addition, from the viewpoint of a provider who decides to block streaming, a weight for each harmful rating and a maximum harmful rating used for calculating the final harmful rating may be differently set according to the characteristics of the service and video provided. A method of blocking streaming in real time may apply various methods, such as disconnecting the connection itself or performing image processing on the screen of a video being played, depending on the service provider.

In a service platform that provides VoD and real-time streaming services applied with adaptive streaming technology, personal broadcasting and real-time streaming, the harmfulness of the video is analyzed by analyzing the harmfulness of the video using the video frame extracted based on the similarity comparison from the video being streamed. It is possible to perform meaningful blocking control in a real-time streaming service by minimizing the load that may occur and controlling the blocking of streaming by reflecting the result of the harmfulness analysis and the length of the extracted video frame.

The embodiments according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include storage, magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Magneto-optical media, and a hardware device specially configured to store and execute program instructions in ROM, RAM, and flash memory may be included. Examples of program instructions may include those produced by a compiler, and high-level language codes that can be executed by a computer using an interpreter as well as machine language codes. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention.

As described above, the method of the present invention is implemented as a program and can be read using software of a computer, and a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) ) Can be stored.

Although it has been described with reference to a preferred embodiment of the present invention, various modifications or variations of the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims by those of ordinary skill in the relevant technical field You will understand that it can be done.

Claims (12)

A video server that encodes video data into segments having a fixed playback length and different bitrates for HTTP adaptive streaming and stores them in the server, and provides real-time streaming video data; And
An HTTP adaptive streaming client is installed, and the similarity between video frames is compared with the original harmful image using a deep learning-based harmfulness analysis engine, and the length information of each frame and the harmful probability for each frame are calculated according to the final harm rating. Includes a user terminal that blocks harmful videos by deciding to block streaming,
The user terminal is installed with the HTTP adaptive streaming client, downloads a Media Presentation Description (MPD) file describing information on a video to be requested from the video server, and a specific quality according to the measured network bandwidth and the state of the terminal ( bitrate) of the video segment,
The deep learning-based harmfulness analysis engine uses a CNN (Convolutional Neural Network) model, and compares the similarity between video frames with the original harmful image by learning the characteristics of input images for a given data set of a video segment. Real-time streaming based on video harmfulness analysis results, which calculates the length information and harmful probability of each frame, analyzes the harmfulness of the video, and provides the video harmfulness analysis result in the form of harmfulness probability for each video frame to the video streaming blocking control module. Blocking system. The method of claim 1,
The user terminal
The video frame is extracted from the video segment being streamed, and the characteristics of the input images of each video frame are learned to compare the harmful original image and the comparison image. That is, the similarity between the video frames is compared with the harmful original image, A deep learning-based harmfulness analysis engine that calculates length information and a harmful probability for each frame, analyzes the harmfulness of the video, and provides a result of the harmfulness analysis for each video frame;
A video frame extraction module for extracting video frames excluding duplicated video frames to increase the efficiency of video harmfulness analysis; And
A video streaming blocking control module for calculating a final harmful rating using the harmfulness analysis result and scene length information, and blocking streaming of a harmful video determined as the final harmful rating;
Real-time streaming blocking system based on video harmfulness analysis results comprising a. delete The method of claim 1,
The deep learning-based hazard analysis engine
A video hazard analysis result that compares the similarity between the original image containing the learned harmful scene and the comparative image of the frame by calculating the SSIM (Structural Similarity; Structural Similarity Index) value for the scene length analysis based on the similarity comparison between the video frames. Based real-time streaming blocking system. The method of claim 4,
The SSIM value is calculated as follows by synthesizing the average brightness I(x,y) of the pixels of the two images, the contrast value c(x,y) of the image, and the image structure index s(x,y),

At this time, the values of the weights a, b, and c are all set to 1, comparing the similarity between the original image and the comparison image.Since the weights are all set to 1, the calculated SSIM value is a specific threshold. Real-time streaming blocking system based on video harmfulness analysis results that compares with values to determine whether they are similar. The method of claim 1,
The deep learning-based hazard analysis engine
Each video frame extracted from the video segment is classified into a scene unit, and a final harmful rating for determining a harmful rating for each scene and blocking streaming using the harmfulness analysis result and the scene length information for each video frame Is calculated,
A real-time streaming blocking system based on a video hazard analysis result, wherein the hazard analysis result of the video frame used as an input of the hazard analysis engine is expressed as a hazard probability for each frame. (a) For HTTP adaptive streaming, video data is encoded in the form of segments having a fixed playback length and different bitrates and stored in a video server, and real-time streaming video data is provided to the user terminal from the video server. Step to do; And
(b) An HTTP adaptive streaming client is installed in the user terminal, and the similarity between video frames is compared with the original image using a deep learning-based harmfulness analysis engine to calculate the length information of the scene of each frame and the harmful probability for each frame. And determining to block streaming according to the final harmful rating to block harmful videos,
The user terminal is installed with the HTTP adaptive streaming client, downloads a Media Presentation Description (MPD) file describing information on a video to be requested from the video server, and a specific quality according to the measured network bandwidth and the state of the terminal ( bitrate) of the video segment,
The deep learning-based harmfulness analysis engine uses a CNN (Convolutional Neural Network) model, and compares the harmful original image with the comparison image by learning the characteristics of the input images for a given data set of the video segment, that is, the similarity between video frames. By comparing the harmful original image, the length information of the scene of each frame and the harmful probability for each frame are calculated, and the harmfulness of the video is analyzed and the result of the video harmfulness analysis in the form of a harmful probability for each video frame is blocked from video streaming. Real-time streaming blocking method based on video harmfulness analysis results provided as a module. delete The method of claim 7,
When the video segments are transmitted from the video server to the user terminal in step (b), the user terminal adaptively extracts the video frame to minimize the load of harmfulness analysis through similarity comparison between video frames, and the extracted video frame Is delivered to a deep learning-based hazard analysis engine,
The deep learning-based harmfulness analysis engine of the user terminal uses a CNN (Convolutional Neural Network) model, and compares the similarity between video frames with the harmful original image by learning the characteristics of input images for a given data set of a video segment. Calculating the length information of the scene of each frame and the harmful probability of each frame, analyzing the harmfulness of the video, and providing a video harmfulness analysis result to a video streaming blocking control module in the form of a harmful probability for each video frame; And
The video streaming blocking control module comparing the similarity between video frames, calculating length information of each frame scene and a harmful probability for each frame, determining streaming blocking according to the calculated final harmful rating, and performing blocking control of the harmful video;
The deep learning-based harmfulness analysis engine calculating a harmfulness level for each scene and a final harmfulness level for determining the streaming blocking by using the harmfulness analysis result of the video frame extracted from the video segment and the scene length information;
Real-time streaming blocking method based on video harmfulness analysis results comprising a. The method of claim 7,
The deep learning-based hazard analysis engine
A video hazard analysis result that compares the similarity between the original image containing the learned harmful scene and the comparison image of the frame by calculating the SSIM (Structural Similarity; Structural Similarity Index) value for the scene length analysis based on the similarity comparison between the video frames. Based real-time streaming blocking method. The method of claim 10,
The SSIM value is calculated as follows by synthesizing the average brightness I(x,y) of the pixels of the two images, the contrast value c(x,y) of the image, and the image structure index s(x,y),

At this time, the values of the weights a, b, and c are all set to 1, comparing the similarity between the original image and the comparison image.Since the weights are all set to 1, the calculated SSIM value is a specific threshold. A real-time streaming blocking method based on the result of video harmfulness analysis that compares with the value to determine whether it is similar. The method of claim 7,
The deep learning-based hazard analysis engine
Each video frame extracted from the video segment is classified into a scene unit, and a final harmful rating for determining a harmful rating for each scene and blocking streaming using the harmfulness analysis result and the scene length information for each video frame Is calculated,
A method for blocking real-time streaming based on a result of a harmfulness analysis of a video, wherein the result of the harmfulness analysis of the video frame used as an input of the harmfulness analysis engine is expressed as a harmful probability for each frame.

Images