KR102574353B1 - Device Resource-based Adaptive Frame Extraction and Streaming Control System and method for Blocking Obscene Videos in Mobile devices - Google Patents

Abstract

A terminal resource-based adaptive frame extraction and streaming control system and method for blocking harmful video in a mobile terminal are disclosed. The system includes a server; An adaptive frame extraction module that receives video frames and extracts adaptive frames, and terminal resources required to block harmful videos are factors related to the processing capacity of the terminal and the number of extracted frames and harmfulness analysis in the case of response results to harmful videos includes a user terminal having a deep learning-based harmfulness analysis engine and a streaming control module for mosaic processing in the case of harmful videos, and frames of video segments received from the server are extracted in real time Then, the harmfulness analysis engine transmits the harmfulness analysis result for each extracted frame to the harmfulness change determination module to determine the harmfulness change of the video being played, and the harmfulness determination result is the change in terminal resources required for harmfulness analysis It is used to adjust the frame extraction method, and the terminal resource characteristic that is the criterion for adjusting the frame extraction method is the state of the remaining battery of the user terminal, and the state of the remaining battery of the user terminal is determined to be sufficient according to the specifications of the terminal. state (40%), a state that is the standard for power saving mode (20%), and a state that is the standard for ultra power saving mode (10%). Adaptive frame extraction and It operates based on HTTP adaptive streaming that adaptively determines the bit rate of video segments.

Description

Device Resource-based Adaptive Frame Extraction and Streaming Control System and method for Blocking Obscene Videos in Mobile devices}

The present invention is a user terminal resource-based adaptive frame that extracts frames adaptively in consideration of the terminal resource (battery state of the user terminal) in use to block harmful videos in a mobile terminal and mosaics harmful videos. It relates to extraction and streaming control systems and methods.

Recently, as the spread of computers and smart devices increases and decreases, the wired and wireless Internet is overflowing with harmful websites and videos that can be accessed with PCs and smart devices. or download illegal videos or are exposed to harmful videos.

To solve this problem, technology to block harmful websites and harmful videos is needed for young children and teenagers.

Terminal resources required to block harmful videos are divided into factors related to processing capability and response results to harmful videos. Factors related to processing power include CPU occupancy and battery consumption, and response results for harmful videos include the number of frames extracted and the time required for harmfulness analysis. The user terminal has a deep learning-based hazard analysis engine. The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine. The harmfulness analysis engine transfers the harmfulness analysis result for each extracted frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with changes in terminal resources required for harmfulness analysis. A terminal resource characteristic that is a criterion for adjusting the frame extraction method is the remaining battery of the terminal. The remaining battery state is divided into a state that is determined to be sufficient (40%), a state that is the standard for power saving mode (20%), and a state that is the standard for ultra power saving mode (10%) according to the specifications of the terminal. If the remaining battery of the terminal is 40% or more, an adaptive frame extraction algorithm based on the similarity comparison between harmfulness changes and frames is applied. When the remaining battery state of the terminal is necessary to prepare for entering the power saving mode, it prepares for rapid battery consumption by controlling the image resizing size and SSIM threshold, which have little effect on the CPU load during the harmful video blocking process. When the remaining battery of the terminal is in the power saving mode or ultra power saving mode, the load of the terminal is reduced by additionally applying frame extraction interval control and similarity comparison.

As a prior art 1 related to this, Patent Registration No. 10-06877320000 has registered “a method and apparatus for blocking harmful videos using content-based multi-modal feature values”.

A method and apparatus for blocking harmful video using content-based multi-modal feature values include: constructing a discrimination model composed of a harmful video classification model and a non-harmful video classification model; Separating video content into a video stream and an audio stream; extracting predetermined visual feature values from the separated video stream to detect shot boundaries, extracting at least one key frame from each shot, and generating shot and key frame information; extracting multi-modal feature values of the key frame based on the shot and key frame information after performing predetermined pre-processing on the video stream; determining the harmfulness of a shot by inputting the multi-modal feature values into the discrimination model; and determining the harmfulness of the video content by integrating the determination results based on the shot criteria.

Blocking harmful videos through the Internet, controlling distribution of harmful videos through P2P, and digital broadcasting streams by using a gradual blocking method using a keyframe-based judgment engine and a frame-based judgment engine, and using multi-modal feature values. It can be applied to real-time harmful information monitoring system of

As prior art 2 related to this, Patent Registration No. 10-09071720000 has registered "a system and method for blocking harmful videos in multiple stages in a video distribution environment".

The multi-level blocking system for harmful videos

a filtering manager for generating a summary image by merging images extracted from video contents and detecting harmful videos by analyzing the summary image; and

and an evaluation analysis server for detecting harmful videos by analyzing reaction information of consumers watching the video contents, and transmitting information on the video contents corresponding to the harmful videos to the filtering manager.

Using the multi-step blocking system and method for harmful videos, it is possible to efficiently block harmful videos throughout the process of production, distribution and consumption of videos, and it is possible to detect harmful videos that were not detected in the previous step through feedback. In addition, since a filtering method based on the content of the video is used, it is possible to efficiently block harmful videos even in videos such as UCC.

As a prior art 3 related to this, Patent Registration No. 10-21690730000 "system and method for blocking real-time streaming based on video harmfulness analysis results" is registered.

1 is a diagram showing the structure of a real-time streaming blocking system based on a conventional video harmfulness analysis result.

Real-time streaming blocking system based on video harmfulness analysis results

A video server that encodes video data in the form of segments having a fixed playback length and different bitrates for HTTP adaptive streaming, stores them in the server, and provides real-time streaming video data; and

An HTTP adaptive streaming client is installed, and a deep learning-based harmfulness analysis engine is used to compare the similarity between video frames with the original harmful image, calculate the length information of each frame and the probability of harmfulness per frame, and determine the final harmfulness level. Including a user terminal that blocks harmful videos by determining streaming blocking;

The user terminal has the HTTP adaptive streaming client installed, downloads a Media Presentation Description (MPD) file describing information about a video to be requested from the video server, and has a specific quality ( bitrate) request a video segment,

The deep learning-based harmfulness analysis engine uses a convolutional neural network (CNN) model, learns the characteristics of input images for a given data set of video segments, compares the similarity between video frames with the original harmful image, and compares the scene of each frame. It calculates length information and harmfulness probability for each frame, analyzes the harmfulness of the video, and provides the video harmfulness analysis result in the form of harmfulness probability for each video frame to the video streaming blocking control module.

The user terminal

A video frame is extracted from a video segment being streamed, the characteristics of the input images of each video frame are learned, and the harmful original image and the comparison image are compared, that is, the similarity between the video frames is compared with the harmful original image to determine the scene of each frame. a deep learning-based harmfulness analysis engine that calculates length information and harmfulness probabilities for each frame, analyzes harmfulness of the video, and provides harmfulness analysis results for each video frame;

A video frame extraction module that extracts video frames except for duplicate video frames to increase the efficiency of video harmfulness analysis; and

and a video streaming blocking control module that calculates a final harmfulness level using the harmfulness analysis result and scene length information, and blocks the streaming of the harmfulness video determined to be the final harmfulness level.

The deep learning-based harmfulness analysis engine calculates SSIM (Structural Similarity; Structural Similarity Index) values for scene length analysis based on similarity comparison between video frames, and compares the original image including the learned harmful scene with the frame comparison image. Compare similarities.

The SSIM value is calculated as follows by combining the average brightness I(x,y) of the pixels of the two images, the contrast value c(x,y) of the image, and the image structure index s(x,y), The values of the weights a, b, and c used at this time are all set to 1, and the similarity between the original image and the comparison image is compared. Since the weights are all set to 1, the calculated SSIM value is set to a specific threshold Compare the values to determine whether they are similar.

The deep-learning-based harmfulness analysis engine classifies each video frame extracted from the video segment into a scene unit, and uses the harmfulness analysis result and scene length information for each video frame to classify and stream harmfulness for each scene. Calculate a final hazard class for determining interception;

A harmfulness analysis result for a video frame used as an input of the harmfulness analysis engine is expressed as a harmfulness probability for each frame.

however,

Patent Registration No. 10-06877320000 (Registration Date: February 21, 2007), "Method and Apparatus for Blocking Harmful Videos Using Content-Based Multi-Modal Characteristics", Korea Electronics and Telecommunications Research Institute, Korea Information and Communications University Industry-University Cooperation Foundation Patent Registration No. 10-09071720000 (registration date: July 02, 2009), "Multi-level blocking system and method for harmful videos in video distribution environment", SK Telecom Co., Ltd. Patent Registration No. 10-21690730000 (registration date: October 16, 2020), "System and method for blocking real-time streaming based on video harmfulness analysis results", Kwangwoon University Industry-University Cooperation Foundation

An object of the present invention to solve the above problem is to adaptively extract frames in consideration of the resources (battery state) of the terminal in use, mosaic-process harmful videos, and specifically, determine the remaining battery state of the user terminal. The proposed system minimizes the battery consumption of the user terminal and blocks harmful videos by classifying and adaptively determining the frame extraction algorithm according to the state, and extracting frames suitable for the state of the terminal playing the video and determining harmfulness change. A terminal resource-based adaptive frame extraction and streaming control system and method for blocking harmful videos in a mobile terminal, which performs frame extraction adaptively in consideration of the terminal resources in use to block harmful videos in the mobile terminal. to provide.

In order to achieve the object of the present invention, a terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal includes a server; A user terminal having an adaptive frame extraction module, a deep learning-based harmfulness analysis engine, and a streaming control module that performs mosaic processing in the case of harmful videos,
The terminal resources required to block harmful videos of the user terminal include factors related to the processing capacity of the terminal and the number of frames extracted and the time required for harmfulness analysis in the case of response results to harmful videos.
In the terminal resources required to block harmful videos of the user terminal, factors related to the processing capacity of the terminal include CPU occupancy and battery consumption,
The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine, and the harmfulness analysis engine transmits the harmfulness analysis result for each frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with the change in terminal resources required for hazard analysis, and the terminal resource characteristic that is the standard for the frame extraction method is the state of the remaining battery of the user terminal.
The state of the remaining battery of the user terminal is divided into a state that is determined to be sufficient according to the specifications of the terminal (40%), a state that is a standard for power saving mode (20%), and a state that is a standard for ultra power saving mode (10%), It operates based on HTTP adaptive streaming that adaptively determines the bit rate of video segments and adaptive frame extraction based on terminal resources for blocking harmful videos in mobile terminals,
The adaptive frame extraction module extracts video frames excluding duplicate video frames to increase the efficiency of video harmfulness analysis, calculates SSIM values for two adjacent video frames, and compares the calculated SSIM values with a specific threshold. to determine the degree of similarity,
When the battery state of the user terminal enters the power saving mode (20%), the harmfulness analysis engine maintains the image resizing size and SSIM threshold value adjusted in the previous step and increases the frame extraction interval to twice the standard frame extraction interval. By reducing the number of operations, the load on terminal resources is reduced.

In addition, in order to achieve another object of the present invention, a terminal resource-based adaptive frame extraction and streaming control method for blocking harmful videos in a mobile terminal is

server; The adaptive frame extraction module and the terminal resources required to block harmful videos include factors related to the processing capacity of the terminal and the number of extracted frames and the time required for harmfulness analysis in the case of response results to harmful videos, based on deep learning. In a terminal resource (battery state)-based adaptive frame extraction and streaming control system including a user terminal having a harmfulness analysis engine and a streaming control module for mosaic processing in the case of harmful videos,
The terminal resources required to block harmful videos of the user terminal include factors related to the processing capacity of the terminal and the number of frames extracted and the time required for harmfulness analysis in the case of response results to harmful videos.
In the terminal resources required to block harmful videos of the user terminal, factors related to the processing capacity of the terminal include CPU occupancy and battery consumption,

The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine, and the harmfulness analysis engine transmits the harmfulness analysis result for each frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with the change in terminal resources required for hazard analysis, and the terminal resource characteristic that is the standard for the frame extraction method is the state of the remaining battery of the user terminal.

The state of the remaining battery of the user terminal is divided into a state that is determined to be sufficient according to the specifications of the terminal (40%), a state that is a standard for power saving mode (20%), and a state that is a standard for ultra power saving mode (10%), Operating based on HTTP adaptive streaming to adaptively determine the bit rate of video segments and adaptive frame extraction based on terminal resources for blocking harmful videos in a mobile terminal,
The adaptive frame extraction module extracts video frames excluding duplicate video frames to increase the efficiency of video harmfulness analysis, calculates SSIM values for two adjacent video frames, and compares the calculated SSIM values with a specific threshold. to determine the degree of similarity,
When the battery state of the user terminal enters the power saving mode (20%), the harmfulness analysis engine maintains the image resizing size and SSIM threshold value adjusted in the previous step and increases the frame extraction interval to twice the standard frame extraction interval. By reducing the number of operations, the load on terminal resources is reduced.

The terminal resource-based adaptive frame extraction and streaming control system for blocking harmful video in a mobile terminal of the present invention classifies the remaining battery state of the terminal and adaptively determines a frame extraction algorithm according to the state, and the proposed system is There is an effect of minimizing battery consumption of the terminal and blocking harmful videos by extracting frames suitable for the state of the terminal playing the video and determining harmfulness changes.

If the battery condition is sufficient, an algorithm that prioritizes the performance of hazardous change determination is applied. However, when the battery consumption rate increases and the terminal needs to prepare for entering power saving mode, the image resizing size is reduced to reduce the CPU load required for similarity comparison, and the load by the hazard analysis engine is reduced by adjusting the SSIM threshold to a small size. . When the terminal enters power saving mode, the frame extraction interval is doubled and applied to reduce the load caused by harmfulness analysis. It is applied to the section to minimize the load by hazard analysis. In the case of various threshold values and battery state standards used in the system proposed in the present invention, values may be set differently according to the preference of a service provider providing a streaming service. In addition, in addition to mosaic processing used as a streaming control method in the proposed system, various methods such as blocking streaming sessions, degrading video quality, and outputting harmful video guide phrases can be applied.

1 is a diagram showing the structure of a real-time streaming blocking system based on a conventional video harmfulness analysis result.
2 is an overall flowchart of a technique for adaptive frame extraction based on UE resource characteristics according to the present invention.
3 is a diagram illustrating a process of adaptive frame extraction and streaming control based on harmfulness change when the battery of the user terminal is sufficient.
4 is a video harmfulness change determination based on the harmfulness analysis result for each extracted frame.
Figure 5 is a frame extraction interval and similarity comparison determination for each harmful video level when the battery of the user terminal is sufficient.
6 is an example of similarity comparison between video frames.
7 is an example of mosaic processing of an image information averaging method.
8 illustrates comparison and determination of frame extraction intervals and similarities in harmful stages when the battery state of a user terminal needs to prepare for entering a power saving mode.
9 is a frame extraction interval and similarity comparison determination for harmful levels when the battery state of the user terminal is in power saving mode (20%).
10 is a comparison and determination of frame extraction interval and similarity for each harmful level when the battery state of the user terminal is in ultra power saving mode (10%).

Hereinafter, the configuration and operation of a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the present invention, if it is determined that a detailed description of a related known technology or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, drawing numbers are assigned the same drawing numbers in different drawings when indicating the same configuration.

The present invention provides a terminal resource-based adaptive frame extraction and streaming control system and method for blocking harmful video in a mobile terminal.

Terminal resources required to block harmful videos are divided into factors related to the processing capacity of the terminal (CPU share, battery consumption) and response results for harmful videos. Factors related to the processing power of the user terminal include CPU occupancy and battery consumption, and in the case of response results to harmful videos, there are the number of frames extracted and the time required for harmfulness analysis. The user terminal has a deep learning-based hazard analysis engine. The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine. The harmfulness analysis engine transfers the harmfulness analysis result for each extracted frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with changes in terminal resources required for harmfulness analysis. A terminal resource characteristic that is a criterion for adjusting the frame extraction method is the remaining battery of the user's terminal. The remaining battery state is divided into a state that is determined to be sufficient (40%), a state that is the standard for power saving mode (20%), and a state that is the standard for ultra power saving mode (10%) according to the specifications of the terminal. If the remaining battery of the terminal is 40% or more, an adaptive frame extraction algorithm based on the similarity comparison between harmfulness changes and frames is applied. When the remaining battery state of the user terminal is in preparation for entering the power saving mode, it prepares for rapid battery consumption by controlling the image resizing size and SSIM threshold, which have little effect on CPU load during the harmful video blocking process. When the remaining battery of the terminal is in the power saving mode or ultra power saving mode, the load of the terminal is reduced by additionally applying frame extraction interval control and similarity comparison.

A terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal operates based on HTTP (Hyper Text Transfer Protocol) adaptive streaming. HTTP adaptive streaming is a technology for ensuring quality of experience (QoE) for users when streaming video over the Internet, and adaptively determines the bit rate of video segments to be requested according to the current network condition. Therefore, it is possible to provide seamless streaming service. In addition, since HTTP adaptive streaming operates based on HTTP, it does not cause a firewall problem and has the advantage of being able to utilize an existing web server. In HTTP adaptive streaming, a server encodes and stores a video in segments having a fixed playback length and different bit rates. Information on video segments is stored as a Media Presentation Description (MPD) file and delivered to the client after streaming starts. The client determines the bit rate of the next video segment to be requested using the video segment related information of the received MPD file and the measured network condition.

The MPD file includes video segment information (720P, 240P quality duration), video segment quality (bit rate), and server URL information when a server streams a video.

A deep learning-based harmfulness analysis engine that performs harmfulness analysis on frames extracted from a terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal is composed of a Convolutional Neural Network (CNN) model. Since the CNN model is specialized in learning using the characteristics of input images, it is suitable for the task of distinguishing between harmful and non-harmful images. When the frame extracted from the user terminal is transmitted to the harmfulness analysis engine, the harmfulness analysis engine outputs whether the frame is harmful and the time required for the harmfulness analysis as an analysis result. The present invention relates to an adaptive frame extraction and streaming control mechanism for responding in real time by reflecting terminal resource characteristics for streaming video in which harmfulness changes exist. do not describe In addition, harmful videos refer to videos with obscenities that are unsuitable for minors.

A CNN-based model receives an image as an input and then extracts features from the image to classify scenes or extract features from the image. The harmfulness analysis engine is a harmful image classification engine that trains a CNN-based model using a data set classified into harmful and harmless images. The image set used for learning is previously labeled as harmful or harmless. When an arbitrary image is input, the harmfulness analysis engine extracts scene information of the image and classifies whether the image is harmful or harmless through a learned model. The scene information of an image means things such as pixel RGB value distribution, motion detection results in the scene, and HSV (Hue, Saturation, Value) values.

Accordingly, the frame transmitted to the harmfulness analysis engine should be a representative frame of a scene for which harmfulness is to be determined. In order to extract a representative frame of a scene, the present invention divides the scene based on non-overlapping frames through similarity comparison and extracts the frame. In the case of similarity comparison calculation, it is not a task performed inside the hazard analysis engine, but a task performed on the frame extraction algorithm. The similarity comparison operation is calculated based on the formula described after converting the extracted frame into Mat format.

2 shows an overall flow chart of a terminal resource-based adaptive frame extraction and streaming control system for blocking harmful video in a mobile terminal according to the present invention. Device resources (remaining battery, CPU load) required to analyze the harmfulness of a video being played in real time affect the harmfulness analysis load. The proposed system performs a frame extraction technique considering the processing capacity of the terminal used for harmfulness analysis by adjusting the contexts that affect the load of terminal resources based on the remaining battery of the terminal playing the video. The remaining battery status of the user device is determined when the battery is sufficient (more than 40%), when the battery consumption rate needs to be reduced to prepare for power saving mode (40 to 20%), and when entering power saving mode (20 to 10%). %) and when entering ultra power saving mode (10~0%). Algorithm changes according to each battery state are described in detail. The criterion for determining the battery state of the user terminal is a value selectable by the service provider according to the specifications of the terminal, and 40%, 20%, and 10% are set as the criteria in the present invention.

3 shows a harmfulness analysis and streaming control process when the remaining battery of a user terminal is in a sufficient state in a terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal.

When a video stream is transmitted from the server, the proposed system extracts video frames according to an HTTP-based adaptive frame extraction algorithm when the remaining battery of the user terminal is sufficient. The extracted frames are delivered to the deep learning-based hazard analysis engine 100. The harmfulness analysis engine 100 transmits the harmfulness analysis result in the form of harmfulness probability for each frame to the video streaming blocking control module 120 and the adaptive frame extraction module 110 through harmfulness analysis on the input video frame.

The adaptive frame extraction module 110 determines the change in harmfulness of the video according to the harmfulness analysis result, and determines the frame extraction interval and similarity comparison according to the harmfulness change. The proposed system extracts frames by applying a fixed interval and similarity comparison until a video is determined to be harmful in order to minimize the load by preventing duplicate frame extraction. Extraction through similarity comparison between frames calculates the SSIM value, and if it exceeds a preset threshold, it is judged as a similar frame and excluded from the extraction target. include in

A terminal resource-based adaptive frame extraction and streaming control system for blocking harmful video in a mobile terminal of the present invention includes a server; A user terminal having an adaptive frame extraction module, a deep learning-based harmfulness analysis engine, and a streaming control module that performs mosaic processing in the case of harmful videos,
The terminal resources required to block harmful videos of the user terminal include factors related to the processing capacity of the terminal and the number of frames extracted and the time required for harmfulness analysis in the case of response results to harmful videos.
In the terminal resources required to block harmful videos of the user terminal, factors related to the processing capacity of the terminal include CPU occupancy and battery consumption,
The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine, and the harmfulness analysis engine transmits the harmfulness analysis result for each frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with the change in terminal resources necessary for the harmfulness analysis, and the terminal resource characteristic that is the criterion for adjusting the frame extraction method is the state of the remaining battery of the user terminal,
The state of the remaining battery of the user terminal is divided into a state that is determined to be sufficient according to the specifications of the terminal (40%), a state that is a standard for power saving mode (20%), and a state that is a standard for ultra power saving mode (10%), It operates based on HTTP adaptive streaming that adaptively determines the bit rate of video segments and adaptive frame extraction based on terminal resources for blocking harmful videos in mobile terminals,
The adaptive frame extraction module extracts video frames excluding duplicate video frames to increase the efficiency of video harmfulness analysis, calculates SSIM values for two adjacent video frames, and compares the calculated SSIM values with a specific threshold. to determine the degree of similarity,
When the battery state of the user terminal enters power saving mode (20%), the harmfulness analysis engine maintains the image resizing size and SSIM threshold value adjusted in the previous step and increases the frame extraction interval to twice the standard frame extraction interval. By reducing the number of operations, the load on terminal resources is reduced.

In addition, the terminal resource-based adaptive frame extraction and streaming control method for blocking harmful videos in a mobile terminal of the present invention

server; Device resource (battery status) based adaptive frame extraction and streaming control system including a user terminal having an adaptive frame extraction module, a deep learning-based harmfulness analysis engine, and a streaming control module that performs mosaic processing in case of harmful video at,
The terminal resources required to block harmful videos of the user terminal include factors related to the processing capacity of the terminal and the number of frames extracted and the time required for harmfulness analysis in the case of response results to harmful videos.
In the terminal resources required to block harmful videos of the user terminal, factors related to the processing capacity of the terminal include CPU occupancy and battery consumption,

The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine, and the harmfulness analysis engine transmits the harmfulness analysis result for each frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with the change in terminal resources necessary for the harmfulness analysis, and the terminal resource characteristic that is the criterion for adjusting the frame extraction method is the state of the user terminal's remaining battery,

The state of the remaining battery of the user terminal is divided into a state that is determined to be sufficient according to the specifications of the terminal (40%), a state that is a standard for power saving mode (20%), and a state that is a standard for ultra power saving mode (10%), Operating based on HTTP adaptive streaming to adaptively determine the bit rate of video segments and adaptive frame extraction based on terminal resources for blocking harmful videos in a mobile terminal,
The adaptive frame extraction module extracts video frames excluding duplicate video frames to increase the efficiency of video harmfulness analysis, calculates SSIM values for two adjacent video frames, and compares the calculated SSIM values with a specific threshold. to determine the degree of similarity,
When the battery state of the user terminal enters the power saving mode (20%), the harmfulness analysis engine maintains the image resizing size and SSIM threshold value adjusted in the previous step and increases the frame extraction interval to twice the standard frame extraction interval. By reducing the number of operations, the load on terminal resources is reduced.

4 illustrates a process of determining a change in harmfulness of a streaming video based on a harmfulness analysis result for each extracted frame when the remaining battery of the user terminal is in a sufficient state. The harmfulness change judgment is performed for each playback length of the video segment, and based on the harmfulness analysis result of the last frame of the segment, whether or not the next video segment is harmful is predicted. (a) Harmless → Harmful is when the last frame of a harmless video segment is determined to be a harmful frame as a result of the harmfulness analysis. The next video segment is predicted as a harmful video segment. Frames are extracted without applying similarity comparison. (b) Harmful → Harmful is when the last frame of a harmful video segment is determined to be a harmful frame as a result of harmfulness analysis. To balance the load by frame extraction and real-time response to harmful video streaming, (a) harmless → harmful Frames are extracted without applying similarity comparison at intermediate intervals between harmful → harmless cases. (c) Harmful → harmless is when the last frame of a harmful video segment is judged to be harmless as a result of harmfulness analysis. In order to minimize unnecessary frame extraction, frames are extracted by applying similarity comparison at long fixed intervals within the segment.

5 shows frame extraction intervals and similarity comparison decisions according to changes in video harmfulness. The frame extraction interval is based on 30 FPS video, and if the harmfulness of the streaming video changes to the harmful part, the shortest interval and forced extraction, if the harmful part is maintained, the middle interval and forced extraction, and the harmless part, the harmfulness of the streaming video If this has changed, the longest interval and similarity comparison are applied. The frame extraction interval used in this stage is used as a criterion for the frame extraction interval used in other stages. In the proposed technique, the minimum number of frames to properly determine the harmfulness of a streaming video is set as the candidate interval without excessive load on the harmfulness analysis engine through repeated experiments in advance.

6 shows an example of performing similarity comparison between frames by calculating SSIM (Structural Similarity; Structural Similarity Index) values for frame extraction based on similarity comparison between video frames proposed by the present system. SSIM is a representative metric mainly used in evaluating the perceptual quality of 2D images. Since the human visual system is specialized in deriving structure-related information from images, it is assumed that the degree of distortion of the image structure has the greatest effect on the perceptual quality. It is a metric that is calculated based on Therefore, the SSIM value is calculated using the brightness information and distortion information of each pixel of the two images to be compared for similarity. The streaming control system based on the characteristics of a video stream proposed in the present invention compares the similarity between extracted video frames based on SSIM values. As shown in FIG. 5, SSIM values for the 10th and 11th frames among the extracted video frames have a small value of 0.4795. If the SSIM value is small, the proposed streaming control system based on video stream characteristics determines that the corresponding frames are included in different scenes. On the other hand, SSIM values for the 11th video frame and the 12th video frame have a large value of 0.8258, so in this case, it is determined that the corresponding frames are included in the same scene. The example of FIG. 5 is an example to show how video frame extraction based on similarity comparison operates. There is a difference with the mechanism for extracting frames. In the present invention, the criterion for scene change is the part where the SSIM value greatly changes when comparing similarities between adjacent video frames. For example, in the case of a scene including a video frame extracted first in FIG. 4 , it can be regarded as a scene in which frames similar to the extracted frame are gathered. On the other hand, in the case of a scene including the second extracted video frame, compared to a scene including the first extracted video frame, it can be regarded as a scene starting from a frame in which the SSIM value changes greatly.

When x is the original image and y is the comparison image, the average brightness I(x,y) of the pixels of the two images is first calculated as follows to calculate the SSIM value. is the average brightness of the original image x, is the average brightness of the comparison image y, and denotes the image brightness constant. The value calculated by the corresponding formula means a normalized value to represent the average brightness of the two images by combining the average brightness of the original image and the comparison image.

After calculating the average brightness, the standard deviation value of each image is the contrast value of the image. , and calculate the standard deviation of the two images as follows. is the original image Brightness standard deviation of , is the comparison image is the brightness standard deviation of , and denotes an image contrast constant. Like the formula for calculating the average brightness of two images, the value calculated by the formula means a normalized value to represent the contrast between the two images by combining the brightness standard deviations of the original image and the comparison image.

After calculating the image contrast, the image structure index is calculated using the brightness covariance and brightness standard deviation between the two images. The value of is calculated by the following formula. brightness covariance is a value indicating how correlated the brightness information of each image is with each other denotes an image structure constant. The value calculated by the corresponding formula means the relative ratio of the value of the associated brightness standard deviation between the two images to the brightness standard deviation of each of the two images.

image structure constant is the image contrast constant is calculated using the following formula. Since the process of calculating the image contrast includes the brightness standard deviation of the image, it is related to the process of calculating the image structure index. However, since the brightness covariance is used to calculate the image structure index, the corresponding formula and In the same way, image structure constants are calculated.

Finally, the SSIM value is calculated as the following formula by combining the average brightness of the two images, the contrast value of the images, and the image structure index. The values of the weights a, b, and c used at this time are all set to 1. What this formula means is to compare the similarity between the original image and the comparison image by reflecting each factor that affects human perception quality. have an influence

7 shows an example of mosaic processing of an image information averaging method used in the proposed streaming control system based on terminal resource characteristics.

When a harmful video is detected by the harmfulness analysis engine 100, the streaming control module 120 may perform mosaic processing or block streaming if the video is determined to be harmful.

To respond to harmful video streaming in real time, the terminal adds an arbitrary layer on the video frame being played and then divides the area within the frame into several grid cells. Mosaic processing is performed by averaging image information of pixels included in areas within a frame divided into grid cells. Therefore, if the number of grid cells is small, the mosaic processing effect increases, but the processing time increases because the averaging process for image information becomes complicated, and if the number of grid cells is large, the processing time decreases but harmful video exposure can be effectively reduced To this extent, the mosaic effect does not appear. In the case of the mosaic processing method used by the system proposed in the present invention, it is possible to analyze harmfulness without damaging the information of the extracted video frame because it is performed by adding an arbitrary layer without directly accessing the image information of the video frame being played. There are advantages.

8 illustrates frame extraction intervals and similarity comparison determinations for harmful stages when the battery state of the user terminal needs to prepare for entering a power saving mode (20%). When the battery consumption of the user terminal increases and it is necessary to prepare for entering power saving mode, the proposed system adjusts the image size resizing and the SSIM threshold, which is the criterion for similarity comparison, in the similarity comparison process with relatively small CPU load fluctuation range. For the image reszing size and SSIM threshold, values of 200 by 200 and 0.8 are used when the battery is sufficient, and values of 100 by 100 and 0.7 are used otherwise. Reducing the size of image resizing can reduce the degree of CPU load required for similarity comparison, and reducing the SSIM threshold can reduce the load by harmfulness analysis by reducing the number of frames extracted.

9 shows frame extraction intervals and similarity comparison determinations for each harmful level in power saving mode. When the battery state of the user terminal enters the power saving mode (20-10%), the proposed system maintains the image resizing size and SSIM threshold adjusted in the previous step. In addition, the load on terminal resources can be reduced by reducing the number of times the harmfulness analysis engine is operated by increasing the frame extraction interval to twice the reference frame extraction interval.

10 illustrates frame extraction intervals and similarity comparison determinations for harmful levels when the battery state of the user terminal is in the ultra power saving mode (10-0%). When the user terminal enters deep power saving mode, the proposed system maintains the image resizing size and SSIM threshold at the values of the previous step. In addition, the frame extraction interval is increased to three times the reference frame extraction interval, and the similarity comparison is collectively applied to all harmful sections to minimize frame extraction. Through this, the number of times the harmfulness analysis engine is operated is minimized, thereby minimizing the load on terminal resources. The multiplier for increasing the reference frame extraction interval is a constant that can be set by the service provider according to the environment, and is set to 2 and 3 in the proposed system, respectively.

The terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal proposed in the present invention classifies the remaining battery state of the terminal and adaptively determines the frame extraction algorithm according to the remaining battery state. . The proposed system aims to minimize battery consumption of the terminal and block harmful videos by extracting frames suitable for the state of the terminal playing video and determining harmfulness changes. If the battery condition is sufficient, an algorithm that prioritizes the performance of hazardous change determination is applied. However, when the battery consumption rate of the user terminal increases and the terminal needs to prepare for entering the power saving mode (20%), the image resizing size is reduced to reduce the CPU load required for similarity comparison and the SSIM threshold value is adjusted small to reduce harmfulness. Reduce the load on the analysis engine. When the battery state of the user terminal enters the power saving mode (20%), the frame extraction interval is doubled and applied to reduce the load caused by harmfulness analysis, and when the terminal enters the ultra power saving mode (10-0%), the frame extraction interval is applied. Minimize the load by hazard analysis by increasing the extraction interval by three times and applying similarity comparison to all hazard change sections.

For reference, a CNN (Convolutional Neural Network)-based model receives an image as an input and then extracts features of the image to extract and classify the features of the scene of the image. CNN is a model designed to process images more effectively by applying an artificial neural network to the filtering technique used in the field of image processing. The input image data is expressed as a 3D tensor structure composed of height, width, and channels. Various images can be distinguished by extracting the features of images coming into the tensor structure, labeling the features of the images with specific values, and proceeding with learning. The features of the extracted image include image borders, color distribution, and degree of distortion. These various features are made into a feature map and compared with the input image features after learning. The structure of the CNN used to create the feature map is described in detail below. The harmfulness analysis engine is a harmful image classification engine that trains a CNN-based model using a data set classified into harmful and harmless images. The image set used for learning is previously labeled as harmful or harmless. When an arbitrary image is input, the harmfulness analysis engine extracts scene information of the image and classifies whether the image is harmful or harmless through a learned model. Scene information of an image means things such as the distribution of pixel RGB values, motion detection results in a scene, and HSV (Hue, Saturation, Value) values.

In the present invention, a MobileNet-based CNN harmfulness analysis engine was used to use a model for real-time harmfulness analysis. 11 shows a general convolution operation for CNN and methods of depthwise convolution and pointwise convolution operation for MobileNet. The filter size of a general convolution operation is K × K, the image height and width are F, the number of input channels is N, and the number of filters is M. Therefore, the total amount of computation is am. However, since depthwise convolution uses a different kernel for each channel of the input feature map, the number of filters is equal to the number of input channels. In this case, since the operation is not performed in the channel direction and convolution is performed in the spatial direction, the total amount of operation is It can be seen that the amount of computation is smaller than that of general convolution. Since pointwise convolution uses 1x1 conv, the number of filters is reduced and the amount of computation is reduced through dimensionality reduction. Since the operation is performed in the channel direction opposite to the operation direction of depthwise convolution, the total amount of operation is Same as

11 is a comparison diagram of convolution operation methods.

12 is an overall structure of a CNN based on MobileNet used in the harmfulness analysis engine used in the present invention. This structure, also called depthwise seperable conv, uses depthwise convolution and pointwise convolution separately, unlike general convolutions that simultaneously perform channel-direction and spatial-direction convolution operations, perform spatial and channel-direction operations separately, and then combine them. Therefore, the amount of operation of a general convolution and the amount of operation of Depthwise seperable conv ( ) + ( ), assuming that F = 5, N = 2, M = 64, and K = 2, a difference of about 3.76 times the amount of calculation occurs.

12 shows the structure of depthwise seperable conv.

는 원본 이미지x의 평균 밝기,

는 비교 이미지 y의 평균 밝기, 그리고

은 이미지 밝기 상수를 의미한다. 이미지의 콘트라스트는 원본 이미지와 비교 이미지의 밝기 표준편차를 종합하여 두 이미지의 콘트라스트를 나타낼 수 있도록 정규화된 값을 의미한다. 이 때,

는 원본 이미지x의 밝기 표준편차,

는 비교 이미지 y의 밝기 표준 편차, 그리고

는 이미지 콘트라스트 상수를 의미한다. 그리고, 이미지의 구조 지수는 두 이미지 각각의 밝기 표준편차 대비 두 이미지 사이의 연관된 밝기 표준 편차의 값의 상대적 비율을 의미한다. 이 때 밝기 공분산

는 각 이미지의 밝기 정보가 서로 얼마나 연관성이 있는지를 나타내기 위한 값이며

는 이미지 구조 상수를 의미한다. a, b, c가 의미하는 것은 평균 밝기, 이미지 콘트라스트, 그리고 이미지 구조 지수를 SSIM을 비교할 때 얼마나 반영할지에 대한 가중치이다. 사람의 지각 품질에 영향을 주는 각 요소를 모두 반영하여 원본 이미지와 비교 이미지 간의 유사도를 비교해야 하기 때문에 가중치를 모두 1로 설정하였고, 이로 인해 SSIM 계산에 있어서 각 요소가 모두 같은 영향력을 가지게 된다.Accordingly, the frame transmitted to the harmfulness analysis engine should be a representative frame of a scene for which harmfulness is to be determined. In order to extract a representative frame of a scene, the present invention divides the scene based on non-overlapping frames through similarity comparison and extracts the frame. In the case of similarity comparison calculation, it is not a task performed inside the hazard analysis engine, but a task performed on the frame extraction algorithm. In the present invention, the similarity comparison operation is calculated based on the formula described below after converting the extracted frame into Mat format. x and y is the image converted to Mat form a, b ,c The values of are all set to 1. The average brightness means a value normalized to represent the average brightness of the two images by combining the average brightness of the original image and the comparison image. At this time,

is the original image average brightness of x,

is the comparison image the average brightness of y, and

denotes the image brightness constant. The contrast of the image means a normalized value to represent the contrast of the two images by combining the brightness standard deviations of the original image and the comparison image. At this time,

is the original image brightness standard deviation of x,

is the comparison image the brightness standard deviation of y, and

denotes an image contrast constant. And, the structure index of an image means a relative ratio of a brightness standard deviation value between two images to a brightness standard deviation value of each of the two images. In this case, the brightness covariance

is a value indicating how correlated the brightness information of each image is with each other

denotes an image structure constant. a, b, c means a weight for how much to reflect average brightness, image contrast, and image structure index when comparing SSIM. Since the similarity between the original image and the comparison image must be compared by reflecting each factor that affects human perception quality, all weights are set to 1, so that each factor has the same influence in SSIM calculation.

average brightness

contrast of images

image structure index

The terminal resource-based adaptive frame extraction and streaming control system for blocking harmful video in a mobile terminal proposed in the present invention classifies the terminal's remaining battery state and adaptively determines a frame extraction algorithm according to the state. The proposed system aims to minimize battery consumption of the terminal and block harmful videos by extracting frames suitable for the state of the terminal playing video and determining harmfulness change. If the battery condition is sufficient, an algorithm that prioritizes the performance of hazardous change determination is applied. However, when the battery consumption rate increases and the terminal needs to prepare for entering power saving mode, the image resizing size is reduced to reduce the CPU load required for similarity comparison, and the load caused by the hazard analysis engine is reduced by adjusting the SSIM threshold value small. . When the terminal enters power saving mode, the frame extraction interval is doubled and applied to reduce the load caused by harmfulness analysis. It is applied to the section to minimize the load due to hazard analysis. In the case of various threshold values and battery state standards used in the system proposed in the present invention, values may be set differently according to the preference of a service provider providing a streaming service. In addition, in addition to mosaic processing used as a streaming control method in the proposed system, various methods such as blocking streaming sessions, degrading video quality, and outputting harmful video guide phrases can be applied.

Embodiments according to the present invention are implemented in the form of program instructions that can be executed through various computer means, and can be recorded in a computer readable recording medium. The computer readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include storage, hard disks, magnetic media such as floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - A hardware device configured to store and execute program instructions in storage media such as magneto-optical media, ROM, RAM, flash memory, and storage may be included. Examples of program instructions may include those produced by a compiler, machine language codes as well as high-level language codes that can be executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention.

As described above, the method of the present invention is implemented as a program and can be read using computer software on a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) ) can be stored in

Although described with reference to specific embodiments of the present invention, the present invention is not limited to the same configuration and operation as the specific embodiments to illustrate the technical idea as described above, and within the limit that does not deviate from the technical spirit and scope of the present invention It can be implemented with various modifications, and the scope of the present invention should be determined by the claims described later.

100: hazard analysis engine
110: adaptive frame extraction module
120: streaming control module

Claims (22)

server;
A user terminal having an adaptive frame extraction module, a deep learning-based harmfulness analysis engine, and a streaming control module that performs mosaic processing in the case of harmful videos,
The terminal resources required to block harmful videos of the user terminal include factors related to the processing capacity of the terminal and the number of frames extracted and the time required for harmfulness analysis in the case of response results to harmful videos.
In the terminal resources required to block harmful videos of the user terminal, factors related to the processing capacity of the terminal include CPU occupancy and battery consumption,
The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine, and the harmfulness analysis engine transmits the harmfulness analysis result for each frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with the change in terminal resources necessary for the harmfulness analysis, and the terminal resource characteristic that is the criterion for the frame extraction method is the state of the remaining battery of the user terminal.
The state of the remaining battery of the user terminal is divided into a state that is determined to be sufficient according to the specifications of the terminal (40%), a state that is a standard for power saving mode (20%), and a state that is a standard for ultra power saving mode (10%), It operates based on HTTP adaptive streaming that adaptively determines the bit rate of video segments and adaptive frame extraction based on terminal resources for blocking harmful videos in mobile terminals,
The adaptive frame extraction module extracts video frames excluding duplicate video frames to increase the efficiency of video harmfulness analysis, calculates SSIM values for two adjacent video frames, and compares the calculated SSIM values with a specific threshold. to determine the degree of similarity,
When the battery state of the user terminal enters the power saving mode (20%), the harmfulness analysis engine maintains the image resizing size and SSIM threshold value adjusted in the previous step and increases the frame extraction interval to twice the standard frame extraction interval. A terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal, which reduces the load on terminal resources by reducing the number of operations. delete According to claim 1,
When the remaining battery state of the user terminal is 40% or more, an adaptive frame extraction algorithm based on similarity comparison between harmfulness changes and frames is applied,
When the remaining battery state of the user terminal needs to prepare for entering the power saving mode (20%), prepare for rapid battery consumption by controlling the image resizing size and SSIM threshold that have little effect on CPU load in the process of blocking harmful videos ,
Device resource-based adaptation for blocking harmful videos in a mobile device, which reduces the load of the device by additionally applying frame extraction interval control and similarity comparison when the remaining battery state of the user device is in power saving mode or ultra power saving mode Enemy frame extraction and streaming control system. According to claim 1,
In HTTP adaptive streaming, the server encodes and stores a video into segments having a fixed playback length and different bit rates, and information about video segments is stored as a Media Presentation Description (MPD) file and sent to a client after streaming starts. and the client determines the bit rate of the next video segment to be requested using the video segment information of the received MPD file and the measured network state;
The MPD file is a terminal resource for blocking harmful videos in a mobile terminal, including video segment information (720P, 240P quality duration), video segment quality (bit rate), and server URL information when a server streams a video. based adaptive frame extraction and streaming control system. According to claim 1,
The deep learning-based harmfulness analysis engine that performs harmfulness analysis on frames extracted from terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal uses a Convolutional Neural Network (CNN) model,
The harmfulness analysis engine is a harmful image classification engine that trains a CNN-based model using a data set classified into harmful and harmless images, and the image set used for learning is previously labeled whether each image is harmful or harmless. When an arbitrary image is input, the hazard analysis engine extracts the scene information of the image and classifies whether the image is harmful or harmless through the learned model, and the scene information of the image is the distribution of pixel RGB values. , motion detection results in the scene, and HSV (Hue, Saturation, Value) values,
Frames transmitted to the harmfulness analysis engine were divided into scenes based on non-overlapping frames through similarity comparison to extract a representative frame of the scene to be determined harmfulness, and frames were extracted. In the case of similarity comparison calculation, the harmfulness analysis A terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal, which is not performed inside the engine but is performed on a frame extraction algorithm. According to claim 1,
The user terminal resources (remaining battery, CPU load) required to analyze the harmfulness of the video being played in real time affect the load of harmfulness analysis, and the load of the terminal resource is affected based on the remaining battery of the device playing the video. By controlling contexts, the frame extraction technique considering the processing capacity of the user terminal used for hazard analysis is performed. It is classified into the case of preparing for power saving mode by reducing the speed (40~20%), entering power saving mode (20~10%) and ultra power saving mode (10~0%). , The criterion for determining the battery state of the user terminal is a value selectable by the service provider according to the specifications of the terminal, and 40%, 20%, and 10% are set as the criteria, and terminal resources for blocking harmful videos in the mobile terminal based adaptive frame extraction and streaming control system. According to claim 1,
When a video stream is transmitted from the server, if the remaining battery of the user terminal is in a sufficient state, video frames are extracted according to an HTTP-based adaptive frame extraction algorithm, and the extracted frames are transmitted to the deep learning-based harmfulness analysis engine. , The harmfulness analysis engine transmits the harmfulness analysis result in the form of harmfulness probability for each frame to the video streaming blocking control module and the adaptive frame extraction module through harmfulness analysis on the input video frame,
The adaptive frame extraction module determines the harmfulness change of the video according to the harmfulness analysis result, determines the frame extraction interval and similarity comparison according to the harmfulness change, and prevents the extraction of duplicate frames to minimize the load of the video. Frames are extracted by applying a fixed interval and similarity comparison until they are determined to be harmful. Extraction through similarity comparison between frames calculates the SSIM value, and if it exceeds a preset threshold, it is judged as a similar frame and excluded from the extraction target. A terminal resource-based adaptive frame extraction and streaming control system for blocking harmful video in a mobile terminal, which is determined as a frame of a different scene and included in the extraction target when the value is lower than the threshold. According to claim 1,
When comparing the similarity between frames by calculating the SSIM (Structural Similarity) value for frame extraction based on the similarity comparison between video frames, the SSIM value is the brightness information for each pixel of the two images to be compared. and distortion information, and the video stream characteristic-based streaming control system compares the similarity between extracted video frames based on the SSIM value,
SSIM values for the 10th frame and 11th frame among the extracted video frames have a small value of 0.4795, and if the SSIM value is small, the video stream characteristic-based streaming control system determines that the frames are included in different scenes, On the other hand, since the SSIM values for the 11th video frame and the 12th video frame have a large value of 0.8258, in this case, it is determined that the frames are included in the same scene,
In order to extract video frames based on similarity comparison, a streaming control system based on video stream characteristics distinguishes multiple scenes included in a video segment, and the criterion for scene change is the part where the SSIM value changes greatly when comparing the similarity between adjacent video frames. In the case of a scene containing the first extracted video frame, it can be viewed as a scene in which frames similar to the extracted frame are gathered, whereas in the case of a scene containing the second extracted video frame, the first extracted Compared to a scene containing video frames, it can be viewed as a scene starting from a frame in which the SSIM value changes greatly.
When x is the original image and y is the comparison image, in order to calculate the SSIM value, the average brightness I(x,y) of the pixels of the two images is first calculated as follows: is the average brightness of the original image x, is the average brightness of the comparison image y, and means the image brightness constant, and the value calculated by the corresponding formula means a normalized value to represent the average brightness of the two images by combining the average brightness of the original image and the comparison image,

After calculating the average brightness, the standard deviation value of each image is the contrast value of the image. , and the standard deviation of the two images is calculated as the following formula, where: is the original image Brightness standard deviation of , is the comparison image is the brightness standard deviation of , and Means an image contrast constant, and the value calculated by the formula means a normalized value to represent the contrast of the two images by combining the brightness standard deviations of the original image and the comparison image,

After calculating the image contrast, the image structure index is calculated using the brightness covariance and brightness standard deviation between the two images. Calculate the value of , where the brightness covariance is a value for indicating how correlated the brightness information of each image is with each other, Means an image structure constant, and the value calculated by the corresponding formula means the relative ratio of the value of the associated brightness standard deviation between the two images to the brightness standard deviation of each of the two images,

The image structure constant is the image contrast constant It is calculated as in the following formula using

Finally, the SSIM value is calculated as follows by combining the average brightness of the two images, the contrast value of the images, and the image structure index, the values of weights a, b, and c are all set to 1, and the original image and the comparison image Comparing the similarity between the two, and calculating the SSIM value because the weights are all set to 1,

Device resource-based adaptive frame extraction and streaming control system for blocking harmful video in mobile terminal. According to claim 1,
When the battery consumption of the user terminal is increased and it is necessary to prepare for entering the power saving mode (20%), the size of the image resizing in the similarity comparison process with relatively small fluctuations in CPU load and the SSIM threshold value that is the criterion for similarity comparison are adjusted. , The image resizing size and SSIM threshold use values of 200 by 200 and 0.8, respectively, when the battery is sufficient, and values of 100 by 100 and 0.7 are used otherwise. Adaptive frame extraction based on terminal resources for blocking harmful videos in a mobile terminal, which can reduce the degree of CPU load and reduce the number of frames extracted by reducing the SSIM threshold to reduce the load by harmfulness analysis. and a streaming control system. delete According to claim 1,
When the battery state of the user terminal is in ultra power saving mode (10-0%), frame extraction intervals and similarity comparisons are determined for harmful stages, and when the terminal enters ultra power saving mode, the image resizing size and SSIM threshold are set to those of the previous stage. value, the frame extraction interval is increased to 3 times the standard frame extraction interval, and similarity comparison is applied collectively to all harmfulness sections to minimize frame extraction. It is possible to minimize the load for the reference frame extraction interval, and the multiplier for increasing the reference frame extraction interval is a constant that can be set by the service provider according to the environment. In the proposed system, it is set to 2 and 3, respectively. Enemy frame extraction and streaming control system. server; Adaptive frame extraction and streaming control system based on terminal resources (battery status) including a user terminal having an adaptive frame extraction module, a deep learning-based harmfulness analysis engine, and a streaming control module that performs mosaic processing in case of harmful videos at,
The terminal resources required to block harmful videos of the user terminal include factors related to the processing capacity of the terminal and the number of frames extracted and the time required for harmfulness analysis in the case of response results to harmful videos.
In the terminal resources required to block harmful videos of the user terminal, factors related to the terminal's processing capability include CPU occupancy and battery consumption,
The frame of the video segment received from the server is extracted in real time and transmitted to the harmfulness analysis engine, and the harmfulness analysis engine transmits the harmfulness analysis result for each frame to the harmfulness change determination module to determine the harmfulness change of the video being played. The harmfulness determination result is used to adjust the frame extraction method along with the change in terminal resources necessary for the harmfulness analysis, and the terminal resource characteristic that is the criterion for the frame extraction method is the state of the remaining battery of the user terminal.
The state of the remaining battery of the user terminal is divided into a state that is determined to be sufficient according to the specifications of the terminal (40%), a state that is a standard for power saving mode (20%), and a state that is a standard for ultra power saving mode (10%), Operating based on HTTP adaptive streaming to adaptively determine the bit rate of video segments and adaptive frame extraction based on terminal resources for blocking harmful videos in a mobile terminal,
The adaptive frame extraction module extracts video frames excluding duplicate video frames to increase the efficiency of video harmfulness analysis, calculates SSIM values for two adjacent video frames, and compares the calculated SSIM values with a specific threshold. to determine the degree of similarity,
When the battery state of the user terminal enters the power saving mode (20%), the harmfulness analysis engine maintains the image resizing size and SSIM threshold value adjusted in the previous step and increases the frame extraction interval to twice the standard frame extraction interval. A terminal resource-based adaptive frame extraction and streaming control method for blocking harmful videos in a mobile terminal, which reduces the load on terminal resources by reducing the number of operations. delete According to claim 12,
When the remaining battery state of the user terminal is 40% or more, an adaptive frame extraction algorithm based on similarity comparison between harmfulness changes and frames is applied,
When the remaining battery state of the user terminal needs to prepare for entering power saving mode, prepare for rapid battery consumption by controlling the image resizing size and SSIM threshold that have little effect on CPU load in the process of blocking harmful videos,
Device resource-based adaptation for blocking harmful videos in a mobile device, which reduces the load of the device by additionally applying frame extraction interval control and similarity comparison when the remaining battery state of the user device is in power saving mode or ultra power saving mode Enemy frame extraction and streaming control method. According to claim 12,
In HTTP adaptive streaming, the server encodes and stores a video into segments having a fixed playback length and different bit rates, and information about video segments is stored as a Media Presentation Description (MPD) file and sent to a client after streaming starts. and the client determines the bit rate of the video segment to be requested next using the video segment related information of the received MPD file and the measured network state, and extracts the terminal resource-based adaptive frame for blocking harmful videos in the mobile terminal. and a streaming control method. According to claim 12,
The deep learning-based harmfulness analysis engine that performs harmfulness analysis on frames extracted from terminal resource-based adaptive frame extraction and streaming control system for blocking harmful videos in a mobile terminal uses a Convolutional Neural Network (CNN) model,
The harmfulness analysis engine is a harmful image classification engine that trains a CNN-based model using a data set classified into harmful and harmless images, and the image set used for learning is previously labeled whether each image is harmful or harmless. When an arbitrary image is input, the hazard analysis engine extracts the scene information of the image and classifies whether the image is harmful or harmless through the learned model, and the scene information of the image is the distribution of pixel RGB values. , motion detection results in the scene, and HSV (Hue, Saturation, Value) values,
Frames transmitted to the harmfulness analysis engine were divided into scenes based on non-overlapping frames through similarity comparison to extract a representative frame of the scene to be determined harmfulness, and frames were extracted. In the case of similarity comparison calculation, the harmfulness analysis A terminal resource-based adaptive frame extraction and streaming control method for blocking harmful videos in a mobile terminal, which is not performed inside an engine but is performed on a frame extraction algorithm. According to claim 12,
The user terminal resources (remaining battery, CPU load) required to analyze the harmfulness of the video being played in real time affect the load of harmfulness analysis, and the load of the terminal resource is affected based on the remaining battery of the device playing the video. By adjusting the contexts, the frame extraction technique considering the processing capacity of the user terminal used for hazard analysis is performed, and the remaining battery state of the user terminal is determined by the battery consumption rate It is classified into cases where it is necessary to prepare for power saving mode by reducing (40~20%), entering power saving mode (20~10%) and ultra power saving mode (10~0%). The criterion for determining the battery state of the user terminal is a value selectable by the service provider according to the specifications of the terminal, and 40%, 20%, and 10% are set as the criterion, based on terminal resources for blocking harmful videos in mobile terminals. Adaptive frame extraction and streaming control method. According to claim 12,
When a video stream is transmitted from the server, if the remaining battery of the user terminal is in a sufficient state, video frames are extracted according to an HTTP-based adaptive frame extraction algorithm, and the extracted frames are transmitted to the deep learning-based harmfulness analysis engine. , The harmfulness analysis engine transmits the harmfulness analysis result in the form of harmfulness probability for each frame to the video streaming blocking control module and the adaptive frame extraction module through harmfulness analysis on the input video frame,
The adaptive frame extraction module determines the change in the harmfulness of the video according to the harmfulness analysis result, determines the frame extraction interval and similarity comparison according to the harmfulness change, and the proposed system minimizes the load by preventing the extraction of duplicate frames. In order to do this, frames are extracted by applying a fixed interval and similarity comparison until the video is determined to be harmful, and extraction through similarity comparison between frames calculates the SSIM value and determines it as a similar frame when it exceeds a preset threshold and extracts it. Excluded from the target, and if the value is lower than the threshold, determined as a frame of a different scene and included in the extraction target, terminal resource-based adaptive frame extraction and streaming control method for blocking harmful video in a mobile terminal. According to claim 12,
When comparing the similarity between frames by calculating the SSIM (Structural Similarity) value for frame extraction based on the similarity comparison between video frames, the SSIM value is the brightness information for each pixel of the two images to be compared. and distortion information, and the video stream characteristic-based streaming control system compares the similarity between extracted video frames based on the SSIM value,
SSIM values for the 10th frame and 11th frame among the extracted video frames have a small value of 0.4795, and if the SSIM value is small, the video stream characteristic-based streaming control system determines that the frames are included in different scenes, On the other hand, since the SSIM values for the 11th video frame and the 12th video frame have a large value of 0.8258, in this case, it is determined that the frames are included in the same scene,
In order to extract video frames based on similarity comparison, the streaming control system based on video stream characteristics distinguishes multiple scenes included in a video segment, and the criterion for scene change is the part where the SSIM value greatly changes when comparing the similarity between adjacent video frames. The scene containing the first extracted video frame can be viewed as a scene in which frames similar to the extracted frame are gathered, while the scene containing the second extracted video frame can be viewed as the first extracted video frame. It can be viewed as a scene starting from a frame in which the SSIM value changes significantly compared to a scene containing frames.
When x is the original image and y is the comparison image, in order to calculate the SSIM value, the average brightness I(x,y) of the pixels of the two images is first calculated as follows: is the average brightness of the original image x, is the average brightness of the comparison image y, and means the image brightness constant, and the value calculated by the corresponding formula means a normalized value to represent the average brightness of the two images by combining the average brightness of the original image and the comparison image,

After calculating the average brightness, the standard deviation value of each image is the contrast value of the image. , and the standard deviation of the two images is calculated as the following formula, where: is the original image Brightness standard deviation of , is the comparison image is the brightness standard deviation of , and Means an image contrast constant, and the value calculated by the formula means a normalized value to represent the contrast of the two images by combining the brightness standard deviations of the original image and the comparison image,

After calculating the image contrast, the image structure index is calculated using the brightness covariance and brightness standard deviation between the two images. Calculate the value of , where the brightness covariance is a value for indicating how correlated the brightness information of each image is with each other, Means an image structure constant, and the value calculated by the corresponding formula means the relative ratio of the value of the associated brightness standard deviation between the two images to the brightness standard deviation of each of the two images,

The image structure constant is the image contrast constant It is calculated as in the following formula using

Finally, the SSIM value is calculated as follows by combining the average brightness of the two images, the contrast value of the images, and the image structure index, the values of weights a, b, and c are all set to 1, and the original image and the comparison image Comparing the similarity between the two, and calculating the SSIM value because the weights are all set to 1,

Device resource-based adaptive frame extraction and streaming control method for blocking harmful video in mobile terminal. According to claim 12,
When the battery consumption of the user terminal is increased and it is necessary to prepare for entering the power saving mode (20%), the size of the image resizing in the similarity comparison process with relatively small fluctuations in CPU load and the SSIM threshold value that is the criterion for similarity comparison are adjusted. , Image resizing size and SSIM threshold use 200 by 200 and 0.8 values, respectively, when the battery is sufficient, and 100 by 100 and 0.7 values, respectively, when the battery is sufficient. Device resource-based adaptation for blocking harmful videos in a mobile terminal, characterized in that the required degree of CPU load can be reduced, and the reduction of the SSIM threshold can reduce the number of frames extracted to reduce the load caused by harmfulness analysis. Enemy frame extraction and streaming control method. delete According to claim 12,
When the battery state of the user terminal is in ultra power saving mode (10-0%), frame extraction intervals and similarity comparisons are determined for harmful stages, and when the terminal enters ultra power saving mode, the image resizing size and SSIM threshold are set to those of the previous stage. value, the frame extraction interval is increased to 3 times the standard frame extraction interval, the similarity comparison is applied collectively in all harmfulness sections to minimize frame extraction, and the number of times the harmfulness analysis engine is operated is minimized, thereby minimizing the load on terminal resources. can be minimized, and the multiplier for increasing the reference frame extraction interval is a constant that can be set by the service provider according to the environment. Extraction and Streaming Control Methods.