KR102676153B1 - Apparatus and method for blocking real-time harmful content based on artificial intelligence algorithm - Google Patents

Abstract

A method and device for blocking harmful content in real time based on an artificial intelligence (AI) algorithm are disclosed. According to an embodiment of the present disclosure, a method of blocking harmful content in real time based on an AI algorithm performed by a device includes classifying a plurality of data included in content uploaded by a user by content type; calculating a harmfulness index of each of the plurality of data by applying at least one harmfulness determination module to each of the plurality of data classified by content type; Obtaining information on whether the specific data is harmful data by inputting specific data whose harmfulness index falls within a certain range among the plurality of data into a pre-trained AI model; and filtering the content based on the specific data being identified as harmful data by the obtained information. and updating the at least one harmfulness determination module to output the harmfulness index of the specific data as a value exceeding the specific range based on the obtained information.

Description

Real-time harmful content blocking device and method based on artificial intelligence algorithm {APPARATUS AND METHOD FOR BLOCKING REAL-TIME HARMFUL CONTENT BASED ON ARTIFICIAL INTELLIGENCE ALGORITHM}

This disclosure relates to the field of blocking harmful content, and more specifically, to a device and method for blocking harmful content in real time based on an artificial intelligence algorithm.

As information and communication technology develops exponentially, the majority of citizens, including children and adolescents, can easily produce and share multimedia content using various online services.

As mentioned above, there is a problem that in a highly informationized environment, harmful adult material containing sexual profanity and violence is being distributed without permission by an unspecified number of people through various online services such as UCC, blogs, web hard, P2P, and instant messaging applications. As a result, violations of personal privacy and juvenile sexual crimes are rapidly increasing, and various methods are being devised to prevent this.

Previously, inspectors first checked for harmfulness through thumbnails of multiple uploaded videos. The inspector played specific content that was recognized as harmful through the first inspection among multiple videos, and finally identified the harmful video.

However, as the number of multimedia contents such as videos increases explosively, the above inspection method has a problem in that the burden on service providers increases rapidly due to an increase in management costs.

In addition, since this inspection method primarily allows distribution and playback of videos, there is a problem that there is always a time when the video is exposed.

Public Patent Publication No. 10-2014-0025113, 2014.03.04

The present disclosure was designed to solve the above-described problems, and the purpose of the present disclosure is to provide a real-time harmful content blocking device and method based on an artificial intelligence algorithm.

Additionally, the purpose of the present disclosure is to provide a device and method for blocking harmful content by determining the level of sensationality and/or violence in media content (eg, video content, image content, text content, etc.).

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

According to an embodiment of the present disclosure, a method of blocking harmful content in real time based on an artificial intelligence (AI) algorithm performed by a device includes dividing a plurality of data included in content uploaded by a user into a content type. Classifying by star; calculating a harmfulness index of each of the plurality of data by applying at least one harmfulness determination module to each of the plurality of data classified by content type; Obtaining information on whether the specific data is harmful data by inputting specific data whose harmfulness index falls within a certain range among the plurality of data into a pre-trained AI model; and filtering the content based on the specific data being identified as harmful data by the obtained information. and updating the at least one harmfulness determination module to output the harmfulness index of the specific data as a value exceeding the specific range based on the obtained information.

And, the content type includes image data, audio data, text data, and metadata, and the classifying step includes extracting at least one of image data, audio data, text data, and metadata from the content. It may include a classification step.

And, the at least one harmfulness determination module includes a module for calculating a harmfulness index of the image data, a module for calculating a harmfulness index of the audio data, a module for calculating a harmfulness index of the text data, and a module for calculating the harmfulness index of the metadata. It may include at least one of the modules that calculate the exponent.

And, extracting the text data from the content includes identifying a text area in the image data; and rotating the image data so that the text area becomes horizontal, and then extracting the text data from the image data.

And, the method may further include identifying the content as harmful content and filtering the content based on the harmfulness index of data of at least one content type among the plurality of data exceeding the specific range. there is.

And, the step of obtaining the information is based on identifying that the harmfulness index of data of the remaining content types, excluding the specific data, among the plurality of data is less than the specific range and that the harmfulness index of the specific data is within the specific range. Thus, it may include the step of inputting the specific data into the pre-trained AI model to obtain information on whether the specific data is harmful data.

And, the pre-trained AI model may be further trained based on at least one content detected as harmful content through the at least one harmfulness determination module.

Additionally, the method may further include providing a harmful index for each of the plurality of data classified by content type and a user interface (UI) indicating that the specific data is harmful content.

According to an embodiment of the present disclosure, an apparatus for blocking harmful content in real time based on an artificial intelligence (AI) algorithm includes at least one memory; and at least one processor, wherein the at least one processor includes at least one memory; and at least one processor, wherein the at least one processor classifies a plurality of data included in content uploaded by a user by content type, and calculates a harmful index for each of the plurality of data classified by content type. a hazard judgment module; and inputting specific data whose harmful index falls within a certain range among the plurality of data into a pre-trained AI model to obtain information on whether the specific data is harmful data, and determining whether the specific data is harmful based on the obtained information. An AI model that filters the content based on identification as data, and updates the at least one harmfulness determination module to output a harmfulness index of the specific data as a value exceeding the specific range based on the obtained information. May include an inference module.

Additionally, the harmfulness determination module may extract and classify at least one of image data, audio data, text data, and metadata from the content.

And, the harmfulness determination module identifies a text area in the image data; And after rotating the image data so that the text area becomes horizontal, the text data on the image data can be extracted.

Additionally, the AI model inference module may identify the content as harmful content and filter the content based on the harmfulness index of at least one content type of data among the plurality of data exceeding the specific range.

And, the at least one processor is based on identifying that the harmfulness index of data of the remaining content types, excluding the specific data, among the plurality of data is less than the specific range and that the harmfulness index of the specific data is within the specific range. , information on whether the specific data is harmful data can be obtained by inputting the specific data into the pre-trained AI model.

In addition, the at least one processor includes a user interface (UI) generation module, and the UI generation module is configured to determine a harmfulness index of each of the plurality of data classified by content type and a UI (UI) indicating that the specific data is harmful content. user interface) can be provided.

In addition, a computer program stored in a computer-readable recording medium for execution to implement the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to various embodiments of the present disclosure, a real-time harmful content blocking device and method based on an artificial intelligence algorithm may be provided.

In addition, according to various embodiments of the present disclosure, an apparatus and method for blocking harmful content by determining the degree of sensationality and/or violence in media content can be provided.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is a diagram briefly illustrating a real-time harmful content blocking system based on an artificial intelligence algorithm according to an embodiment of the present disclosure.
Figure 2 is a block diagram showing the configuration of a device for blocking harmful content in real time based on an artificial intelligence algorithm, according to an embodiment of the present disclosure.
Figure 3 is a diagram illustrating a flow for blocking harmful content in real time based on an artificial intelligence algorithm, according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating a process in which a device extracts multiple types of data from content, according to an embodiment of the present disclosure.
5A and 5B are diagrams for explaining a process in which a device extracts text data from image data, according to an embodiment of the present disclosure.
Figure 6 is a flowchart for explaining a real-time method of blocking harmful content based on an artificial intelligence algorithm, according to an embodiment of the present disclosure.

Like reference numerals refer to like elements throughout this disclosure. This disclosure does not describe all elements of the embodiments, and general content or overlapping content between embodiments in the technical field to which this disclosure pertains is omitted. The term 'part, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'part, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part is said to “include” a certain component, this means that it may further include other components, rather than excluding other components, unless specifically stated to the contrary.

Throughout the specification, when a member is said to be located “on” another member, this includes not only cases where a member is in contact with another member, but also cases where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the present disclosure will be described with reference to the attached drawings.

In this specification, 'device according to the present disclosure' includes all various devices that can perform computational processing and provide results to the user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take the form of any one.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may consist of one or multiple processors. At this time, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. One or more processors control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be accomplished in the device itself that performs the artificial intelligence according to the present disclosure, or may be accomplished through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, a plurality of weights may be updated so that loss or cost values obtained from the artificial intelligence model are reduced or minimized during the learning process. Artificial neural networks may include deep neural networks (DNN), for example, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, etc., but are not limited to the examples described above.

According to an exemplary embodiment of the present disclosure, a processor may implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that allows machines to learn by imitating human biological neurons. Methodology of artificial intelligence includes supervised learning, in which the answer (output data) to the problem (input data) is determined by providing input data and output data together as training data according to the learning method, and only input data is provided without output data. In unsupervised learning, in which the solution (output data) to the problem (input data) is not determined, and a reward is given from the external environment whenever an action is taken in the current state, , It can be divided into reinforcement learning, which conducts learning in the direction of maximizing these rewards. In addition, artificial intelligence methodologies can be divided according to the architecture, which is the structure of the learning model. The widely used architecture of deep learning technology is convolutional neural network (CNN) and recurrent neural network (RNN). , Transformer, generative adversarial networks (GAN), etc.

The devices and systems may include artificial intelligence models. An artificial intelligence model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. Artificial intelligence models may be composed of neural networks (or artificial neural networks) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to an overall model in which artificial neurons (nodes), which form a network through the combination of synapses, change the strength of the synapse connection through learning and have problem-solving capabilities. Neurons in a neural network can contain combinations of weights or biases. A neural network may include one or more layers consisting of one or more neurons or nodes. By way of example, a device may include an input layer, a hidden layer, and an output layer. The neural network that makes up the device can infer the result (output) to be predicted from arbitrary input (input) by changing the weight of neurons through learning.

The processor creates a neural network, trains or learns a neural network, performs calculations based on received input data, generates an information signal based on the results, or generates a neural network. The network can be retrained. Neural network models include CNN (Convolution Neural Network), R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), and RNN, such as GoogleNet, AlexNet, and VGG Network. (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network , LSTM (Long Short-Term Memory) Network, Classification Network, etc., but the processor may include one or more processors for performing operations according to neural network models. For example, a neural network may include a deep neural network.

Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), and LSTM. (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield) Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network) ), Liquid State Machine (LSM), Extreme Learning Machine (ELM), Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), Those skilled in the art will understand that it may include any neural network, including, but not limited to, KN (Kohonen Network) and AN (Attention Network).

According to an exemplary embodiment of the present disclosure, the processor may support a Convolution Neural Network (CNN), a Region with Convolution Neural Network (R-CNN), a Region Proposal Network (RPN), a Recurrent Neural Network (RNN), such as GoogleNet, AlexNet, VGG Network, etc. ), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC/QA, Text Analysis, Dialog System, GPT-3 , GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, and Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation, and Data Creation for ResNet data intelligence. , but is not limited to this.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

1 is a diagram briefly illustrating a real-time harmful content blocking system based on an artificial intelligence algorithm according to an embodiment of the present disclosure.

As shown in FIG. 1, the real-time harmful content blocking system 1000 based on an artificial intelligence algorithm includes a device 100 and a terminal device 200-1, 200-2, ... 200-N used by a plurality of users. ) (N is a natural number of 2 or more) and a data server 301.

In FIG. 1, the device 100 is implemented as a desktop, and the terminal devices 200-1, 200-2, ... 200-N used by multiple users are implemented as smartphones. is disclosed, but is not limited thereto.

As an example, the device 100 and the terminal devices 200-1, 200-2, ... 200-N used by a plurality of users are various types of electronic devices (e.g., laptops, desktops, laptops, tablets). It can be implemented as a PC, slate PC device, server device, etc.), and can also be implemented as a group of devices where one or more types of devices are connected.

The device 100 included in the system 1000 and the terminal devices 200-1, 200-2, ... 200-N used by a plurality of users can communicate through the network (W).

Here, the network W may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

Additionally, the network W may include the known World Wide Web (WWW). However, the network (W) according to an embodiment of the present disclosure is not limited to the networks listed above, and may include at least some of a known wireless data network, a known telephone network, and a known wired and wireless television network.

The device 100 may perform an operation to block harmful content in real time based on an artificial intelligence (AI) algorithm. Additionally, the device 100 may create/execute web-based and/or application-based software that can block harmful content in real time based on artificial intelligence algorithms.

Terminal devices 200-1, 200-2, ... 200-N used by a plurality of users may execute web-based and/or application-based software created by the device 100. Therefore, the real-time harmful content blocking method based on an artificial intelligence algorithm executed by the device 100 in the present disclosure can also be executed on terminal devices 200-1, 200-2, ... 200-N used by multiple users. there is.

The device 100 may extract and classify data of multiple content types from content uploaded by a user. The device 100 may primarily filter a plurality of types of classified data through a hazard determination module. The device 100 may identify whether the specific data is harmful by inputting specific data that is not certain whether it is harmful data among a plurality of types of data into an AI model. Here, the corresponding AI model may be learned based on training data included in the data server 301.

Operations related to this will be described in detail with reference to FIGS. 2 to 6.

Figure 2 is a block diagram showing the configuration of a device for blocking harmful content in real time based on an artificial intelligence algorithm, according to an embodiment of the present disclosure.

As shown in FIG. 2 , device 100 may include memory 110, communication module 120, display 130, and processor 140.

The memory 110 may store one or more instructions for the processor 140 to perform various operations. The memory 110 may store data supporting various functions of the device 100 and a program for the operation of the processor 140, and may store input/output data (e.g., learning data, input from the user). voice/text data, etc.) can be stored.

The memory 110 is a flash memory type, hard disk type, solid state disk type, SDD type (Silicon Disk Drive type), and multimedia card micro type. ), card-type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable programmable read) -only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk may include at least one type of storage medium.

The communication module 120 may include one or more components including circuits that enable communication with external devices (eg, data servers and/or terminal devices used by users, etc.). For example, the communication module 120 may include at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module. The communication module 120 may receive monitoring data from an external device.

The display 130 displays (outputs) information processed by the device 100 (e.g., learning data, data extracted from content, UI including the harmfulness index of each of a plurality of data classified by content type, etc.) do.

For example, the display may display execution screen information of an application running on the device 100, or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

The processor 140 can block harmful content in real time based on an artificial intelligence algorithm by executing one or more instructions stored in the memory 110. That is, the processor 140 can control overall operations and functions using each component of the device 100. At this time, the memory 110 and the processor 140 may each be implemented as separate chips. Alternatively, the memory 110 and processor 140 may be implemented as a single chip.

The processor 140 may include a hazard determination module 140-1, an AI model learning module 140-2, an AI model inference module 140-3, and a UI creation module 140-4. Each of the hazard determination module 140-1, the AI model learning module 140-2, the AI model inference module 140-3, and the UI creation module 140-4 may be controlled by the processor 140. .

The harmfulness determination module 140-1 can detect and filter whether multiple types of content included in content uploaded by a user are harmful.

Referring to FIG. 3, the content 300 uploaded by the user includes text data 300-1, image data 300-2, audio data 300-3, video data 300-4, and meta data. It may consist of at least one of (300-5).

The hazard determination module (140-1) includes text data (300-1), image data (300-2), audio data (300-3), video data (300-4), and metadata (300-5). Automatic filtering (310) can be performed on at least one.

As an example, the hazard determination module 140-1 includes text data 300-1, image data 300-2, audio data 300-3, video data 300-4, and metadata 300-1. 5) The hazard index can be calculated for each of at least one of the following. Here, the harmful index quantifies the harmfulness of the data.

As an example, the harmfulness determination module 140-1 performs natural language processing on the text data 300-1 to include profanity and/or violent or suggestive phrases in the text data 300-1. It is possible to identify whether it is in place or not. If the text data 300-1 contains profanity and/or violent or sensational phrases, the harmfulness determination module 140-1 may set a high harmfulness index for the text data 300-1.

That is, the harmfulness determination module 140-1 can detect whether a harmful phrase exists in the text data 300-1 and then calculate the harmfulness index by reflecting this.

As another example, the harmfulness determination module 140-1 may detect whether the image data 300-2 and/or the video data 300-4 contain violent or sensational areas. If the image data 300-2 and/or the video data 300-4 contain violent or sensational areas, the harmfulness determination module 140-1 may use the image data 300-2 and/or the video data. A high hazard index can be assigned to the data (300-4).

As another example, the hazard determination module 140-1 may extract text data on the image data 300-2 and/or video data 300-4 and calculate a hazard index for the extracted text data. there is.

As an example, referring to FIG. 5A , the hazard determination module 140-1 may detect/identify text areas 510 and 520 in image data 530. At this time, the harmfulness determination module 140-1 can detect/identify not only the text areas 510 and 520 but also the text boundary. Since the text areas 510 and 520 are not horizontal, the device may rotate the image data 530 so that the text areas 510 and 520 are horizontal and then detect the text.

As another example, the harmfulness determination module 140-1 may identify whether the audio data 300-3 includes profanity and/or violent or suggestive phrases. First, the harmfulness determination module 140-1 may obtain text data corresponding to the audio data 300-3 by applying a speech-to-text (STT) algorithm to the audio data 300-3. The harmfulness determination module 140-1 can identify whether the text data contains violent or sensational phrases by performing natural language processing on the text data.

Additionally or alternatively, as shown in FIG. 4, the harmfulness determination module 140-1 may extract/classify text data, image data/video data, audio data, and metadata from content uploaded by the user. there is.

For example, based on a video taught remotely by a user being uploaded, the device may extract/classify image/video data, audio data, and metadata from the video. As another example, based on metaverse-related data being uploaded by a user, the device may extract audio data and text data from the metaverse-related data.

The harmfulness determination module 140-1 may identify content as harmful content and filter the content based on the harmfulness index of data of at least one content type among the plurality of data exceeding a specific range.

The harmfulness determination module 140-1 may be composed of a plurality of modules that calculate the harmfulness index for each content type. That is, the harmfulness determination module 140-1 may include a text data harmfulness determining module, an image data harmfulness determining module, an audio data harmfulness determining module, an image data harmfulness determining module, and a metadata harmfulness determining module.

Each of the text data harmfulness judgment module, image data harmfulness judgment module, audio data harmfulness judgment module, video data harmfulness judgment module, and metadata harmfulness judgment module automatically detects whether harmful parts exist in data of the corresponding content type. It can be set to do so.

Additionally or alternatively, the harmfulness determination module 140-1 may calculate the degree of similarity between pre-registered harmful content and newly input content. The higher the similarity between the pre-registered harmful content and the newly input content, the higher the harmfulness determination module 140-1 can calculate the harmfulness index of the newly input content.

The AI model learning module 140-2 can train an AI model to output information about whether specific data is harmful data. The AI model learning module 140-2 may additionally learn an AI model based on at least one content detected as harmful content through at least one harmfulness determination module.

The AI model inference module 140-3 can obtain information about whether specific data is harmful data by inputting specific data into the AI model. That is, the AI model inference module 140-3 can utilize the learned AI model.

As an example, referring to FIG. 3, the AI model learning module 140-2 may store at least one content detected as harmful content and/or at least one data detected as harmful content through the harmfulness determination module 140-1. The AI model 320 can be additionally trained based on the content that has been removed (330).

As another example, the AI model learning module 140-2 may additionally learn an AI model based on content identified as harmless content through the AI model 320.

The hazard determination module 140-1 may be updated based on information output from the AI model 320. That is, the harmfulness determination module 140-1 may be updated based on information indicating whether the specific content output by the AI model 320 is harmful content or harmless content.

The AI model inference module 140-3 inputs specific data (e.g., data that is not certain whether it is harmful data) into the AI model 320 to obtain information about whether the specific data is harmful data or harmless data. You can.

If the specific data is harmful data, the processor 140 may remove the specific data or/and content containing the specific data (360). If the specific data is harmless data, the processor 140 may provide content including the specific data to the user.

The AI model may be composed of multiple AI models that output information on whether the input data for each content type is harmful data. That is, the AI model may include an AI model corresponding to text data, an AI model corresponding to image data, a model corresponding to audio data AI, an AI model corresponding to video data, and an AI model corresponding to metadata. .

For example, an AI model corresponding to text data can learn high-level semantic features through an encoder-decoder structure. Multi-scale feature integration can be applied between the encoder and decoder to combine feature maps. The AI model corresponding to text data can learn not only the text area but also the text boundary.

The AI model corresponding to image data and/or the AI model corresponding to audio may be composed of one or more CNN layers and/or LSTM layers.

The UI creation module 140-4 may generate a UI indicating the harmfulness index of each of the plurality of data classified by content type and that the specific data is harmful content.

Figure 6 is a flowchart illustrating a method for blocking harmful content in real time based on an artificial intelligence algorithm, according to an embodiment of the present disclosure.

The device can classify a plurality of data included in content uploaded by the user by content type (S610).

Here, content types can be divided into image data, audio data, text data, and meta data. The device may extract and classify at least one of image data, audio data, text data, and metadata from content uploaded by the user.

As an example, the device can identify a text area in image data included in content. The device may rotate the image data so that the text area becomes horizontal and then extract text data on the image data.

The device may calculate the harmfulness index of each of the plurality of data by applying at least one harmfulness determination module to each of the plurality of data classified by content type (S620).

Here, the at least one harmfulness determination module includes a module for calculating the harmfulness index of image data, a module for calculating the harmfulness index of audio data, a module for calculating the harmfulness index of text data, and a module for calculating the harmfulness index of metadata. It may include at least one of the following.

In other words, when image data and text data are extracted/classified from content, the device can apply the harmfulness judgment module corresponding to the image data to the image data, and the harmfulness judgment module corresponding to the text data to the text data. there is.

Based on the harmful index of data of at least one content type among the plurality of data exceeding a specific range, the device may identify the content as harmful content and filter the content. That is, if the harmful index of one type of data in the content exceeds a certain range (i.e., if the data of that type is harmful data), the device can detect and filter the content as harmful content.

The device can obtain information about whether the specific data is harmful data by inputting specific data whose harmful index falls within a certain range among the plurality of data into a pre-trained AI model (S630). Here, the specific range may be a predefined range, but may be changed by the device.

Specifically, based on the fact that the harmfulness index of the data of the remaining content types excluding the specific data among the plurality of data is identified as being less than a certain range and the harmfulness index of the specific data being identified as being within a certain range, the device uses the specific data as a pre-trained AI model. You can obtain information about whether specific data is harmful data by entering it in .

The fact that the hazard index of specific data falls within a certain range may mean that it is unclear whether the specific data is hazardous data. In other words, if the harmfulness determination module does not clearly determine whether specific data is harmful, the device can input the specific data into a pre-trained AI model to obtain information indicating whether it is harmful.

Based on the obtained information identifying specific data as harmful data, the device can filter the corresponding content (S640).

Additionally, the device may update at least one hazard determination module to output the hazard index of specific data as a value exceeding a specific range based on information acquired by the AI model (S650). That is, the device can update the hazard determination module based on the results obtained through the AI model.

Additionally or alternatively, the device may further learn an AI model based on at least one content detected as harmful content through at least one harmfulness determination module.

Additionally, the device may provide the user with a harmful index for each of a plurality of data classified by content type and a user interface (UI) indicating that the specific data is harmful content.

As another example, based on the specific data being identified as harmless data by the acquired information, the device may provide the content as is. At this time, the device may additionally provide a UI including a hazard index calculated for one or more types of data constituting the content.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be Read Only Memory (ROM), Random Access Memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, etc.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which this disclosure pertains will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

100: device
110: memory
120: communication module
130: display
140: processor

Claims (16)

In a method of blocking harmful content in real time based on an artificial intelligence (AI) algorithm performed by a device, the method includes:
Classifying a plurality of data included in content uploaded by a user by content type;
calculating a harmfulness index of each of the plurality of data by applying at least one harmfulness determination module to each of the plurality of data classified by content type;
Obtaining information on whether the specific data is harmful data by inputting specific data whose harmfulness index falls within a certain range among the plurality of data into a pre-trained AI model; and
filtering the content based on the specific data being identified as harmful data by the obtained information; and
Comprising the step of updating the at least one harmfulness determination module to output a harmfulness index of the specific data as a value exceeding the specific range based on the obtained information,
The content type includes image data, audio data, text data, and metadata,
The classification step is,
Extracting and classifying at least one of image data, audio data, text data, and metadata from the content,
The at least one hazard determination module,
At least one of a module for calculating a harmfulness index of the image data, a module for calculating a harmfulness index of the audio data, a module for calculating a harmfulness index of the text data, and a module for calculating a harmfulness index of the metadata; ,
The step of obtaining the information is,
Among the plurality of data, except for the specific data, the harmfulness index of the data of the remaining content types is less than the specific range, and the harmfulness index of the specific data is identified as being within the specific range, so whether the specific data is harmful cannot be clearly determined. In this case, inputting the specific data into the pre-trained AI model to obtain information on whether the specific data is harmful data,
Further comprising identifying the content as harmful content and filtering the content based on the harmfulness index of data of at least one content type among the plurality of data exceeding the specific range,
The pre-trained AI model is,
Based on at least one of at least one content detected as harmful content through the at least one harmful content, content from which at least one data detected as harmful content has been removed, and content identified as harmless content through the AI model. Additional learning is achieved,
The method wherein the at least one harmfulness determination module is updated based on information indicating whether the specific content output by the AI model is harmful content or harmless content.
delete delete According to paragraph 1,
The step of extracting the text data from the content includes:
identifying a text area in the image data; and
A method comprising rotating the image data so that the text area is in a horizontal form, and then extracting the text data on the image data. delete delete delete According to paragraph 1,
The method further includes providing a harmful index for each of the plurality of data classified by content type and a user interface (UI) indicating that the specific data is harmful content. In a device that blocks harmful content in real time based on an artificial intelligence (AI) algorithm, the device:
at least one memory; and
Contains at least one processor,
The at least one processor,
Classifies multiple data included in content uploaded by the user by content type,
a harmfulness determination module that calculates a harmfulness index for each of the plurality of data classified by content type; and
Among the plurality of data, input specific data whose harmful index falls within a certain range into a pre-trained AI model to obtain information on whether the specific data is harmful data;
Filtering the content based on the specific data being identified as harmful data by the obtained information, and
An AI model inference module that updates the at least one harmfulness determination module to output a harmfulness index of the specific data as a value exceeding the specific range based on the obtained information,
The content type includes image data, audio data, text data, and metadata,
The hazard determination module is,
Extracting and classifying at least one of image data, audio data, text data, and metadata from the content,
The at least one hazard determination module,
At least one of a module for calculating a harmful index of the image data, a module for calculating a harmful index of the audio data, a module for calculating a harmful index of the text data, and a module for calculating a harmful index of the meta data; ,
The AI model inference module is,
Among the plurality of data, except for the specific data, the harmfulness index of the data of the remaining content types is less than the specific range, and the harmfulness index of the specific data is identified as being within the specific range, so whether the specific data is harmful cannot be clearly determined. In this case, inputting the specific data into the pre-trained AI model to obtain information on whether the specific data is harmful data,
The AI model inference module is,
Based on the harmful index of data of at least one content type among the plurality of data exceeding the specific range, identifying the content as harmful content and filtering the content,
The pre-trained AI model is,
Based on at least one of at least one content detected as harmful content through the at least one harmfulness determination module, content from which at least one data detected as harmful content has been removed, and content identified as harmless content through the AI model. Additional learning is achieved,
The at least one harmfulness determination module is updated based on information indicating whether the specific content output by the AI model is harmful content or harmless content.
delete delete According to clause 9,
The hazard determination module is,
identify a text area in the image data; and
A device for rotating the image data so that the text area becomes horizontal and then extracting the text data on the image data. delete delete delete According to clause 9,
The at least one processor includes a user interface (UI) creation module,
The UI creation module is,
An apparatus that provides a harmful index for each of the plurality of data classified by content type and a user interface (UI) indicating that the specific data is harmful content.

Images