KR102276728B1 - Multimodal content analysis system and method - Google Patents

Abstract

The present invention relates to a multi-modal content analysis system and a method therefor, wherein the multi-modal content analysis method performed by the multi-modal content analysis system for analyzing content uploaded through a media channel includes: a) sharing content through the media channel collecting contents including audio, video, subtitles and metadata from at least one content sharing platform that provides a service; b) generating the collected contents as image document data in which content text information including context for each content and user response text information including a viewer's posting response are combined through natural language processing-based pre-processing; c) extracting a feature vector for each frame of the image document data using a multimodal analysis model based on text analysis, and learning a classifier for content classification using the extracted feature vector; and d) when new content is found through content search for the media channel, providing an analysis result for new content using the learned multi-modal analysis model, wherein the multi-modal analysis model comprises: Based on the similarity between the feature vectors, a classifier generated through a clustering algorithm that classifies the entire image document data into K groups is learned, and the learned classifier uses a feature vector extracted from new content to obtain a predefined category ( category) automatically.

Description

Multimodal content analysis system and method {MULTIMODAL CONTENT ANALYSIS SYSTEM AND METHOD}

The present invention relates to a multimodal content analysis system and method therefor.

The device environment in which users access media is diversifying, and the amount of content that users can access is increasing. In particular, in the rapidly developing mobile environment, users use personalized devices to consume content and share experiences with users around them. The content provision service intends to activate content consumption by analyzing and utilizing data generated from such individual content consumption histories and SNS relationships.

A content recommendation algorithm that selects suitable content for users from among content providing services is becoming an essential element in all content sharing platforms. The content recommendation algorithm derives and provides the content that is inferred that the user will most need through the content analysis algorithm based on the viewer's consumption history or content metadata.

The content analysis algorithm is used as important information to analyze not only the content of content such as audio, video, and subtitles, but also the number of views, impressions, and viewing time, as well as comments, likes/dislikes, and shares.

1 is a diagram for explaining data changes used in a general content analysis algorithm.

As shown in FIG. 1 , as new types of content sharing platforms such as social network services (Facebook, Twitter) and content sharing sites (Youtube, Flickr) are activated, video content being shared through the content sharing platform is single-modal data. It is developing into multimodal data in the form of multivariate data such as audio, video, subtitles, and metadata.

As the number of new content and users is rapidly increasing through content sharing platforms, general data analysis algorithms cannot handle the rapidly increasing amount of computation to analyze large amounts of multi-modal content, There is a problem that I cannot afford. Therefore, the content analysis algorithm requires a technique for quickly and accurately analyzing a large amount of multimodal data.

In order to solve the above problems, according to an embodiment of the present invention, according to an embodiment of the present invention, it is possible to quickly and accurately analyze content in the form of multimodal data, and provides an analysis result in a form combining the content of the content and the viewer's reaction. has a purpose

However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for achieving the above technical problem, a multi-modal content analysis method performed by a multi-modal content analysis system for analyzing content uploaded through a media channel according to an embodiment of the present invention comprises: a) the media channel Collecting content including audio, video, subtitles and metadata from at least one content sharing platform that provides a content sharing service through b) generating the collected contents as image document data in which content text information including context for each content and user response text information including a viewer's posting response are combined through natural language processing-based pre-processing; c) extracting a feature vector for each frame of the image document data using a multimodal analysis model based on text analysis, and learning a classifier for content classification using the extracted feature vector; and d) when new content is found through content search for the media channel, providing an analysis result for new content using the learned multi-modal analysis model, wherein the multi-modal analysis model comprises: Based on the similarity between the feature vectors, a classifier generated through a clustering algorithm that classifies the entire image document data into K groups is learned, and the learned classifier uses a feature vector extracted from new content into a predefined category ( category) automatically.

In addition, a multi-modal content analysis system for analyzing content uploaded through a media channel according to another embodiment of the present invention includes a memory in which a program for performing a multi-modal content analysis method is recorded; and a processor for executing the program, wherein the processor receives audio, video, subtitles and metadata from at least one content sharing platform that provides a content sharing service through the media channel by executing the program. Contents are collected, and the collected contents are generated as image document data in which content text information including context for each content and user response text information including viewer's posting response are combined through natural language processing-based pre-processing. and extracting a feature vector for each frame of the image document data using a multimodal analysis model based on text analysis, learning a classifier for content classification using the extracted feature vector, and searching for content for the media channel When new content is found through K, an analysis result for the new content is provided using the learned multi-modal analysis model, wherein the multi-modal analysis model calculates the entire image document data based on the similarity between the feature vectors. A classifier generated through a clustering algorithm for classifying into groups is learned, and the learned classifier is automatically classified into a predefined category using a feature vector extracted from new content.

According to the problem solving means of the present invention described above, by using the multi-modal analysis model, users can quickly and accurately analyze a large amount of multi-modal data-type contents by labeling a specific category without having to look at numerous contents one by one, and You can look at the contents of the contents and the reaction of viewers at once.

1 is a diagram for explaining data changes used in a general content analysis algorithm.
2 is a view for explaining the overall configuration of a multi-modal content analysis system according to an embodiment of the present invention.
3 is a diagram for explaining the detailed configuration of a multi-modal content analysis system according to an embodiment of the present invention.
4 is a flowchart illustrating a multimodal content analysis method according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a screen and metadata form of content.
6 is an exemplary diagram for explaining image document data according to an embodiment of the present invention.
7 is a view for explaining the structure of a data storage module according to an embodiment of the present invention.
8 is an exemplary diagram for explaining a learning process of a multimodal analysis model according to an embodiment of the present invention.
9 is a view for explaining a content classification process of a multi-modal analysis model according to an embodiment of the present invention.
FIG. 10 is an exemplary diagram for explaining the classification result of FIG. 9 .
11 is an exemplary diagram for explaining a concentration algorithm of a multimodal analysis model according to an embodiment of the present invention.
12 is an exemplary diagram for explaining a concentration algorithm using the concentration weight of FIG. 11 .
13 is an exemplary diagram for explaining an analysis result of a multimodal analysis model according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In the present specification, a 'terminal' may be a wireless communication device with guaranteed portability and mobility, for example, any type of handheld-based wireless communication device such as a smart phone, a tablet PC, or a notebook computer. In addition, the 'terminal' may be a wired communication device such as a PC that can connect to another terminal or server through a network. In addition, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW). : World Wide Web), wired and wireless data networks, telephone networks, and wired and wireless television networks.

Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

The following examples are detailed descriptions to help the understanding of the present invention, and do not limit the scope of the present invention. Accordingly, an invention of the same scope performing the same function as the present invention will also fall within the scope of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 2 is a diagram illustrating the overall configuration of a multi-modal content analysis system according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a detailed configuration of a multi-modal content analysis system according to an embodiment of the present invention.

2 and 3 , the multi-modal content analysis system 100 includes a communication module 110 , a memory 120 , a processor 130 , a data storage module 140 , and a display module 150 . On the other hand, the content sharing platform 200 may be a system providing a multi-channel network (Multi-Channel Networks, MCN) service or YouTube, Afreeca TV, Twitch, Instagram, etc. provided by a service provider, and a user terminal ( 300) may be a terminal owned by a single video creator or creator, or a terminal owned by a viewer watching content uploaded on a media channel.

The communication module 110 provides a communication interface necessary for the multi-modal content analysis system 100 to provide a transmission/reception signal with the user terminal 300 and the content sharing platform 200 in the form of packet data by interworking with the communication network. Furthermore, the communication module 110 may perform a role of receiving a data request from the user terminal 300 and transmitting data in response thereto. Also, the communication module 110 may serve to transmit a data request to the content sharing platform 200 and receive data in response thereto.

Here, the communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

In the memory 120, a program for performing a multi-modal content analysis method is recorded. In addition, the memory 120 performs a function of temporarily or permanently storing data processed by the processor 130 . Here, the memory 120 may include a volatile storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto.

The processor 130 controls the entire process of providing the analysis result of the content using the multi-modal analysis model to the user terminal 300 or the content sharing platform 200 by executing a program for performing the multi-modal content analysis method. . Each operation performed by the processor 130 will be described in more detail later.

Here, the processor 130 may include all kinds of devices capable of processing data, such as a processor. Here, the 'processor' may refer to a data processing device embedded in hardware, for example, having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

The data storage module 140 including the relational database 140 and the storage 145 stores data accumulated while performing the multimodal content analysis method. For example, data having high-level characteristics such as text information, time information, numerical information, and small-sized binary data may be stored in the relational database 141 , and the storage 145 may be difficult to store in the relational database 140 . Data having a low-level characteristic, which are source data such as an image, video, and audio, may be stored. The storage method of the relational database 141 and the storage 145 includes a modification method according to data loading, access, and management cost of data storage.

The display module 150 outputs the analysis result of the content in the form of a text, table, or graph on the screen under the control of the processor 130 .

4 is a flowchart illustrating a multimodal content analysis method according to an embodiment of the present invention, FIG. 5 is an exemplary diagram for explaining a screen and metadata form of content, and FIG. 6 is an embodiment of the present invention It is an exemplary diagram for explaining the image document data according to the

The plurality of user terminals 300 may upload content they have created on a media channel by using various content sharing platforms 200 , and click comments and like/dislike while viewing or viewing the content uploaded to the media channel. A media activity may be performed while posting a post such as (S1).

The multi-modal content analysis system 100 collects content for each channel through the content sharing platform 200 using an application programming interface (API) (S2, S3). In this case, the content refers to various digital information provided through a communication network such as the Internet, and may refer to various information contents such as video, subtitles, images, and audio, programs, movies, music, game software, and the like.

The multi-modal content analysis system 100 may request data again after a preset waiting time when a temporary failure occurs or a data request response time is delayed when requesting data through API or crawling. However, if the data access right is lost, the logic execution for the data request is stopped or skipped after the data request is individually made to each content sharing platform.

Since the content providing method is different for each content sharing platform, the multi-modal content analysis system 100 searches for objects related to the collected content and identifies a relationship according to the searched object structure. That is, the multi-modal content analysis system 100 searches for what content is configured inside the content sharing platform 200, searches for objects such as audio, video, subtitles, and metadata constituting the content, and searches for objects within the metadata. Objects related to the viewer's post response can be identified. For example, when the content sharing platform 200 is YouTube, the multi-modal content analysis system 100 identifies channels existing in YouTube through a search function, and checks whether new channels or video contents are uploaded in YouTube at regular intervals. can be explored and discovered.

The multi-modal content analysis system 100 stores log information on the collected image content in the data storage module 140, and when searching for new content, it checks the content already searched by using the log information of the data storage module 140 and can perform a fast and efficient search for newly uploaded content.

As shown in FIG. 5 , the multimodal content analysis system 100 may collect content A through a YouTube API request, and the collected content A includes video, audio, subtitles, and metadata along a timeline. have. At this time, the metadata includes the title, description, tag, category, number of views, number of likes, number of dislikes, payload time, thumbnail, and comments of content A.

As shown in FIG. 6 , the multi-modal content analysis system 100 performs content text information including the context for each content through natural language processing-based pre-processing, and user response text information including the viewer's post response. are combined to generate image document data (S4).

In the pre-processing process of content, stopword processing, alphabet separation, stem extraction, word tokenization, etc. are performed to generate image document data in which content text information of each content and user response text information are combined. The process is done in the same way.

The image document data thus generated is stored in the data storage module 140 for learning the multi-modal analysis model. In this case, the data storage module 140 may be used by dividing the DB into a learning DB, a word dictionary DB, an analysis DB, etc. according to a related data set, and may integrate and manage each DB.

7 is a view for explaining the structure of a data storage module according to an embodiment of the present invention.

7, the relational database (RBDMS, 141) stores data having high-level characteristics such as text information, time information, numerical information, and small-sized binary data, and the storage 145 is a relational database ( 141) stores data having low-level characteristics, such as image sources, video sources, and audio sources, which are difficult to store.

The data storage module 140 may adopt and use various structures in a complex manner according to the data structure and the need for DB extension management for each content sharing platform. For example, the data storage module 140 may preferentially use the relational database 141 and use a non-relational database (NoSQL, 143) for unstructured data management. In addition, the data storage module 140 includes a DB extension node and a replication node 147 for backup, and read/write operations may be implemented to exchange a consistent response for each data object.

When the non-relational database 143-based unstructured data is stored in the relational database 141, the processor 130 stores each data in the relational database 141 according to whether there is actually a field of the collected data even if a data schema exists. Whether to save the field or process the conditional branch. When unstructured data cannot be stored using a database transaction, data storage can be omitted. If such asynchronous processing logic is performed, computing resources for processing such as data collection and processing due to simultaneous data requests and the like can be efficiently used.

If the processor 130 concurrently accesses the data storage module 140 and makes a write request, the data storage module 140 may temporarily reach the limit of simultaneous processing possible. In this case, the processor 130 may store the data collected through the work queue, and when data exists in the work queue, retrieve the data in a FIFO manner and store it in the data storage module 140 .

On the other hand, the processor 130 may perform distributed processing of data in the data storage module 140 through a method such as scale out / scale up according to the amount of data to be processed, or a plurality of Computing resources can be managed.

Referring to FIG. 4 again, the multi-modal content analysis system 100 extracts a feature vector for each frame of image document data using a multi-modal analysis model based on text analysis, and learns a classifier using the extracted feature vector. (S5, S6).

When new content is found through content search for a media channel (S7), an analysis result for the new content is provided using the learned multi-modal analysis model (S8, S9).

That is, the multi-modal content analysis system 100 classifies categories for content content by using content text information of image document data, and uses viewer response text information for content for each category. After setting these as the viewer reaction, the analysis result in which the category and the viewer reaction are combined for new content is provided. Accordingly, users can look at contents in a corresponding category at once by labeling a specific category without having to look at numerous contents one by one.

8 is an exemplary diagram for explaining a learning process of a multimodal analysis model according to an embodiment of the present invention.

The multimodal analysis model can be implemented using a neural network-based memory network, a recurrent neural network (RNN), a dilated convolutional neural network (Dilated CNN), and the like. In this case, the multimodal analysis model may have a different neural network structure for processing sequence data according to the total sequence data length that is inversely proportional to the length of the time sequence or the length of the time unit interval.

This multi-modal analysis model takes the image document data from the data storage module 140, converts it into training data through a pre-processing process, and stores it in the training DB, and learns the classifier using the training data of the training DB.

In the case of Korean data, the preprocessing process removes foreign words, specific special characters, and stopwords such as e-mail, or replaces them with other word tokens, and converts them to lowercase letters in the case of English data.

The multi-modal analysis model builds a word dictionary DB for all words extracted through morpheme analysis, stem extraction, and normalization processes for learning data, and topic words appearing more than a certain frequency in the word dictionary DB are used for analysis. . At this time, words appearing less than a certain frequency in the word dictionary DB are highly probable to be proper nouns, so they are treated as special tokens (Unknown tokens). If the multimodal analysis model uses character-based learning data, a preprocessing process of separating Hangul into initial, middle, and final consonants is performed, eliminating the need to process proper nouns as special tokens.

The multi-modal analysis model uses a word dictionary DB to replace topic words with tokens and then performs a learning process for each image document data. The learning process uses deep learning-based natural language processing algorithms such as doc2vec and BERT (Bidirectional Encoder Representations from Transformers).

doc2vec's PV-DBOW (Distributed Bag of Words version of Paragraph Vector) and PV-DB (Distributed Memory of Paragraph Vector) algorithms based on the word2vec algorithm train to predict which words appear in a specific document, cognitive learning.

Among the training courses of BERT, MLM (Masked Language Model) is one of the unsupervised learning methods using a large amount of natural language corpus. A document (paragraph or document) vector can be connected by employing the PV-DB algorithm in the pre-trained Transformer Encoder. Fine-tuning the natural language processing model on the content data collected through unsupervised learning methods such as MLM.

9 is a diagram for explaining a content classification process of a multimodal analysis model according to an embodiment of the present invention, and FIG. 10 is an exemplary diagram for explaining the classification result of FIG. 9 .

The multimodal analysis model defines the similarity of feature vectors extracted for each content through the learning process based on the distance or probability, and classifies similar contents among the similar contents through clustering of the entire image document data based on the similarity. (S110).

Here, clustering is a data mining technique that extracts features of corresponding content for similar content search and user response related to the found content.

In addition, the multimodal analysis model performs a feedback process to confirm the clustering result, and performs reorganization of the cluster according to the feedback ( S120 ). That is, after reclassifying the cluster based on the distance based on the redefined center value to recover the initial erroneous clustering, if the boundary between the clusters is not changed, the clustering algorithm is terminated.

As shown in FIG. 10 , when the reorganization is completed, the final content classification is completed through structural labeling in which the labels are reorganized step by step by observing the cluster labels, and a classifier is generated based on the classification result ( S130 ). The generated classifier determines which category the new content belongs to by using the feature vector extracted based on natural language processing for the new content. That is, the classifier allocates new content to the corresponding category using the feature vector.

For example, the multimodal analysis model can construct a two-stage label structure through structural labeling when labels such as teenage mukbang, 20s mukbang, 30s mukbang, Chinese food mukbang, and Korean food mukbang exist in the cluster. can That is, after setting the mukbang as the top label, each detailed mukbang label is configured as the sub-label.

Since the image document data subject to content classification is unstructured data written in natural language, it is necessary to express it as structured data to process it. Therefore, the image document data is expressed as a feature vector constructed by considering the importance according to the stop word and frequency for all words in the word dictionary DB, and the feature in the feature vector is composed of the word and the value of each feature. The feature values are frequency, presence or absence, and weight.

In this way, content classification through clustering can find a meaningful data structure without prior information on given image document data, takes a short calculation time, and can be applied to a large amount of content.

11 is an exemplary diagram illustrating a concentration algorithm of a multimodal analysis model according to an embodiment of the present invention, and FIG. 12 is an exemplary diagram illustrating a concentration algorithm using the concentration weight of FIG. 11 .

The multimodal analysis model can extract feature vectors by applying a concentration algorithm for each mode data of audio, motion, and frame on the content timeline, considering which frame of data is highly correlated with other frames. By performing a clustering algorithm using the extracted feature vector, it is possible to classify the content.

In the case of video mode data, the number of frames is reduced to about 1 FPS through a preprocessing process, and the number of frames is downsampled. For the video mode data that has been pre-processed in this way, a feature vector of each frame is extracted using a multimodal analysis model based on natural language processing.

In the case of audio mode data, the pre-trained multimodal analysis model extracts a feature vector for each frame and then compresses it on the timeline to extract the final feature vector.

In the case of subtitle mode data, a feature vector of each frame is extracted using a multimodal analysis model based on natural language processing.

The focused algorithm learns a parameter that determines the relationship between one feature vector and another for feature vectors for each mode data.

As shown in FIG. 12 , when the pronoun 'it' is analyzed using the concentration weight through the concentration algorithm, the analysis is performed by focusing on the word 'animal' indicated by it. That is, although the word it refers to in the upper and lower sentences is different, it can be seen that among the concentration algorithms, 'animal' and 'street', which are marked in dark blue color, are appropriately given more weight through the self-concentration algorithm.

In this way, the multimodal content analysis model may consider the correlation for each mode data within the time stamp or the correlation between data at different time stamps within the entire timeline through a concentrated algorithm.

13 is an exemplary diagram for explaining an analysis result of a multimodal analysis model according to an embodiment of the present invention.

As shown in FIG. 13 , the multi-modal analysis model analyzes new content and provides information combined with a viewer's reaction to the content as an analysis result. By designating the content metadata (title, description, category, etc.) as label data, the multimodal analysis model can be trained in a way that matches the metadata for each content and can analyze the content.

For example, if the category of the new content (viedo 1) is a moving story in a storytelling format, the viewer's reaction to the corresponding category may be 'impressed' or 'compassionate'.

Therefore, users or content sharing platform operators can check similar content and viewer reaction at once by labeling a specific category without having to look at numerous content one by one.

The multi-modal content analysis method according to the embodiment of the present invention described above may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Such recording media includes computer-readable media, and computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Computer readable media also includes computer storage media, which include volatile and nonvolatile embodied in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. , both removable and non-removable media.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: multimodal content analysis system
110: communication module 120: memory
130: processor 140: data storage module
150: display module

Claims (13)

A multi-modal content analysis method performed by a multi-modal content analysis system that analyzes content uploaded through a media channel, the method comprising:
a) collecting content including audio, video, subtitles and metadata from at least one content sharing platform that provides a content sharing service through the media channel;
b) Content text information including context for each content through natural language processing-based preprocessing of the collected contents and user response text information including viewer's posting response, and video document data written in natural language to each content generating each star;
c) extracting a feature vector for each frame of the image document data using a multimodal analysis model based on text analysis, and learning a classifier for content classification using the extracted feature vector; and
d) when new content is found through content search for the media channel, providing an analysis result for the new content using the learned multi-modal analysis model;
The multimodal analysis model is,
Based on the similarity between the feature vectors, a classifier generated through a clustering algorithm that classifies the entire image document data into K groups is learned, and the learned classifier uses a feature vector extracted from new content to obtain a predefined category ( automatically classified into categories),
The image document data is divided into mode data including audio, motion, and frame, and an attention algorithm is applied to each mode data to extract a feature vector considering the correlation between frames,
The concentrated algorithm is a multimodal content analysis method, in which the concentrated weight is applied to another feature vector associated with one feature vector for intensive analysis. The method of claim 1,
Step a) stores log information about the collected content,
In step d), the multimodal content analysis method is to search for new content after excluding the content that has already been searched by using the stored log information. The method of claim 1,
The content text information is
at least one of video data, audio data, and subtitle data of the corresponding content;
The user response text information,
At least one of the content's title, description, tags, views, impressions, clicks to impressions, positive/negative ratings, upload time, thumbnails, comments, metadata including viewer retention, watch time, or viewer demographic information. The method comprising the above, multimodal content analysis method. The method of claim 1,
Step a) is,
Storing data having high-level characteristics including text information, time information, and numerical information among the collected contents in a relational database,
The multi-modal content analysis method of storing data having low-level characteristics including image data, video data, and audio data of the collected contents in a storage. 5. The method of claim 4,
When at least one data write request occurs in the relational database or storage at the same time, the data by the data write request is stored using the work queue, and then data is retrieved and stored from the work queue in a first-in-first-out (FIFO) method. That is, a multimodal content analysis method. The method of claim 1,
Step c) is,
At least one word is extracted from the image document data, topic words are extracted through natural language processing analysis including stopword processing, stem extraction, and normalization for the extracted words, and then a word dictionary database is created using the topic words. To provide, a multi-modal content analysis method. 7. The method of claim 6,
The multi-modal analysis model performs learning on each image document data by replacing topic words with a predetermined frequency or more among topic words stored in the word dictionary database with tokens,
A multimodal content analysis method that treats topic words less than a preset frequency as a special token (Unknown token). delete The method of claim 1,
The concentration algorithm selects either a method of extracting a feature vector considering the correlation between mode data in the time stamp of the content or a method of extracting a feature vector considering the correlation between mode data of different time stamps within the entire timeline. A multimodal content analysis method to use. The method of claim 1,
The clustering algorithm is
defining the similarity of the extracted feature vectors for each content based on a distance or a probability;
forming K clusters by classifying similar contents among similar contents through clustering of the entire image document data based on the defined similarity;
performing reorganization by reclassifying K clusters based on the distance or probability;
Completing the final content classification through structural labeling of reorganizing cluster labels step by step when the boundary between clusters is not changed through the reorganization. The method of claim 1,
The multimodal analysis model is,
Classifying categories for content content using the content text information,
By using the viewer response text information for the contents of each category, high-ranking words are set as viewer responses according to the frequency,
A multimodal content analysis method that provides an analysis result in which the category and viewer reaction are combined for new content. The method of claim 1,
Step c) is,
Using multiple learning algorithms based on natural language processing models including Embeddings from Language Models (ELMO), Bidirectional Encoder Representations from Transformers (BERT), Big Bird, Generative Pre-Training (GPT), and Masked Language Model (MLM). to learn the feature vector, a multimodal content analysis method. In a multi-modal content analysis system for analyzing content uploaded through a media channel,
a memory in which a program for performing a multimodal content analysis method is recorded; and
a processor for executing the program;
The processor, by executing the program,
Collecting content including audio, video, subtitles and metadata from at least one content sharing platform that provides a content sharing service through the media channel;
Through natural language processing-based pre-processing of the collected contents, content text information including context for each content and user response text information including viewer's posting reaction are included, and video document data written in natural language is processed for each content, respectively. create,
Extracting a feature vector for each frame of the image document data using a multimodal analysis model based on text analysis, learning a classifier for content classification using the extracted feature vector,
When new content is found through content search for the media channel, an analysis result for the new content is provided using the learned multi-modal analysis model,
The multimodal analysis model is,
Based on the similarity between the feature vectors, a classifier generated through a clustering algorithm that classifies the entire image document data into K groups is learned, and the learned classifier uses a feature vector extracted from new content to obtain a predefined category ( automatically classified into categories),
The image document data is divided into mode data including audio, motion, and frame, and an attention algorithm is applied to each mode data to extract a feature vector considering the correlation between frames,
The concentrated algorithm is a multimodal content analysis method, in which the concentrated weight is applied to another feature vector associated with one feature vector for intensive analysis.

Images