KR20230163045A - A method and recording medium for providing video content production service using resource conversion matching of multimedia collected in metaverse environment - Google Patents

Abstract

A method of operating a service providing device connected to a metaverse system according to an embodiment of the present invention includes the steps of building a resource database using metaverse environment data of the metaverse system; In response to a first event conducted on the metaverse system, collecting target data for configuring multimedia converted content of the first event; extracting element information corresponding to the target setting information from the target data, based on metaverse environment-based target setting information corresponding to the target data; Obtaining multimedia converted content by performing multimedia content synthesis and conversion processing based on the element information and image resource matching of the resource database; and outputting the multimedia converted content.

Description

{A method and recording medium for providing video content production service using resource conversion matching of multimedia collected in metaverse environment}

The present invention relates to a method and recording medium for providing video content production services. More specifically, the present invention relates to a method of providing a video content production service using resource conversion matching of multimedia collected in a metaverse environment and a recording medium on which a program for executing the method on a computer is recorded.

With the recent development of video and content production technology, a variety of personalized multimedia content is being produced and distributed and consumed through social networking services. The number of creators and users of the multimedia content sharing service represented by YouTube is increasing exponentially, and various specialized tools for video content production are also being developed in terms of user convenience.

However, the use of these specialized tools is still not convenient. In order to produce high-quality multimedia content, the time and technical manpower of experts who are good at using specialized tools must be invested, and it is difficult to transport materials for production and the resulting production costs are incurred.

In particular, this problem appears more significantly in the work of composing multimedia content to record various activities in the currently emerging Metaverse environment.

Here, Metaverse is a compound word of 'meta', meaning virtuality and transcendence, and 'universe', meaning world and universe, and means a three-dimensional virtual world linked to reality. This metaverse is a representative service that provides augmented reality or mixed reality to users. It is an advanced concept than the existing term virtual reality environment, and is an augmented reality where the virtual world such as the web and the Internet is absorbed into the real world. Provides a realistic environment.

Recently, as the global COVID-19 pandemic has prolonged, contact and movement between people has been restricted, and 5G communication networks that can transmit large amounts of information at high speed have been expanded, creating various forms of virtual worlds that support lifestyle or game-type virtual worlds. Metaverse services are emerging.

Accordingly, various activities such as meetings, lectures, and performances are taking place in the metaverse environment. However, in order to record these activities and produce them as video content, the collection of related data within the metaverse, the use of specialized tools, and the cost of expert technical manpower and time are inevitably required.

The present invention was conceived to solve the problems described above, and provides an authoring interface using automated text recognition-based resource matching so that activity records in a metaverse environment can be produced as multimedia video content, and based on this, The purpose is to provide a video content production service method and recording medium using resource conversion matching of multimedia collected in the metaverse environment, which allows easy, easy, and quick production of activities in the metaverse environment into multimedia video content. .

A method according to an embodiment of the present invention to solve the above-described problem is a method of operating a service providing device connected to a metaverse system, which includes building a resource database using metaverse environment data of the metaverse system. step; In response to a first event conducted on the metaverse system, collecting target data for configuring multimedia converted content of the first event; extracting element information corresponding to the target setting information from the target data, based on metaverse environment-based target setting information corresponding to the target data; Obtaining multimedia converted content by performing multimedia content synthesis and conversion processing based on the element information and image resource matching of the resource database; and outputting the multimedia converted content.

In addition, a device according to an embodiment of the present invention for solving the problems described above is a database that builds a resource database using metaverse environment data of the metaverse system in a service providing device connected to the metaverse system. wealth; a target data collection unit that collects target data for configuring multimedia conversion content of the first event in response to a first event conducted on the metaverse system; an element information extraction unit that extracts element information corresponding to the target setting information from the target data, based on metaverse environment-based target setting information corresponding to the target data; a content synthesis and conversion unit that performs multimedia content synthesis and conversion processing based on the element information and image resource matching of the resource database to obtain multimedia converted content; and an output unit that outputs the multimedia converted content.

According to another aspect, the present invention includes a recording medium recording a program to be executed on a computer to execute each of the above steps.

According to an embodiment of the present invention, a resource database can be built using the metaverse environment data of the metaverse system, and the target data is selected based on target setting information based on the metaverse environment of the corresponding target data. Element information corresponding to the setting information can be extracted, and multimedia content synthesis and conversion processing can be performed based on matching the element information and video resources in the resource database.

Accordingly, according to an embodiment of the present invention, a multimedia video content authoring service and related interface using resource matching based on automated element information recognition can be provided for activity records in various metaverse environments, which can be provided separately. A method of providing a video content production service using resource conversion matching of multimedia collected in a metaverse environment that allows activities in the metaverse environment to be easily, easily, and quickly produced as multimedia video content while minimizing manpower and time costs; and The device can be provided.

1 is a conceptual diagram showing the entire system according to an embodiment of the present invention.
Figure 2 is a block diagram illustrating in more detail a service providing device according to an embodiment of the present invention.
Figure 3 is a block diagram illustrating in more detail a target setting unit according to an embodiment of the present invention.
Figure 4 is a flowchart for explaining the operation of a service providing device according to an embodiment of the present invention.
Figure 5 is an example diagram for explaining the element information matching process according to an embodiment of the present invention.
Figure 6 is an exemplary diagram of generating various multimedia video contents for each event according to an embodiment of the present invention.
7 to 8 are diagrams to explain the resource database in more detail according to an embodiment of the present invention.
Figure 9 is a diagram for explaining in more detail the production interface according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art will be able to invent various devices that embody the principles of the present invention and are included in the spirit and scope of the present invention, although not explicitly described or shown herein. In addition, it is understood that all conditional terms and embodiments listed herein are, in principle, expressly intended only for the purpose of ensuring that the concept of the invention is understood, and are not limited to the embodiments and conditions specifically listed as such. It has to be.

Additionally, it is to be understood that any detailed description reciting principles, aspects, and embodiments of the invention, as well as specific embodiments, is intended to encompass structural and functional equivalents thereof. In addition, these equivalents should be understood to include not only currently known equivalents but also equivalents developed in the future, that is, all elements invented to perform the same function regardless of structure.

Accordingly, for example, the block diagrams herein should be understood as representing a conceptual view of an example circuit embodying the principles of the invention. Similarly, all flow diagrams, state transition diagrams, pseudo-code, etc. are understood to represent various processes that can be substantially represented on a computer-readable medium and are performed by a computer or processor, whether or not the computer or processor is explicitly shown. It has to be.

Additionally, the clear use of terms such as processor, control, or similar concepts should not be construed as exclusively referring to hardware capable of executing software, and should not be construed as referring exclusively to hardware capable of executing software, including without limitation digital signal processor (DSP) hardware and ROM for storing software. It should be understood as implicitly including ROM, RAM, and non-volatile memory. Other hardware for public use may also be included.

The above-described purpose, features and advantages will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art will be able to easily implement the technical idea of the present invention. There will be. Additionally, in describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

1 is a diagram schematically showing the entire system according to an embodiment of the present invention.

Referring to FIG. 1, a system according to an embodiment of the present invention includes a video content production service providing device 100, a user terminal 200, a multimedia content server 300, and a metaverse system 400.

First, the user terminal 200, the metaverse system 400, the service providing device 100, and the multimedia content server 300 may be connected wired or wirelessly through a network, and each user terminal ( 200), the service provision device 100 and the multimedia content server 300 are Internet networks, LANs, WANs, PSTN (Public Switched Telephone Network), PSDN (Public Switched Data Network), cable TV networks, WIFI, mobile communication networks and others. Data can be transmitted and received through a wireless communication network, etc. The user terminal 200, the service providing device 100, and the multimedia content server 300 may include respective communication modules for communicating using a protocol corresponding to each communication network.

In addition, the user terminal 200 described in this specification refers to a mobile phone, smart phone, laptop computer, digital broadcasting portable terminal, PDA (Personal Digital Assistants), and PMP (Portable Multimedia Player). , VR (Virtual Reality) environment provision devices, HMD (Head Mount Display) devices, etc. may be included, but the present invention is not limited thereto and may include various devices capable of user input and information display according to metaverse system access. You can.

The user terminal 200 according to an embodiment of the present invention may be a device for accessing the metaverse system 400. For example, the user terminal 200 may be a typical display device such as the above-described mobile phone, smart phone, or laptop computer, and has the form of a Head Mounted Display (HMD), and can display the user's information by the shape of glasses or a frame, etc. It is temporarily fixed to the head and may include a near-to-eye display that displays an image within a few centimeters of the user's eyes. Additionally, the user terminal 200 may be implemented as a wearable computing display system.

In addition, the user terminal 200 can access the metaverse system 400 according to an embodiment of the present invention through the display, and the metaverse system 400 is a metaverse system linked to reality through the metaverse interface. A virtual space based on 3D reality can be provided, and various activities within the 3D reality-linked virtual space can be experienced through manipulation of the user terminal 200.

In addition, the service providing device 100 according to an embodiment of the present invention may be a device capable of simultaneous access to the metaverse system 400 to which the user terminal 200 is connected, and the metaverse environment of the metaverse system 400 Build a resource database using data, collect target data for configuring multimedia conversion content of the first event in response to a first event that occurred on the metaverse system 400, and metaverse corresponding to the target data. Based on bus environment-based target setting information, element information corresponding to the target setting information is extracted from the target data, and multimedia content synthesis and conversion processing is performed based on matching the element information and video resources of the resource database, It may be configured as a service device that acquires multimedia converted content and outputs the multimedia converted content.

Here, the converted multimedia content can be output to the multimedia content server 300 and distributed to one or more service user terminals.

In addition, the multimedia content server 300 can classify and manage the multimedia converted content in response to the first event generated in the metaverse system 400, and block the multimedia converted content according to the request of the service user terminal. Chain-based shared services can be provided.

To this end, the video content production service providing device 100 may perform a process of sharing the converted multimedia content on a blockchain network where the block multimedia content server 300 is located.

Furthermore, the service providing device 100 according to an embodiment of the present invention may further include a separate metaverse environment data processing module 105, which will be described later in FIG. 2, and the metaverse environment data processing module 105 , It can perform the function of collecting and databaseizing various conversation data and environmental data generated according to the activities of users of the user terminal 200 on the metaverse system 400.

In addition, when target setting information is configured according to a request from the user terminal 200, the service providing device 100 can obtain target data based on the target setting information, and the target setting information is zone information. , may include language information, speaker information, and feedback information.

Here, element information can be extracted from the target data and various data model classifications for matching video resources can be determined according to target setting information.

For example, when target setting information includes zone information, the zone information may include time section information and location information for specifying a spatiotemporal zone within the metaverse in which multimedia conversion content will be composed. Each location information may correspond to a conference hall, performance hall, lecture hall, etc. formed within the metaverse system 400, and time section information may correspond to a meeting time, performance time, lecture time, etc.

Additionally, for example, when the target setting information includes language information, the zone information may include language information for specifying the conversion language of conversation data that will constitute multimedia conversion content.

Language information may be determined according to target setting information specified according to input from the user terminal 200. Here, language information can be set to various target languages for text recognition of conversation data. The service providing device 100 can process video content resource matching for each target language, and even for target data input in one language, multimedia conversion content in a metaverse environment appropriate for the culture of various countries is provided. can be created. This has the effect of enabling users who are not only non-experts in video editing but also have no knowledge of the language or culture of a specific country to easily and easily produce global multimedia conversion content tailored to the language or culture of each country.

Meanwhile, the target setting information may include speaker setting information. Here, speaker setting information can be used to determine main conversation data for extracting element information of multimedia converted content. This speaker setting information may be set differently depending on each event, and may be input from the user terminal 200 or automatically determined by a preset for each event. For example, the user may be set as a lecturer in a lecture event, a performer in a performance event, or a presenter in a conference event. Accordingly, the service providing device 100 can extract important conversation data that matches element information among conversation data by classifying them according to speaker setting information.

Additionally, the target setting information may further include feedback setting information. Feedback setting information may determine whether to include event-related feedback information of other users on the metaverse system 400 corresponding to each event in the target data. Here, the feedback information may be, for example, feedback chat data from spectators or viewers at a lecture, performance, or conference event rather than the speaker, and such feedback information can be added to multimedia converted content according to user settings and production interface configuration. there is.

In this way, when the target data is configured, the service providing device 100 extracts key element information according to the pattern and statistical similarity of the conversion target data on a text basis according to a preset natural language processing algorithm in extracting element information. Resource matching processing can be performed to optimize video, image, text, subtitles, animation, font (color, size, font), and audio for each frame merge layer using extracted text-based element information. It is possible to provide a production interface using processed element information and generate optimized multimedia conversion content based on the frame merge layer according to user input.

Furthermore, the service providing device 100 can disseminate multimedia converted content through a blockchain network and provide data for providing sharing services using the multimedia content server 300, and various performance videos and lectures accordingly. Additional services such as content trading and copyright sharing, such as videos, will become possible.

FIG. 2 is a block diagram illustrating in more detail a service providing device according to an embodiment of the present invention, and FIG. 3 is a block diagram illustrating a target setting unit in more detail according to an embodiment of the present invention.

Referring to Figures 2 and 3, the service providing device 100 according to an embodiment of the present invention includes a metaverse environment data processing module 105, a target data collection unit 110, a target setting unit 117, and an element. Information extraction unit 120, image resource matching unit 130, production interface providing unit 140, content synthesis conversion unit 150, learning database 160, resource database 180, output unit 170, and block Includes a chain service unit 180.

First, the metaverse environment data processing module 105 connects to the metaverse system 400 and processes various environmental data necessary for configuring and learning the image resource database, creating each learning database 160 and the resource database 180. ) can be forwarded and processed to the database unit containing.

Additionally, the metaverse environment data processing module 105 may configure necessary metaverse environment data according to a request from the target data collection unit 110 and provide it to the target data collection unit 110.

To this end, the metaverse environment data processing module 105 may be equipped with a metaverse environment data crawler for accessing and collecting data on the metaverse system 400, and the metaverse environment data collected here is metaverse environment data. Event information generated in the bus system 400, and 3D geographic data, 3D background data, 3D object data, texture data, and animation that constitute the metaverse environment for each zone or region collected in response to each event information. Various environmental information, including data, voice data, chat data, and time-series activity data of each user, can be collected and pre-processed and transmitted to the target data collection unit 110, learning database 160, and resource database 180. .

Then, the target data collection unit 110 receives target data for multimedia content conversion from the metaverse environment data processing module 105 and delivers it to the element information extraction unit 120.

Here, the target data collection unit 110 may include one or more input interfaces for receiving target data information from the user terminal 200.

For example, the target data may be determined according to event information in the metaverse requested from the user terminal 200, and may include various event information such as performances, lectures, and conferences. Accordingly, the target data may include each metaverse environment data for a performance, lecture, or conference designated by the user terminal 200.

And, the target data collection unit 110 filters the metaverse environment data received from the metaverse environment data processing module 105 according to the target setting information set in the target setting unit 117, and selects the filtered target data. It can be provided to the element information extraction unit 120.

Here, the target setting information is information used to determine data to be extracted as element information from the target data, and can be set according to the input of the user terminal 200 or according to a preset for each event. there is. The set target setting information is transmitted to the element information extraction unit 120, and the element information extraction unit 120 selects an element information extraction algorithm and extraction model according to the target setting information, and collects it from the target data collection unit 110. You can extract element information from the data.

More specifically, referring to FIG. 3, the target setting unit 117 includes a zone setting unit 1171, a language setting unit 1173, a speaker setting unit 1175, and a feedback setting unit 1177.

First, the zone setting unit 1171 sets zone information to identify first zone information on the metaverse system in order to extract element information for multimedia content conversion of the target data.

When zone information is set, the target data collection unit 110 collects conversation data generated from the first zone information on the metaverse system in response to the first zone information, and according to text recognition of the conversation data. , can be converted into a text sentence, and the converted text sentence can be obtained as the target data and transmitted to the element information extraction unit 120.

Here, the first zone information includes time section information corresponding to the first event on the metaverse system, area information within the metaverse system corresponding to the first event, and coordinate information where conversation data was generated. can do.

More specifically, the zone information may be set differently depending on the zone and event in which each user is active on the metaverse system 400. And the zone information can be separately organized according to various criteria.

For example, in a performance, etc., each user may be located in an area that is separated from each other based on language. Participants who participate as spectators in the first Metaverse performance can set their language before entering and can be assigned to a specific area that speaks only their language. And, when the user terminal 200 requests a multimedia content conversion service of the first metaverse performance to the service providing device 100, the target data collection unit 110 includes the specific zone information in the target setting information Element information can be extracted from conversation data limited to the specific area.

Meanwhile, the language setting unit 1173 stores and manages language setting information for setting the conversion target language in order to extract element information for converting multimedia content of the target data.

Accordingly, the target data collection unit 110 identifies one or more target language information corresponding to the target data, based on the target setting information, and selects one or more conversation data in the target data based on the target language information. It can be converted into text sentences for each language and transmitted to the element information extraction unit 120.

For example, in cases where communication with fans of various races is required, such as a metaverse fan meeting, conversation data of different nationalities and cultures for each conversationalist according to an embodiment of the present invention constitutes the target data. This can be done, and the service providing device 100 can configure global multimedia content based on the conversation data and distribute it to users in other countries.

To this end, the language setting unit 1173 according to an embodiment of the present invention can set speech language information and target language information to be converted, respectively. That is, speech language information may also be set differently for each speaker, and target language information, which is the language to be converted from this, may also be determined in advance depending on the user terminal 200 and basic preset settings.

Meanwhile, the speaker setting unit 1175 stores and manages speaker setting information for setting a speaker in order to extract element information for converting multimedia content of the target data.

Then, the target data collection unit 110 identifies speaker information corresponding to the target data based on the target setting information, and converts the conversation data into text sentences for one or more speakers based on the speaker information. , can be transmitted to the element information extraction unit 120.

More specifically, conversation data collected from various events in the metaverse system 400, such as lectures, performances, fan meetings, and conferences, may have each main speaker, and even if they are not the main speaker, users who want to compose multimedia conversion content There may be major speakers that you would like to specify separately. Accordingly, the speaker setting unit 1175 according to an embodiment of the present invention can specify speaker information in advance according to input from the user terminal 200 or preset settings. Here, the speaker information may be, for example, ID information for each user who generated each conversation data on the metaverse system 400.

Additionally, each speaker may be set to be fixed during the event period, or may be set to change flexibly. For example, in a speech or presentation, the speaker must be fixed, but in a conference room or daily conversation, the speaker can be separated.

Additionally, the talker setting unit 1175 may identify talker information and priority information from environmental information on the metaverse system 400 and configure them as talker setting information. In this case, the target data collection unit 110 may separate and configure the speaker's data from which element information is to be extracted in advance according to priority and transmit it to the element information extraction unit 120.

For example, depending on an event in the metaverse system 400, it may be necessary to fix the presenter or pass the speaking rights to the next presenter, and may be a user designated as a key player (representative presenter) or in a specific area. Only the located user can speak as the speaker. When such environmental information is confirmed in advance, the speaker setting unit 1175 separates the text converted information or text input information from the user information set as the key player or the voice information of the user located in the specific area, and extracts element information. It can be transmitted to Boo (120).

Meanwhile, the feedback setting unit 1177 may set feedback data corresponding to the target data as target information to be extracted as element information.

Accordingly, the target data collection unit 110 may collect feedback data corresponding to the target data, convert the collected feedback data into one or more feedback text sentences, and transmit them to the element information extraction unit 120.

For example, the feedback data may include comment data, emoticon data, etc. collected through a separate channel in response to target data according to an embodiment of the present invention and multimedia converted content converted in real time from the target data, and the present invention. The feedback setting unit 1177 according to the embodiment may configure setting information for merging and processing each of these feedback data into the multimedia conversion content.

Meanwhile, the target setting unit 117 stores and manages the overall setting information of each of the above-described zone setting units 1171, language setting units 1173, speaker setting units 1175, and feedback setting units 1177. Depending on the event preset, the priority and application of each setting information can be specified in advance. For example, depending on the event in each metaverse environment, when multinational languages are required, when the presenter must be fixed, when the presenter is flexible, etc., the above-mentioned event presets can be determined and responded accordingly, reflecting various event properties. Target setting information can be configured in advance.

Additionally, the element information extraction unit 120 may extract element information for classifying the image resource into one or more element data that can be matched from the target data transmitted from the target data collection unit 110.

Here, a preset natural language processing algorithm can be used to extract element information. Accordingly, the element information extraction unit 120 extracts element data in text format from the target data filtered by the target setting unit 117 and transmits the extracted element information to the image resource matching unit 130.

In particular, the element information extraction unit 120 may determine a natural language processing process for the target data to match it to an image resource, based on the main element data and target setting information of the target data. Here, the natural language processing process may be an example of a text summary process pre-learned by a deep learning process.

Accordingly, the element information extraction unit 120 may perform a text summary process to extract important sentences or words from the target data, synthesize one or more summary sentences, and output them as element information.

Additionally, when determining the text summary process, the element information extraction unit 120 may apply one or more different language models according to the target setting information of the target data. The language model may be an extraction model or a synthesis model, and different models may be determined depending on the characteristics and types of each metaverse event in the metaverse system 400 environment.

For example, when fixed speaker information according to a speech event of a specific representative is identified in the target setting information set corresponding to the target data, the element information extraction unit 120 applies the extraction model to the target data to determine the representative's Important sentence information from the utterance content can be extracted as element information.

In addition, when the multilingual language setting and variable speaker information according to the fan meeting are identified in the target setting information set corresponding to the target data, the element information extraction unit 120 selects the country and speaker-specific language corresponding to the change in speaker and language. After changing and applying the conversion model, a synthesis model that converts and synthesizes the target data can be applied to select important keyword information for each speaker and extract the synthesized sentence information as a summary sentence as element information.

Accordingly, element information according to an embodiment of the present invention may include one or more important sentence information extracted from target data or obtained based on a synthetic language model. Such sentence information can correspond to the layer unit of one video resource matching frame, and appropriate resource matching is processed for each sentence information to form one video frame layer unit.

And, the video resource matching unit 130 performs optimized resource matching processing in response to the element information based on the learning database 160 and the resource database 180, and converts the resource matching information into the content synthesis conversion unit ( 150) and the production interface providing unit 140.

More specifically, the image resource matching unit 130 performs resource matching processing for content composition conversion corresponding to element information. Resources for content composition conversion include a background image processed within a preset frame layer unit, It may include various contents such as background images, background music, layout, motion, animation, etc., which may be collected and configured from metaverse environment data transmitted from the metaverse environment data processing module 105.

As described above, metaverse environmental data includes event information generated in the metaverse system 400, three-dimensional geographic data constituting the metaverse environment for each zone or region collected in response to each event information, and three-dimensional background. It is pre-processed data of various environmental information including data, 3D object data, texture data, animation data, voice data, chat data, and time-series activity data of each user, and may be pre-stored in the resource database 180. there is.

Additionally, the resource database 180 may store and manage resource content data received from various content servers connected through an external network. Here, the resource content data may include at least one of content attribute information, content identification information, content link information, and content data information, and the matched resource information is sent to the production interface providing unit 140 or the content synthesis conversion unit 150. It can be delivered.

Furthermore, the resource database 180 may include a plurality of resource databases 180 classified by country and language. In this case, resource content data suitable for the country and language corresponding to the target language set during content synthesis conversion may be extracted and matched in the video resource matching unit 130.

Additionally, the video resource matching unit 130 may build and utilize the learning database 160 to match more appropriate content corresponding to element information from the resource database 180. The learning database 160 can build a relationship learning model that learns relationship information between resource content and element information, and in particular, a weight variable that allows more appropriate resource content to be matched in response to the type of target data and key element information. can be set. Accordingly, the video resource matching unit 130 can utilize the learning database 160 to calculate matching information that matches the optimal resource content corresponding to the element information, and the calculated matching information is provided by the production interface providing unit ( 140) and may be transmitted to the content synthesis conversion unit 150.

For example, the video resource matching unit 130 stores the background, sound, character type, target language, etc. for each video frame layer divided by a certain time unit in the learning database 160, in response to the sentence information of the element information. Based on this, it can be matched with a pre-built resource database 180.

The learning database 160 can define a major category and a detailed classification category of each sentence information, and analyzes the correlation between the deep learning learning results of the major category and the detailed classification, so that the matched background, sound, or letter type is the target data. It is possible to arithmetically analyze how probabilistically related the purpose of a metaverse event is.

Accordingly, the video resource matching unit 130 can obtain resource contents such as background, sound, text type, and target language for which the most optimized correlation is calculated as matching information for matching the video frame layer unit.

In addition, for example, the image resource matching unit 130 according to an embodiment of the present invention may directly generate image or audio resource content depicting a sentence of element information or search for it in the resource database 180, The generated or searched resource content may be transmitted to the production interface providing unit 140 and the content synthesis conversion unit 150.

Then, the production interface providing unit 140 configures a production interface capable of synthesizing and converting the content matched by the video resource matching unit 130 based on the matching information, and provides the production interface to the user terminal 200.

The production interface provider 140 transmits the resource content data and resource matching information to an interface application running on the user terminal 200, or transmits the resource content data and resource matching information to the user terminal 200 through a separate API. ), or a real-time web production interface based on the resource content data and resource matching information may be configured and provided to the user terminal 200.

Accordingly, the user terminal 200 can extract element information in response to the target data entered by the user and check the resource content to which the image resource is matched, process appropriate editing and processing, and input a composite conversion command. You can. Additionally, depending on the settings of the user terminal 200, a conversion request may be input directly to the content synthesis conversion unit 150 without separate editing or processing in the user terminal 200.

The content synthesis conversion unit 150 synthesizes and converts the target data into multimedia conversion content based on resource content data, resource matching information, and input information from the user terminal 200.

Accordingly, the multimedia converted content may include multimedia data in which at least one of video, sound, image, animation, subtitle, and font has been synthesized and converted corresponding to the target data. The synthesized and converted multimedia content may be provided to the production interface providing unit 140 and may be transmitted to the output unit 170 according to the confirmation or upload input of the production interface providing unit 140.

The output unit 170 may output the finally determined multimedia converted content as converted content of target data, which may be provided to the multimedia content server 300 and used for various information provision services based on target data. It may also be shared with one or more other user terminals through a social network service.

For example, the information provision service may include a multimedia content conversion service corresponding to various events in the metaverse, and a multimedia content streaming service based on this may be an example, and these services are processed in real time and are processed in the metaverse system. It can also be implemented as a real-time video content production service delivered within (400).

Meanwhile, the block chain service unit 180 may perform a process of sharing multimedia converted content produced according to the multimedia content conversion service on the blockchain network where the block multimedia content server 300 is located.

Accordingly, the blockchain service unit 180 can share and distribute multimedia converted content, store and manage transaction information, etc., and provide cryptocurrency services and decentralized services for use and copyright payment processing for multimedia converted content. Centralized application linkage services, etc. can be processed.

Figure 4 is a flowchart for explaining the operation of a service providing device according to an embodiment of the present invention.

Referring to FIG. 4, the service providing device 100 according to an embodiment of the present invention acquires metaverse environment data corresponding to a metaverse event (S101) and performs preprocessing corresponding thereto.

Then, the service providing device 100 collects target data for processing multimedia video content from the metaverse environment data (S103).

Thereafter, the service providing device 100 identifies preset target setting information corresponding to the target data (S105).

Then, the service providing device 100 extracts element information to be converted into text from the target data based on the target setting information (S107).

Thereafter, the service providing device 100 performs image resource matching based on metaverse environment data corresponding to element information (S109).

And, the service providing device 100 provides a production interface based on the matched video resource content (S111).

Thereafter, the service providing device 100 synthesizes and converts multimedia content based on user input to the production interface (S113).

And, the service providing device 100 can output the converted multimedia content and provide it through a blockchain network (S115).

Additionally, the service providing device 100 provides transaction record and copyright management services corresponding to multimedia content (S117).

Figure 5 is an exemplary diagram for explaining the element information matching process according to an embodiment of the present invention, and Figure 6 is an exemplary diagram of generating various multimedia video contents for each event according to an embodiment of the present invention.

First, referring to FIG. 5, the target data collection unit 110 may identify a voice composed of the first language from voice and chat data collected from metaverse environment data and convert it into text. Here, in the case of chat data, separate conversion may not be necessary.

In addition, the element information extraction unit 120 can extract sentence element information according to the settings in the target setting unit 117, and the video resource matching unit 130 can extract the zone, speaker, and feedback according to each target setting information. and resource matching that takes into account each country's language.

For example, in order to extract key sentences from target data input in the first language, the image resource matching unit 130 may process resources corresponding to various element information extracted from the element information extracting unit 120. You can.

For example, even for one element information, different resources may be matched according to language models for Korean, English, Japanese, Chinese, etc., and different resources may be matched according to each speaker's settings, and each speaker Depending on which zone you are in, another resource may be matched.

Then, when the target language and region are set, the video resource matching unit 130 selects the most appropriate resource content corresponding to the word keyword of each element information corresponding to the target language for each country and language based on the learning database 160. It can be obtained from the resource database 180 classified by category. For example, the University A entrance ceremony metaverse background video resource corresponding to the entrance ceremony keyword, the speakers' selfie video resource corresponding to the selfie keyword, and the text resource indicating the ID information of each speaker are matched, as shown at the bottom of Figure 5. Video multimedia content as described above may be configured.

In addition, the video resource matching unit 130 can match subtitles, fonts, and font resources corresponding to sentence information of element information, and match audio collected in the metaverse environment as a sound resource in response to the sentence information. You can.

Accordingly, the content composition conversion unit 150 matches and converts the video resources, subtitles, font and font resources, and sound resources according to the layout and animation information in response to the video frame layer unit of the preset time section. It becomes possible to create multimedia content.

For example, multimedia content related to one sentence output as a subtitle may be played in the video of the frame layer unit section, and the content synthesis conversion unit 150 may store subtitles, videos, and images in the video of the frame layer unit section. By arranging them together, they can be synthesized and converted so that the sound is output at a preset timing. The video resource matching unit 130 can match the appropriate content data combination, animation effect, and arrangement of the content synthesis conversion unit 150 through machine learning technology, deep learning technology, etc.

In addition, the content synthesis conversion unit 150 can match partial image contents for each text element, perform boundary processing between the matched partial images, and fuse them into one image, and create a set of image frames using the fused image. It can be built.

Furthermore, the content synthesis conversion unit 150 can refine text elements, set output timing for each text element, and match the image frame set to the output timing. Then, the content synthesis conversion unit 150 inserts synthesized subtitles synchronized with the output timing of each text element, merges TTS voices of the synthesized subtitles, or gives voice characteristics to the TTS voices according to user settings. Further processing can be performed. Accordingly, data collected in various metaverse environments can be fused, converted, and synthesized into appropriate multimedia content for each zone, speaker, target language, and feedback.

Figure 6 is a diagram illustrating multimedia video conversion contents generated according to various event-specific presets according to this example of the present invention.

As shown in FIG. 6, when a meeting event occurs in the metaverse, the user terminal 200 can request multimedia conversion content corresponding to the meeting event, and the service providing device 100 automatically converts the meeting recorded video content. It can be configured and distributed on a blockchain, or a resource matching-based production interface for configuring meeting recorded video content can be configured and provided to the user terminal 200.

In addition, as shown in FIG. 6, when a performance event occurs in the metaverse, the user terminal 200 may request multimedia conversion content corresponding to the performance event, and the service providing device 100 may request performance highlight content. It can be automatically configured and distributed on a blockchain, or a resource matching-based production interface for configuring performance highlight content can be configured and provided to the user terminal 200.

6 to 7 are diagrams for explaining a resource database according to an embodiment of the present invention.

Referring to FIG. 6, the resource database 180 according to an embodiment of the present invention includes an interface unit 185, a logical model management unit 181, a physical environment management unit 183, a metastore database 183, and a data storage unit. Includes (184).

According to an embodiment of the present invention, the resource database 180 classifies and labels media content data based on meta information based on metaverse environment data and loads it in a form that can be analyzed in the learning database 160. It facilitates sharing of resource content data.

To this end, the resource database 180 can remove duplicate data, correct missing data, and detect abnormal data through preprocessing of resource content data, perform a scaling process on the preprocessed data, and perform the well-known Long Short-Term Memory. Data classification processing can be performed to build a learning database 160 using algorithms such as models (LSTM).

More specifically, the interface unit 185 performs distributed input/output interface processing of resource content data classified and stored in each management unit 181 and 182.

The logical model management unit 181 can classify, store, and manage resource content through the metastore database 183. Here, the metastore database 183 can store and manage metadata for indexing big data-based content data in the data storage unit 184 physically stored in the physical environment management unit 182. For example, metadata may include at least one of classification information by user, classification information by function, or storage classification information, and each classification information may correspond to the storage structure of the physically distributed and stored data storage unit 184. .

For example, the data storage unit 184 is resource content and may store animation, background image, sound, font, layout information, etc.

Figure 7 is an example of a resource content format stored according to an embodiment of the present invention, including data type information such as video, sound, and image, identifier information, tag information, URL information, and virtual hosting URL information. there is.

The metastore database 183 can store and manage metadata as shown in Table 1 below as classification information.

Data classification meta information 1 meta information 2 meta information 3 type animated movie /store /data /animation background image /image sound /sound font (font) /log /realtime Layout Information /batch

As shown in Table 1, meta information can be classified by each classification information according to data classification, and therefore necessary resources can be indexed using meta information.

Accordingly, the resource database 180 according to an embodiment of the present invention can manage the data storage unit 184 of a big data structure that is physically distributed and stored, and stores the necessary resource content in the meta of the metastore database 183. Information can be used to index.

Therefore, the resource database 180 according to an embodiment of the present invention is not only for the purpose of storage, but can be constructed considering the aspect of loading stored data in an analyzable form and sharing necessary data in various analysis environments. Furthermore, it is possible to query data information based on SQL, thereby increasing the convenience and speed of accessing the data.

Figure 9 is a diagram for explaining in more detail the production interface according to an embodiment of the present invention.

Referring to Figure 9, the production interface according to an embodiment of the present invention may include a graphic user interface output through the user terminal 200, a target data input interface 201, a video editing interface 204, and , may include a subtitle editing interface 202, and a sound source and language editing interface 203.

In addition, the service providing device 100 according to an embodiment of the present invention can receive text data of a specific document through the target data input interface 201, and the input text data is divided into elements according to the summary button input, etc. It can be used for element information extraction processing in the information extraction unit 120.

In addition, recommended resource content according to the matching process of the video resource matching unit 130 based on the extracted element information is sent to each video editing interface 204, the subtitle editing interface 102, and the sound source and language editing interface 203. It may be suggested as a recommended item. Accordingly, the user terminal 200 can generate multimedia conversion content by selecting recommended resource content.

The user of the user terminal 200 can select resource content from each editing interface and input video conversion and SNS upload through the output interface 205, and accordingly, conversion processing in the content composition conversion unit 150. is performed, and the results are output to the user terminal 200 or uploaded to the multimedia content server 300, can be shared through a preset SNS account, and can be shared and traded through a blockchain service.

Meanwhile, the methods according to various embodiments of the present invention described above may be implemented as a program and stored in various non-transitory computer readable media and provided to each server or device. Accordingly, the user terminal 100 can connect to the server or device and download the program.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above may be stored and provided on non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

In addition, although preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the invention pertains without departing from the gist of the present invention as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

Claims (11)

In the method of operating a service providing device connected to the metaverse system,
Building a resource database using metaverse environment data of the metaverse system;
In response to a first event conducted on the metaverse system, collecting target data for configuring multimedia converted content of the first event;
extracting element information corresponding to the target setting information from the target data, based on metaverse environment-based target setting information corresponding to the target data;
Obtaining multimedia converted content by performing multimedia content synthesis and conversion processing based on the element information and image resource matching of the resource database; and
Comprising the step of outputting the multimedia converted content.
Method of operation of the service provision device. According to paragraph 1,
The step of collecting the target data is,
Identifying first zone information on the metaverse system based on the target setting information;
In response to the first zone information, collecting conversation data generated from the first zone information on the metaverse system; and
Converting the dialogue data into text sentences according to text recognition; and
Comprising the step of obtaining the converted text sentence as the target data.
Method of operation of the service provision device. According to paragraph 2,
The first zone information is,
Containing time section information corresponding to the first event on the metaverse system, area information within the metaverse system corresponding to the first event, and coordinate information at which conversation data was generated
Method of operation of the service provision device. According to paragraph 2,
The step of collecting the target data is,
Based on the target setting information, further comprising identifying one or more target language information corresponding to the target data,
The step of converting the text sentence is,
Converting the conversation data into text sentences for one or more languages based on the target language information.
Method of operation of the service provision device. According to paragraph 2,
The step of collecting the target data is,
Based on the target setting information, further comprising identifying speaker information corresponding to the target data,
The step of converting the text sentence is,
Converting the conversation data into text sentences for one or more speakers based on the speaker information.
Method of operation of the service provision device. According to paragraph 2,
The step of collecting the target data is,
Based on the target setting information, further comprising collecting feedback data corresponding to the target data,
The step of converting the text sentence is,
Converting the collected feedback data into one or more feedback text sentences.
Method of operation of the service provision device. According to paragraph 2,
The step of extracting the element information is,
Comprising the step of extracting one or more sentence information for matching video resources from the target data,
The step of extracting the sentence information is,
Comprising: performing a text summary process of the target data,
The text summary process is a process using different language models determined according to the target setting information of the target data,
The language model includes an extraction model or a synthesis model.
Method of operation of the service provision device. According to paragraph 2,
The resource database includes at least one of video, background, image, sound, and animation collected and classified from metaverse environmental data so that it can be matched to the element information.
Method of operation of the service provision device. According to paragraph 1,
Further comprising providing a metaverse video content production interface based on video resource matching corresponding to the element information,
The step of acquiring the multimedia converted content is,
Comprising the step of performing multimedia content synthesis and conversion processing according to a user input to the production interface, and obtaining multimedia converted content.
Method of operation of the service provision device. According to paragraph 1,
Further comprising sharing the output multimedia converted content to one or more other user terminals through a blockchain network.
Method of operation of the service provision device. A computer-readable recording medium on which a program for executing the method of any one of claims 1 to 10 on a computer is recorded.