KR20240074619A - Method and system for generating real-time video content - Google Patents

Abstract

The present disclosure relates to a method for generating real-time video content performed by at least one processor. A method of generating real-time video content includes generating text content based on a broadcast outline, generating a first voice based on the text content, and generating a first view based on at least one of the text content and the first voice. It includes generating and transmitting the first video including the first voice and the first view in real-time streaming.

Description

Real-time video content generation method and system {METHOD AND SYSTEM FOR GENERATING REAL-TIME VIDEO CONTENT}

This disclosure relates to a method and system for generating real-time video content, and specifically, to a method and system for automatically generating video content broadcast through real-time streaming.

With the advancement of science and technology, new media, a new communication medium that is not tied to existing mass media such as TV, radio, and newspapers, has emerged. The biggest difference between existing media and new media is that two-way communication is possible. New media transmits information through communication connections, allows people to share their opinions and reactions, and discuss various topics.

As an example of new media, real-time streaming broadcasts are showing high growth. Real-time streaming broadcasting is a broadcasting method that transmits video/audio content to many viewers in real time over the Internet. Live streaming broadcasts have the advantage of being able to interact with viewers in real time, such as responding to viewers' chatting in real time during the broadcast or reflecting viewer feedback in real time.

Meanwhile, a host who conducts a real-time streaming broadcast must plan broadcast content during each broadcast and devise lines in real time during the broadcast, which requires skilled skills in real-time broadcasting. In addition, the host of a live streaming broadcast must respond to viewers' reactions in real time while broadcasting, and deal with negative or inappropriate chats. The problem is that as the number of viewers increases, more human resources are required for such real-time response. There is.

The present disclosure provides a method for generating real-time video content, a computer program stored in a recording medium, and a device (system) to solve the above problems.

The present disclosure may be implemented in various ways, including as a method, device (system), or computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a method for generating real-time video content, performed by at least one processor, comprising: generating text content based on a broadcast outline; generating first voice based on the text content; generating a first view based on at least one of text content and the first voice, and transmitting the first video including the first voice and the first view in real-time streaming.

According to an embodiment of the present disclosure, the steps of obtaining a viewer's real-time chat for a first video transmitted through real-time streaming and generating a reaction text for at least a portion of the real-time chat using a chatbot model are further included. Includes.

According to an embodiment of the present disclosure, generating a second voice based on text content and a reaction text, generating a second view based on at least one of the text content or the second voice, and generating the second voice. And it further includes transmitting the second image including the second view through real-time streaming.

According to an embodiment of the present disclosure, generating a reaction text for at least a portion of the real-time chat may include generating a reaction text for at least a portion of the real-time chat, using a chatbot model, based on a broadcast outline or text content, and associated with broadcast content for at least a portion of the real-time chat. Includes the step of generating reaction text.

According to an embodiment of the present disclosure, the step of generating a reaction text for at least a portion of the real-time chat includes selecting one or more real-time chats associated with broadcast content among the real-time chat using a similarity measurement model and creating a chatbot model. It includes generating a reaction text for one or more selected real-time chats.

According to one embodiment of the present disclosure, the method further includes filtering the real-time chat using a hate-speech detection model configured to determine the harmfulness of the input text.

According to an embodiment of the present disclosure, the viewer's real-time chat includes at least one of text chat, image chat, sound chat, or video chat.

According to one embodiment of the present disclosure, based on the text content and the reaction text, generating the second voice includes generating the second voice using the transmission priority associated with the text content and the transmission priority associated with the reaction text. Includes creation steps.

According to one embodiment of the present disclosure, the viewer's real-time chat includes a donation chat associated with a donation and a general chat not associated with a donation, and the transmission priority associated with the reaction text is ahead of the transmission priority associated with the text content, and the reaction Among texts, the transmission priority associated with the reaction text for donation chat is ahead of the transmission priority associated with the reaction text for general chat.

According to an embodiment of the present disclosure, the step of generating text content based on the broadcast outline includes generating text content based on the broadcast outline in text format using a story creation model.

According to an embodiment of the present disclosure, the broadcast summary is a song list including a plurality of song names, and the step of generating text content based on the broadcast outline includes normalizing the plurality of song names and using a search engine. , collecting information associated with a plurality of songs included in the song list based on a plurality of normalized song names, summarizing the collected information using a summary model, and using a style conversion model, the summarized information Includes steps for changing the style.

According to an embodiment of the present disclosure, the broadcast outline is created based on the collection results of collecting popular search word information using a web that provides at least one of a popular search word or trend service.

According to an embodiment of the present disclosure, the broadcast summary is created based on the results of collecting popular news information using a web that provides news.

According to an embodiment of the present disclosure, the step of generating text content based on the broadcast outline includes generating text content based on the broadcast outline in an image format using an image recognition model.

According to an embodiment of the present disclosure, generating text content based on the broadcast synopsis includes generating text content based on the broadcast synopsis in sound format using a speech recognition model.

According to an embodiment of the present disclosure, the step of generating text content based on the broadcast outline includes generating text content based on the broadcast outline in video format using an image recognition model.

According to an embodiment of the present disclosure, obtaining a real-time chat of a viewer for a first video transmitted in a real-time streaming manner, modifying a broadcast outline based on at least part of the real-time chat, using a story creation model Thus, generating modified text content based on the modified broadcast outline, generating a third voice and a third view based on the modified text content, and a third including the third voice and the third view. It further includes transmitting the video in real-time streaming.

According to one embodiment of the present disclosure, the first view includes at least one of a virtual streamer in the form of a character, a virtual streamer in the form of a virtual person, or a background view associated with text content.

According to an embodiment of the present disclosure, the first view includes a virtual streamer in the form of a character or a virtual person, and generating the first view based on at least one of text content or the first voice includes emotion prediction. It includes generating a first view including a change in the virtual streamer's facial expression reflecting an emotion associated with the text content, using the model and the facial expression change model.

According to an embodiment of the present disclosure, the first view includes a virtual streamer in the form of a character or a virtual person, and generating the first view based on at least one of text content or the first voice includes talking head and generating, using the model, a first view including a change in the mouth shape of the virtual streamer uttering the first voice.

According to an embodiment of the present disclosure, the first view includes a virtual streamer in the form of a character or a virtual person, and generating the first view based on at least one of text content or the first voice includes: a gesture model It includes generating a first view including a gesture of the virtual streamer using .

A computer program stored in a computer-readable recording medium is provided to execute the method according to an embodiment of the present disclosure on a computer.

An information processing system according to an embodiment of the present disclosure includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and providing text content based on a broadcast outline. Generate a first voice based on the text content, generate a first view based on at least one of the text content or the first voice, and generate a first image including the first voice and the first view in real time. Contains commands for streaming transmission.

According to some embodiments of the present disclosure, broadcast content is automatically generated and transmitted in real time, and immediate response to viewers' reactions is possible, thereby saving human resources required for real-time broadcasting.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 is a diagram illustrating an example of a method for generating real-time video content according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a configuration in which an information processing system according to an embodiment of the present disclosure is connected to enable communication with a plurality of user terminals.
Figure 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure.
Figure 4 is a block diagram showing the internal configuration of a processor of an information processing system according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an example of generating text content based on a broadcast outline using a story creation model according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating an example of generating text content for a music broadcast based on a broadcast outline including a plurality of song names according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating an example of generating text content based on a broadcast outline of various modalities according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating an example of generating a reaction text for a viewer's real-time chat, according to an embodiment of the present disclosure.
FIG. 9 is a diagram illustrating an example of generating a script based on text content and reaction text according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating an example of generating a voice based on a script and generating a view based on the script and/or the voice according to an embodiment of the present disclosure.
FIG. 11 is a diagram illustrating an example of generating a view including a virtual streamer based on a script and/or voice according to an embodiment of the present disclosure.
FIG. 12 is a diagram illustrating an example of generating a video including audio and a view and transmitting it to a viewer according to an embodiment of the present disclosure.
Figure 13 is an exemplary diagram showing an artificial neural network model according to an embodiment of the present disclosure.
14 is a flowchart illustrating an example of a method for generating real-time video content according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In this disclosure, 'machine learning model' may include any model used to infer an answer to a given input. According to one embodiment, the machine learning model may include an artificial neural network model including an input layer (layer), a plurality of hidden layers, and an output layer. Here, each layer may include multiple nodes. In this disclosure, a machine learning model may refer to an artificial neural network model, and an artificial neural network model may refer to a machine learning model. In the present disclosure, a content generator (e.g., story generation model, summary model, image recognition model, speech recognition model, image recognition model, etc.), hate speech filter, similarity measurement model, content chatbot, voice synthesizer, view synthesizer (e.g. For example, emotion prediction models, facial expression change models, talking head models, gesture models, etc.) can be implemented as machine learning models. According to one embodiment, a machine learning model can be run on a server that includes a GPU for fast inference, and can communicate with a client using a REST API.

In this disclosure, 'display' may refer to any display device associated with a computing device, e.g., any display device capable of displaying any information/data controlled by or provided by the computing device. can refer to.

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A. .

1 is a diagram illustrating an example of a method for generating real-time video content according to an embodiment of the present disclosure. The information processing system can automatically generate video content and transmit it in a real-time streaming manner under the management of a first user (e.g., a real-time streaming broadcast manager, hereinafter referred to as an administrator). Additionally, the information processing system can immediately respond to real-time chatting by a second user (eg, a viewer of a real-time streaming broadcast, hereinafter referred to as viewer 160) regarding video content transmitted through real-time streaming.

According to one embodiment, the broadcast overview 110 may be given by an administrator. For example, the manager can input a broadcast outline 110, which is an overall outline of broadcast content, through the manager terminal, and the information processing system can receive the broadcast outline 110 from the manager terminal.

Additionally or alternatively, the broadcast outline 110 may be automatically generated based on information collection results using the web (eg, crawling results). For example, the broadcast outline 110 may be created based on the results of collecting popular search word information and/or popular news information using the web. The process of collecting popular search word information or popular news information using the web and/or creating the broadcast outline 110 based on this is performed by an information processing system and/or a user terminal (for example, an administrator terminal). It can be done.

The information processing system may generate text content based on the broadcast outline 110 using the content generator 120. For example, the information processing system may use a story generation model to generate text content of a story broadcast based on the broadcast outline 110 in text format.

According to one embodiment, the broadcast summary 110 may be of various types, such as a broadcast summary in text format, a broadcast summary in sound format, a broadcast summary in image format, a broadcast summary in video format, or a combination of at least some of these. The information processing system may generate text content using a separate content generator 120 that is configured differently or learned differently depending on the type or format of the broadcast outline 110. A specific example in which the information processing system generates text content based on the broadcast outline 110 will be described in more detail later with reference to FIGS. 4 to 7 .

The generated text content may be included in the script after passing through the stream controller 130. During this process, the administrator can modify the text content included in the script. If the text content is modified by the administrator, the modified text content may be included in the script.

In addition to text content, the script may include reaction text for real-time chat. The stream controller 130 can configure a script by considering the transmission priority of text content and reaction text. In this regard, we introduce a process in which an information processing system generates reaction text for real-time chat using a content chatbot. It will be described later.

The information processing system may then use the audio and view generator 140 to generate an image 150 including the audio and view. Here, the voice may include a voice speaking the script. Additionally, the view may include at least one of the appearance of the virtual streamer in the form of a character, the appearance of the virtual streamer in the form of a virtual person, or a background view associated with text content. In one embodiment, when the view includes the appearance of the virtual streamer, the view may include changes in the virtual streamer's facial expression, changes in the shape of the virtual streamer's mouth, and/or gestures of the virtual streamer. A specific example in which the information processing system generates the image 150 including audio and views will be described in more detail later with reference to FIGS. 4 and 10 to 12.

The information processing system can transmit video 150 including voice and views in real-time streaming. The viewer 160 can watch the video 150 broadcast in real-time streaming using the viewer 160 terminal, and can transmit real-time chat about the video 150. The information processing system can obtain real-time chat about the video 150 from a plurality of viewer 160 terminals.

The information processing system may generate reaction text for at least a portion of the real-time chat. First, the information processing system may filter the real-time chat using a hate speech filter 170 configured to determine the harmfulness of the input text before generating reaction text for the real-time chat. You can then generate reaction text for your live chat.

According to one embodiment, instead of simply generating a reaction to the chat based on the chat, the information processing system uses the content chatbot 180 to generate a reaction text associated with the broadcast content by considering the broadcast outline 110 or text content. can be created. A specific example in which the information processing system generates reaction text for real-time chatting will be described in more detail later with reference to FIGS. 4 and 8.

When a reaction text is generated, the information processing system can construct a script based on the text content and the reaction text. Specifically, the information processing system can use the stream controller 130 to configure a script by considering the transmission priority associated with text content and the transmission priority associated with reaction text. In one embodiment, for immediate response to the viewer's 160 real-time chat, the information processing system may preferentially include reaction text in the transcript. In other words, the transmission priority associated with the reaction text may be ahead of the transmission priority associated with the text content. A specific example in which the information processing system configures a script using the stream controller 130 will be described in more detail later with reference to FIGS. 4 and 9.

The information processing system can generate voice and views using a script composed based on text content and reaction text, and can transmit video 150 including voice and views in real-time streaming.

As described above, the information processing system can automatically generate and transmit broadcast content in real time and immediately respond to the reactions of viewers 160, so human resources required for real-time broadcasting can be saved.

In the above description of FIG. 1, for convenience of explanation, it is assumed that the process of generating video content and transmitting it in a real-time streaming manner is performed according to a predetermined order, but the present disclosure is not limited to the order described. Due to the nature of the real-time streaming method, at least some processes may be performed simultaneously and/or repeatedly.

In addition, in the above description, the real-time video content generation method is described as being mainly performed by an information processing system, but it is not limited thereto, and at least some of the processes included in the real-time video content generation method of the present disclosure are performed using a separate device (e.g. For example, it may be performed by a user terminal or a separate device for computing assistance, etc.). However, for convenience of explanation, the following description will be made assuming that the real-time video content generation method of the present disclosure is mainly performed by an information processing system.

Figure 2 is a schematic diagram showing a configuration in which the information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals 210_1, 210_2, and 210_3. As shown, a plurality of user terminals 210_1, 210_2, and 210_3 may be connected to an information processing system 230 that can provide a real-time video content creation service, etc. through a network 220. Here, the plurality of user terminals 210_1, 210_2, and 210_3 may include user terminals that will be provided with a real-time video content creation service. In one embodiment, information processing system 230 is one or more server devices and/or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to real-time video content creation services, etc. , or may include one or more distributed computing devices and/or distributed databases based on cloud computing services.

The real-time video content creation service provided by the information processing system 230 includes a real-time video content creation application, a real-time broadcast viewing application, a mobile browser application, or a web browser installed on each of the plurality of user terminals 210_1, 210_2, and 210_3. It can be provided to the user through . For example, the information processing system 230 provides or responds to a real-time video content creation request, a video content change request, etc. received from the user terminals 210_1, 210_2, and 210_3 through a real-time video content creation application. Processing can be performed.

A plurality of user terminals 210_1, 210_2, and 210_3 may communicate with the information processing system 230 through the network 220. The network 220 may be configured to enable communication between a plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.) as well as user terminals (210_1, 210_2, 210_3). ) may also include short-range wireless communication between

In Figure 2, the mobile phone terminal (210_1), tablet terminal (210_2), and PC terminal (210_3) are shown as examples of user terminals, but they are not limited thereto, and the user terminals (210_1, 210_2, 210_3) use wired and/or wireless communication. This is possible and may be any computing device on which a real-time video content creation application, mobile browser application, or web browser, etc. can be installed and executed. For example, user terminals include AI speakers, smartphones, mobile phones, navigation, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, It may include wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, in Figure 2, three user terminals (210_1, 210_2, 210_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limited to this, and a different number of user terminals are connected to the network ( It may be configured to communicate with the information processing system 230 through 220).

According to one embodiment, the information processing system 230 may receive a broadcast overview from user terminals 210_1, 210_2, and 210_3 (eg, an administrator terminal). The information processing system 230 may then generate text content based on the broadcast outline. Then, the information processing system 230 can generate an image including voice and views based on the text content, and send the generated image to a plurality of user terminals 210_1, 210_2, and 210_3 (e.g. For example, it can be transmitted to a viewer terminal).

Additionally, the information processing system 230 may obtain real-time chat from viewers about the video. Then, the information processing system 230 may generate reaction text for real-time chatting and include the voice uttering the reaction text in the video. Additionally, the information processing system 230 may transmit an image including a voice uttering a reaction text to a plurality of user terminals 210_1, 220_2, and 230_3 in real-time streaming.

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of running a real-time video content creation application, a real-time broadcast viewing application, a mobile browser application, or a web browser, and capable of wired/wireless communication, for example, Figure 2 It may include a mobile phone terminal (210_1), a tablet terminal (210_2), a PC terminal (210_3), etc. As shown, the user terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, information processing system 230 may include memory 332, processor 334, communication module 336, and input/output interface 338. As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. Additionally, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or to output information and/or data generated from the user terminal 210.

Memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 are non-perishable mass storage devices such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), and flash memory. It may include a (permanent mass storage device). As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 210 or the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, the memories 312 and 332 may store an operating system and at least one program code (for example, code for a real-time video content generation application installed in the user terminal 210).

These software components may be loaded from a computer-readable recording medium separate from the memories 312 and 332. This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into memory 312, 332 based on a computer program installed by files provided over the network 220 by developers or a file distribution system that distributes installation files for applications. It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336. For example, processors 314 and 334 may be configured to execute received instructions according to program codes stored in recording devices such as memories 312 and 332.

The communication modules 316 and 336 may provide a configuration or function for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and may provide a configuration or function for the user terminal 210 and/or information processing. The system 230 may provide a configuration or function for communicating with other user terminals or other systems (for example, a separate cloud system, etc.). For example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as the memory 312 (e.g., text content creation request, text content modification request, transmission ranking) change request, etc.) may be transmitted to the information processing system 230 through the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 is transmitted through the communication module 316 of the user terminal 210 through the communication module 336 and the network 220. It may be received by the user terminal 210. For example, the user terminal 210 may receive text content from the information processing system 230 through the communication module 316.

The input/output interface 318 may be a means for interfacing with the input/output device 320. As an example, input devices may include devices such as cameras, keyboards, microphones, mice, etc., including audio sensors and/or image sensors, and output devices may include devices such as displays, speakers, haptic feedback devices, etc. You can. As another example, the input/output interface 318 may be a means for interfacing with a device that has components or functions for performing input and output, such as a touch screen, integrated into one. For example, the processor 314 of the user terminal 210 uses information and/or data provided by the information processing system 230 or another user terminal when processing instructions of a computer program loaded in the memory 312. A service screen, etc. configured as such may be displayed on the display through the input/output interface 318. In FIG. 3 , the input/output device 320 is shown not to be included in the user terminal 210, but the present invention is not limited to this and may be configured as a single device with the user terminal 210. Additionally, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. In FIG. 3 , the input/output interfaces 318 and 338 are shown as elements configured separately from the processors 314 and 334, but the present invention is not limited thereto, and the input/output interfaces 318 and 338 may be configured to be included in the processors 314 and 334. there is.

The user terminal 210 and information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the input/output devices 320 described above. Additionally, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, if the user terminal 210 is a smartphone, it may include components generally included in a smartphone, such as an acceleration sensor, a gyro sensor, a camera module, various physical buttons, and a touch screen. Various components such as buttons using a panel, input/output ports, and a vibrator for vibration may be implemented to be further included in the user terminal 210. According to one embodiment, the processor 314 of the user terminal 210 may be configured to operate an application that provides a real-time video content creation service. At this time, code associated with the corresponding application and/or program may be loaded into the memory 312 of the user terminal 210.

While a program for a real-time video content creation application is running, the processor 314 inputs input through an input device such as a touch screen, keyboard, camera including an audio sensor and/or an image sensor, and a microphone connected to the input/output interface 318. or receive selected text, images, videos, voices and/or actions, etc., and store the received text, images, videos, voices and/or actions in the memory 312 or use the communication module 316 and the network ( It can be provided to the information processing system 230 through 220). For example, the processor 314 may receive a user input requesting the creation of text content based on the broadcast outline and provide the user input to the information processing system 230 through the communication module 316 and the network 220. As another example, processor 314 may receive user input indicating a request for modification to reaction text. It can be provided to the information processing system 230 through the communication module 316 and the network 220.

The processor 314 of the user terminal 210 manages, processes, and/or stores information and/or data received from the input device 320, other user terminals, the information processing system 230, and/or a plurality of external systems. It can be configured to do so. Information and/or data processed by processor 314 may be provided to information processing system 230 via communication module 316 and network 220. The processor 314 of the user terminal 210 may transmit information and/or data to the input/output device 320 through the input/output interface 318 and output the information. For example, the processor 314 may display the received information and/or data on the screen of the user terminal.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 210 and/or a plurality of external systems. Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220. According to one embodiment, the information processing system 230 may generate text content based on the broadcast outline received from the user terminal 210 (eg, an administrator terminal). Then, the information processing system 230 may generate an image including voice and views based on the text content, and send the generated image to a plurality of user terminals 210 (e.g., viewer terminal) in real-time streaming. ) can be transmitted.

Additionally, the information processing system 230 may generate reaction text for the viewer's real-time chat and include a voice uttering the reaction text in the video. Additionally, the information processing system 230 may transmit an image including a voice uttering a reaction text to a plurality of user terminals 210 in real-time streaming.

The processor 334 of the information processing system 230 uses the output device 320, such as a display output capable device (e.g., touch screen, display, etc.), an audio output capable device (e.g., speaker), of the user terminal 210. It may be configured to output processed information and/or data. For example, the processor 334 of the information processing system 230 provides the generated image to the user terminal 210 through the communication module 336 and the network 220, and displays the image on the user terminal 210. It may be configured to output through an output capable device, a sound output capable device, etc.

Figure 4 is a block diagram showing the internal configuration of the processor 334 of the information processing system according to an embodiment of the present disclosure. Figure 4 is only an example of the internal configuration of the processor 334 and does not show only the essential configuration of the processor 334, so it may be configured differently depending on the embodiment. For example, other configurations other than the illustrated configuration may be additionally included, some of the illustrated configurations may be omitted, and some of the illustrated configurations may be included in another device (e.g., a processor of a user terminal). . Additionally, in FIG. 4, the processor 334 is shown as a single processor, but the present invention is not limited thereto, and the processor 334 may be comprised of a plurality of processors.

According to one embodiment, the processor 334 of the information processing system includes a content generator 410, a reaction text generator 420, a transmission control unit 430, a voice generator 440, a view generator 450, It may include an image transmission unit 460.

The content generator 410 may generate text content based on the broadcast outline. Here, the broadcast outline may be input by the administrator through the administrator terminal, may be created based on the results of information collection using the web, or may be created/modified based on the viewer's real-time chat. Additionally, the broadcast outline may be a broadcast outline of various modalities. The content generator 410 may generate text content using a separate content generator that is configured differently or learned differently depending on the type or format of the broadcast outline.

For example, the broadcast synopsis may be any format of broadcast synopsis entered by an administrator. In this case, the content generator 410 may use the story creation model to generate text content for the story broadcast based on the broadcast outline. This will be described in more detail later with reference to FIG. 5 .

As another example, the broadcast summary may include a song list in text format including a plurality of song names entered by an administrator. In this case, the content generator 410 may generate text content of the music broadcast based on the broadcast outline using a search engine, summary model, style conversion model, etc. This will be described in more detail later with reference to FIG. 6.

As another example, the broadcast outline may be at least one of a broadcast outline in an image format, a broadcast outline in a sound format, or a broadcast outline in a video format input by an administrator. The content generator 410 may generate text content based on the broadcast outline using an image recognition model, a voice recognition model, a video recognition model, etc., depending on the modality of the broadcast outline. This will be described in more detail later with reference to FIG. 7 .

According to one embodiment, the broadcast outline may be automatically generated based on information collection results using the web (eg, crawling results). For example, the broadcast outline may be created based on the results of collecting popular search term information using the web that provides at least one of popular search term or trend services. As another example, the broadcast summary may be created based on the results of collecting popular news information using the web that provides news services. As another example, the broadcast outline may be created based on the results of collecting popular music information using the web that provides a popular music chart service. The process of collecting popular search word information, popular news information, popular music information, etc. using the web and/or creating a broadcast outline based on this is performed by the content creation unit 410 and/or the user terminal (e.g., administrator It can be performed by a terminal).

The content generator 410 may generate text content based on the broadcast outline generated based on the collected information. A method of generating text content based on a broadcast outline generated based on collected information may be performed in the same/similar manner to a method of generating text content based on a broadcast outline input by an administrator. As an example, the content generator 410 may use a summary model to generate a broadcast overview or text content by summarizing at least some of the results of collecting popular news information using a web that provides news services. there is. As another example, the content generator 410 uses a paraphrasing model to reconstruct at least part of the results of collecting popular news information using a web that provides news services, thereby providing a broadcast summary or text content. can be created.

According to one embodiment, broadcast content may be determined or modified based on viewers' real-time chatting about the video. For example, a broadcast synopsis may be created or modified based on at least part of a live chat. In this case, the content generator 410 may generate text content based on the created/modified broadcast outline.

The reaction text generator 420 may generate reaction text for at least some of the viewers' real-time chat. For example, first, the reaction text generator 420 may filter real-time chat using a hate speech filter configured to determine the harmfulness of the input text. Then, the reaction text generator 420 may select one or more real-time chats from among the filtered real-time chats for which to generate a reaction text. Then, the reaction text generator 420 may generate reaction text for one or more selected real-time chats.

According to one embodiment, the reaction text generator 420 may generate a reaction text related to the broadcast content by considering the broadcast outline or text content instead of simply generating a reaction to the chat based on the chat. A specific example of how the reaction text generator 420 generates a reaction text will be described in more detail later with reference to FIG. 8 .

The transmission control unit 430 may construct a script based on the text content created/edited by the content creation unit 410 and the reaction text by the reaction text creation unit 420. The transmission control unit 430 may configure a script including text content and reaction text by considering the transmission priority associated with the text content and the transmission priority associated with the reaction text.

According to one embodiment, for an immediate response to a viewer's real-time chat, the transmission control unit 430 may preferentially include reaction text in the script. In other words, the transmission priority associated with the reaction text may be ahead of the transmission priority associated with the text content.

According to one embodiment, the viewer's real-time chat may include donation chat associated with donations and general chat not associated with donations. The transmission control unit 430 may preferentially include reaction text for donation chat related to donations in the script. In other words, among reaction texts, the transmission priority associated with the reaction text for donation chat may be ahead of the transmission priority associated with the reaction text for general chat.

According to one embodiment, the manager may, through the administrator terminal, text content generated by the content creation unit 410, transmission priority associated with the text content, reaction text generated by the reaction text generation unit 420, and/or You can modify the transmission priority associated with the reaction text. If the text content, reaction text, and/or transmission priority are modified by the administrator, the transmission control unit 430 may configure the script by considering the modifications made by the administrator. A specific example in which the transmission control unit 430 configures a script based on text content and reaction text will be described in more detail later with reference to FIG. 9.

The voice generator 440 may generate voice based on the script. Here, the voice may be a voice that utters the script. For example, the voice generator 440 may generate a voice for uttering a script using a random voice synthesis (Text To Speech (TTS)) model. According to one embodiment, the administrator can select the virtual streamer's voice characteristics (eg, gender, age, region, voice level, speaking speed, personality, tone, etc.) of the virtual streamer through the administrator terminal. In this case, the voice generator 440 may generate a voice for uttering the script by reflecting the voice characteristics selected by the administrator.

The view generator 450 may generate a view based on a script and/or voice. Here, the view may include at least one of the appearance of the virtual streamer in the form of a character, the appearance of the virtual streamer in the form of a virtual person, or a background view associated with text content. In one embodiment, when the view includes the appearance of the virtual streamer, the view may include changes in the virtual streamer's facial expression, changes in the shape of the virtual streamer's mouth, and/or gestures of the virtual streamer. For example, the view generator 450 may use an emotion prediction model and an expression change model to generate a change in the virtual streamer's facial expression that reflects the emotion associated with the text content. Additionally or alternatively, the view generator 450 may use a talking head model to generate a change in the mouth shape of the virtual streamer uttering the first voice. Additionally or alternatively, the view generator 450 may generate a gesture of the virtual streamer using a gesture model. A specific example in which the voice generator 440 and the view generator 450 generate an image including voice and a view will be described in more detail later with reference to FIGS. 10 to 12 .

The video transmitter 460 can transmit video including audio and views in real-time streaming.

FIG. 5 is a diagram illustrating an example of generating text content 520 based on a broadcast outline 510 using a story creation model 500 according to an embodiment of the present disclosure. According to one embodiment, the information processing system may use the story generation model 500 to generate text content 520 of the story broadcast based on the broadcast outline 510. Here, the broadcast outline 510 is input by the administrator through the administrator terminal, or is a result of information collection using the web (for example, a crawl result for a website providing a popular search term service, trend service, or news service, etc. ), it may be created or modified based on the viewer's real-time chat, or it may be a combination of at least some of these.

According to one embodiment, the story generation model 500 may be a pre-trained language model (eg, Transformer) or a transfer-learned model based on a pre-trained language model. For example, the story generation model 500 may be a model of an encoder-decoder structure learned to generate text content 520 based on the broadcast outline 510. In this case, the broadcast outline 510 (or a feature vector representing it, etc.) may be an input to the encoder of the story generation model 500, and the text is converted into text in an autoregressive manner by the decoder of the story generation model 500. Content 520 may be predicted one unit (such as one word or one token).

According to one embodiment, the information processing system may use a tokenization module to convert the broadcast outline 510 into an input format that can be processed by the story creation model 500. For example, the broadcast outline 510 in text format may be input into the story creation model 500 in a tokenized state with a predefined token after passing through a tokenization module.

Additionally, the broadcast overview 510 may further include data in various modalities as well as data in text format. For example, the broadcast outline 510 may further include data in at least one of a sound format, a video format, and an image format as well as data in a text format. In this case, a special token may be included in the input of the story creation model 500. For example, a compressed embedding vector obtained based on data of various modalities (e.g., sound format, video format, image format, etc.) may be included at the forefront of the input of the story generation model 500. As a specific example, the special token is a compressed vector obtained by vector quantization of a hidden representation sequence generated as a recognition result of a speech recognition model, a recognition result of an image recognition model, or a recognition result of an image recognition model. You can.

The story creation model 500 may generate text content 520 based on the input (tokenized) broadcast outline 510. For example, the story creation model 500 can generate a tokenized output value based on the tokenized broadcast outline 510, and the tokenized output value is data in text form through a reverse tokenization process. It may be converted to text content 520. In one embodiment, when the input of the story generation model 500 includes a special token (e.g., a compressed embedding vector), the story generation model 500 generates text content 520 by considering the special token. can do.

The structure, inference method, and/or learning method of the story generation model 500 described above are only examples, and may be implemented differently from those described above or learned by other methods. For example, any story generation model 500 that can be adopted by those skilled in the art can be used in a real-time video content generation method.

FIG. 6 is a diagram illustrating an example of generating text content 660 of a music broadcast based on a broadcast outline 610 including a plurality of song names according to an embodiment of the present disclosure. According to one embodiment, the information processing system may generate text content 660 of a music broadcast based on a broadcast outline 610 that includes a list of songs in text format including a plurality of song names. For example, the broadcast outline 610 may be a song list including the order number, title, singer, and description for each of a plurality of songs. In one embodiment, the broadcast outline 610 is input by an administrator through an administrator terminal, or is based on information collection results using the web (for example, crawling results for a website providing a popular music chart service, etc.). It may be created based on the viewer's real-time chat, may be created or modified based on the viewer's real-time chat, or may be a combination of at least some of these.

As an example, the information processing system may first normalize the broadcast outline 610, which includes a text format song list including a plurality of song names. As a specific example, the information processing system can distinguish from the song name 'LV-After LOVE (feat. Hip boy)' that 'LV' is the singer and 'After LOVE' is the song title, and additional information such as featuring information can be removed. You can. In one embodiment, if the broadcast synopsis 610 is already given in a normalized format, the information processing system may skip the normalization process.

Then, the information processing system can collect information based on the (normalized) broadcast outline 610 using the web to obtain a collection result 620. As a specific example, the information processing system can obtain a collection result 620 by crawling information about songs included in the broadcast outline 610 using a search engine.

Then, the information processing system may use the summary model 630 to generate a summary result 640 that summarizes the collected results 620. According to one embodiment, the summary model 630 may be a transfer learned model based on a pre-trained language model (eg, Transformer). For example, the summary model 630 may be a model with an encoder-decoder structure learned to generate summary text data based on long text data that is the subject of summary. In this case, the collection result 620 may be the input of the encoder of the summary model 630, and the summary result 640 may be converted into one unit (one word or one token) in an auto-regressive manner by the decoder of the summary model 630. etc.) can be predicted one by one. The structure, inference method, and/or learning method of the above-described summary model 630 are only examples, and may be implemented differently from those described above or learned by other methods. For example, any summary model 630 that can be adopted by those skilled in the art can be used in the real-time video content generation method.

Additionally, the information processing system may use the style transformation model 650 to change the style of the summary result 640 to generate text content 660. For example, the information processing system may use the style conversion model 650 to generate text content 660 by converting the summary result 640 of written language into spoken language with a friendly tone. According to one embodiment, information about the style to be converted using the style conversion model 650 may be set by the administrator through the administrator terminal.

FIG. 7 is a diagram illustrating an example of generating text contents 716, 726, and 736 based on broadcast outlines 712, 722, and 732 of various modalities according to an embodiment of the present disclosure. According to one embodiment, the information processing system creates the broadcast outline (712, 722, 732) using the image recognition model 710, the voice recognition model 720, or the image recognition model 730, depending on the modality of the broadcast outline. Based on this, text content (716, 726, 736) can be created. The broadcast outline may include a broadcast outline 712 in an image format, a broadcast outline 722 in a sound format, or a broadcast outline 732 in a video format. These broadcast summaries 712, 722, and 732 may be input by the administrator through the administrator terminal, may be obtained as a result of information collection using the web, or may be collected based on viewers' real-time chat.

As an example, the information processing system may use the image recognition model 710 to generate text content 716 based on the broadcast outline 712 in image format. According to one embodiment, the image recognition model 710 may be a pre-trained image recognition model (eg, Swin Transformer) or a transfer-learned model based on the pre-trained image recognition model. In one embodiment, the image recognition model 710 may be a model learned to predict objects included in the image based on the image.

As a specific example, the information processing system may first convert the broadcast outline 712 in image format into an input format that can be processed by the image recognition model 710. For example, the information processing system can convert the broadcast outline 712 in image format into a set of a plurality of patches (e.g., a set of patches with a size of 4 pixel by 4 pixel) and input it into the image recognition model 710. . The image recognition model 710 may output a sequence of hidden expressions as a recognition result 714 based on a set of a plurality of input patches. The information processing system may generate text content 716 based on the output recognition result 714.

As another example, the information processing system may use the speech recognition model 720 to generate text content 726 based on the broadcast outline 722 in sound format. According to one embodiment, the speech recognition model 720 may be a pre-trained speech recognition model or a transfer learned model based on a pre-trained speech recognition model. For example, the speech recognition model 720 may be a transformer encoder-decoder series model learned based on training data consisting of a pair of training speech and transcription data.

As a specific example, the information processing system may first convert the broadcast outline 722 in sound format into an input format that can be processed by the speech recognition model 720. For example, the information processing system can convert the sound format broadcast outline 722 into a log scale spectrogram and input it into the voice recognition model 720. The speech recognition model 720 may output a sequence of hidden expressions as a recognition result 724 based on the input spectrogram. For example, the speech recognition model 720 may predict the sequence of the hidden expression by predicting the next token based on the previously predicted token sequence and output it as the recognition result 724. The information processing system may generate text content 726 based on the output recognition result 724.

As another example, the information processing system may use the image recognition model 730 to generate text content 736 based on the broadcast outline 732 in video format. According to one embodiment, the image recognition model 730 may be a model with a structure that extends the time dimension of a pre-trained image recognition model (eg, Swin Transformer). In one embodiment, the image recognition model 730 may be a model learned to predict actions, actions, etc. included in the image based on the image.

As a specific example, the information processing system may first convert the broadcast outline 732 in video format into an input format that can be processed by the video recognition model 730. For example, in the video format broadcast overview 732, the information processing system divides the images included in the video into a set of a plurality of patches (for example, 4pixel by 4pixel with a dimension representing time added in a 4pixel sized patch). It can be converted into a set of patches of 4 pixel by 1) and input into the image recognition model 730. The image recognition model 730 may output a sequence of hidden expressions as a recognition result 734 based on a set of a plurality of input patches. The information processing system may generate text content 736 based on the output recognition result 734.

The information processing system is based on the recognition result 714 of the image recognition model 710, the recognition result 724 of the voice recognition model 720, or the recognition result 734 of the image recognition model 730. The process of generating 726, 736) can be performed similarly to that described above with reference to FIG. 5. For example, the information processing system can generate a compressed embedding vector by vector quantizing the sequence of hidden expressions output as recognition results (714, 724, 734), and include the compressed embedding vector in the input of the story generation model. By doing this, text content (716, 726, 736) can be generated.

Additionally or alternatively, the information processing system may use image recognition model 710, speech recognition model 720, or image recognition model 730 to generate reaction text. This will be described in detail later with reference to FIG. 8.

The structure, inference method, and/or learning method of the above-described image recognition model 710, voice recognition model 720, and image recognition model 730 are only examples, and may be implemented differently from those described above or learned by other methods. You can. For example, any image recognition model 710, voice recognition model 720, or image recognition model 730 that can be adopted by those skilled in the art can be used in the real-time video content generation method.

FIG. 8 is a diagram illustrating an example of generating a reaction text 836 for a viewer's real-time chat, according to an embodiment of the present disclosure. The information processing system may generate reaction text 836 for at least some of the viewers' real-time chat.

First, the information processing system can obtain real-time chat from viewers about videos transmitted in a streaming manner. For example, the information processing system can obtain viewers' real-time chats by crawling the viewers' real-time chats entered into the video streaming platform. Received real-time chat may be accumulated in a chat queue 812.

According to one embodiment, the information processing system may filter real-time chat using a hate speech filter 810 configured to determine the harmfulness of input text. For example, the information processing system uses a tokenization module to tokenize real-time chats included in the chat queue 812 and inputs them into the hate speech filter 810, thereby obtaining a harmful probability 814 for each real-time chat. You can. Then, the information processing system determines the chats whose harmful probability 814 is above/exceeds the predefined threshold as harmful chats, and removes the harmful chats from the chat queue to obtain a chat queue 822 from which the harmful chats have been removed. there is.

Additionally or alternatively, the information processing system may determine the real-time chat 832 associated with broadcast content during the real-time chat as the chat for which reaction text is to be generated. For example, the information processing system can obtain the similarity 824 for each real-time chat by inputting real-time chats included in the chat queue 822 (with harmful chats removed) into the similarity measurement model 820, Real-time chat 832 related to broadcast content can be selected based on similarity 824. Here, the similarity 824 may be the similarity between real-time chat and broadcast content, in which case the similarity measurement model 820 includes not only real-time chat, but also broadcast content information 834 (e.g., broadcast overview, text content, and /Or at least some of these (summary, keywords, etc.) may be entered together. As a specific example, the information processing system may input real-time chat and broadcast content information 834 included in the chat queue 822 (with harmful chat removed) into the similarity measurement model 820. Here, the similarity measurement model 820 may be a model of the transformer encoder series and may be a model learned through supervised contrast learning. Additionally, real-time chat and broadcast content information 834 may be input to the similarity measurement model 820 in tokenized form through a tokenization module. The similarity measurement model 820 may output a similarity 824 (for example, cosine similarity with a value between -1 and 1) between the input real-time chat and broadcast content information 834. The information processing system can select the real-time chat with the highest similarity (824) as the real-time chat (832) associated with the broadcast content.

The information processing system can generate reaction text 836 for real-time chat using the content chatbot 830. For example, the information processing system may use a content chatbot 830 to generate reaction text 836 based on real-time chat 832 associated with broadcast content. According to one embodiment, the content chatbot 830 may be a pre-trained language model (eg, Transformer) or a transfer-learned model based on a pre-trained language model. For example, the content chatbot 830 may be a model with an encoder-decoder structure learned using multi-turn conversation data.

According to one embodiment, instead of simply generating reaction text based on chatting, the information processing system may consider broadcast content and generate reaction text 836 associated with the broadcast content. For example, the information processing system may display real-time chat 832 and broadcast content information 834 associated with broadcast content (e.g., broadcast synopsis, text content, and/or at least some of them (summaries, keywords, etc.), etc.) It can be entered into the content chatbot (830). Here, real-time chat 832 and broadcast content information 834 related to broadcast content can be input to the content chatbot 830 in tokenized form through a tokenization module.

According to one embodiment, the viewer's real-time chat may include chat of various modalities (eg, image chat, sound chat, and/or video chat). In this case, real-time chat input to the content chatbot 830 may include a special token. For example, a (tokenized) real-time chat 832 in which compressed embedding vectors obtained based on data of various modalities (e.g., image format, sound format, video format, etc.) are input to the content chatbot 830. can be included at the forefront of As a specific example, the special token may be a compressed vector obtained by vector quantizing a hidden expression sequence generated as a recognition result of a speech recognition model, a recognition result of an image recognition model, or a recognition result of an image recognition model. Here, the voice recognition model, image recognition model, and image recognition model may be the same/similar to the model described above with reference to FIG. 7.

The content chatbot 830 may generate a reaction text 836 for the real-time chat 832 by considering the broadcast content information 834. For example, the content chatbot 830 can generate a tokenized output value based on tokenized real-time chat, and the tokenized output value is converted into reaction text (836), which is data in text form through a reverse tokenization process. ) can be converted to In one embodiment, when the input of the content chatbot 830 includes a special token (e.g., a compressed embedding vector), the content chatbot 830 can generate reaction text 836 by taking the special token into account. there is.

The structure, inference method, and/or learning method of the hate speech filter 810, similarity measurement model 820, and content chatbot 830 are only examples, and may be implemented differently from those described above or learned by other methods. there is. For example, any speech filter 810, similarity measurement model 820, or content chatbot 830 that can be adopted by a person skilled in the art can be used in the real-time video content generation method.

FIG. 9 is a diagram illustrating an example of generating a script 950 based on text content 910 and reaction texts 920 and 930 according to an embodiment of the present disclosure. The information processing system may construct the script 950 based on the text content 910 and the reaction text 920 and 930. According to one embodiment, the information processing system considers the transmission priority associated with the text content 910 and the transmission priority associated with the reaction texts 920 and 930, and transmits the text content 910 and the reaction texts 920 and 930. A script 950 including can be configured. Due to the nature of real-time video content, real-time chatting from viewers can be continuously received, and reaction text can also be continuously generated. Accordingly, the information processing system can continuously perform the task of configuring/updating the script 950.

For example, the information processing system uses the stream controller 900 to configure the tuple list 940 based on the text content 910, the first reaction text 920, and the second reaction text 930. You can. The tuple list 940 may include a plurality of tuples, and each tuple may include text (text content 910 or reaction text 920, 930) and a transmission priority. The information processing system may construct the script 950 based on the tuple list 940.

According to one embodiment, for an immediate response to the viewer's real-time chat, the information processing system may preferentially include reaction texts 920 and 930 in the script 950. That is, the transmission priorities associated with the reaction texts 920 and 930 (1 and 0, respectively, in the example of FIG. 9) may be ahead of the transmission priorities associated with the text content 910 (2 in the example of FIG. 9).

Additionally or alternatively, the viewer's live chat may include donation chat associated with donations and general chat not associated with donations. The information processing system may preferentially include reaction text for the donation chat associated with the donation into the script 950. That is, among the reaction texts 920 and 930, the transmission priority (0 in the example of FIG. 9) associated with the reaction text for donation chat (the second reaction text 930 in the example of FIG. 9) is the reaction text for general chat. It may be ahead of the transmission priority (1 in the example of FIG. 9) associated with (the first reaction text 920 in the example of FIG. 9).

According to one embodiment, text content 910, reaction text 920, 930, tuple list 940, and/or script 950 may be output to the administrator terminal. The administrator, through the administrator terminal, provides text content 910, transmission priority associated with text content 910, reaction text 920, 930, transmission priority associated with reaction text 920, 930, and/or script 950. ) can be modified. If the text content 910, reaction text 920, 930, and/or transmission priority, etc. are modified by the administrator, the information processing system may reconstruct the script 950 by taking into account the modifications made by the administrator. .

The above-described script configuration method is only an example of the present disclosure, and the script 950 of video content may be configured by another method. For example, a script 950 may be configured that includes a plurality of text content items (eg, each sentence of text content is composed of each text content item). The information processing system may update the script 950 whenever the reaction text is generated or periodically by adding reaction text to appropriate locations in the script 950 (e.g., between two specific text content items). .

10 illustrates an example of generating a voice 1012 based on a script 950 and generating a view 1022 based on the script 950 and/or the voice 1012 according to an embodiment of the present disclosure. It is a drawing. The information processing system may generate speech 1012 based on the script 950. Here, the voice 1012 may be a voice that utters the script 950. For example, the information processing system may use a speech synthesizer 1010 (e.g., a random speech synthesis (TTS) model, etc.) to generate speech 1012 that utters the script based on the script 950. You can.

According to one embodiment, the administrator can select the virtual streamer's voice characteristics (eg, gender, age, region, voice level, speaking speed, personality, tone, etc.) of the virtual streamer through the administrator terminal. In this case, the information processing system may use the voice synthesizer 1010 to generate a voice 1012 that utters the script 950 by reflecting the voice characteristics selected by the administrator.

Additionally, the information processing system may generate the view 1022 based on the script 950 and/or the voice 1012. For example, the information processing system may use a view synthesizer 1020 (e.g., any image synthesis system, etc.) to generate the view 1022 based on the script 950 and/or the audio 1012. You can. Here, the view 1022 may include at least one of the appearance of the virtual streamer in the form of a character (2D character or 3D character), the appearance of the virtual streamer in the form of a virtual person, or a background view associated with text content.

FIG. 11 is a diagram illustrating an example of generating a view 1022 including a virtual streamer based on a script 950 and/or voice 1012 according to an embodiment of the present disclosure. In one embodiment, when the view 1022 includes the appearance of the virtual streamer, the view 1022 may include a change in the virtual streamer's facial expression 1122, a change in the virtual streamer's mouth shape 1132, and/or a change in the virtual streamer's mouth shape 1132. It may include a trimmer gesture 1142.

According to one embodiment, the view 1022 may include a change in the virtual streamer's facial expression 1122 that reflects the emotion 1112 associated with the script 950 and/or the voice 1012. The information processing system may use the emotion prediction model 1110 and the facial expression change model 1120 to generate a facial expression change 1122 of the virtual streamer that reflects the emotion 1112 associated with the script 950.

For example, first, the information processing system can predict the emotion 1112 associated with the script 950 using the emotion prediction model 1110. Here, the emotion prediction model 1110 may be a transformer encoder series model learned to output an emotion label based on the input text. The information processing system may input at least part of the script into the emotion prediction model 1110. Here, at least part of the script may be input to the emotion prediction model 1110 in tokenized form through a tokenization module. The emotion prediction model 1110 may predict an emotion label based on at least part of the (tokenized) script.

Then, the information processing system may use the facial expression change model 1120 to generate a facial expression change 1122 of the virtual streamer based on the emotion 1112. Here, the facial expression change model 1120 may be a model learned to enable image manipulation of the input image based on the text prompt. The information processing system combines the virtual streamer's reference image (e.g., the virtual streamer's basic facial image) and the emotion 1112 (e.g., the emotion label predicted by the emotion prediction model 1110) into an expression change model. It can be entered at (1120). The facial expression change model 1120 can output an image of a virtual streamer with a changed facial expression based on the input emotion 1112, and the output image of the virtual streamer can be viewed as a facial expression change 1122 of the virtual streamer. (1022).

Additionally or alternatively, view 1022 may include mouth shape changes 1132 of the virtual streamer uttering voice 1012. For example, the information processing system may use the talking head model 1130 to generate mouth shape changes 1132 of the virtual streamer uttering the voice 1012. Here, the talking head model 1130 is based on an image containing audio and a person (an image in which the lower part of the person is masked) to generate an image in which the mouth shape of the person included in the image changes to naturally utter the audio. It may be a model of a learned generative adversarial network (GAN) structure. The information processing system may input a voice 1012 (e.g., a voice speaking a script) and a reference image of the virtual streamer (e.g., a primary facial image of the virtual streamer) into the talking head model 1130. there is. Here, the voice 1012 may be input in the form converted into a log-scale spectrogram. The talking head model 1130 may generate a mouth shape change 1132 of the virtual streamer uttering the voice 1012, and the generated mouth shape change 1132 may be included in the view 1022.

Additionally or alternatively, view 1022 may include a virtual streamer's gesture 1142 associated with voice 1012 . For example, the information processing system may use the gesture model 1140 to generate the virtual streamer's gesture 1142. Here, the gesture model 1140 may include a voice encoder that generates a latent expression from the voice 1012 and a style encoder that learns the latent space from the location information of the animation clip. The information processing system may input voice 1012 (e.g., a voice speaking script 950) and a clip of gesture animation into gesture model 1140. Here, the voice 1012 may be input in the form converted into a log-scale spectrogram. The gesture model 1140 may output a voice embedding based on the voice 1012 and a style embedding based on a clip of the gesture animation. The information processing system may implement the virtual streamer's gesture 1142 based on the output voice embedding and style embedding. The gesture 1142 of the implemented virtual streamer may be included in the view 1022.

The structure, inference method, and/or learning method of the emotion prediction model 1110, facial expression change model 1120, talking head model 1130, and gesture model 1140 are only examples and may be implemented differently from those described above. It can be learned by different methods. For example, any emotion prediction model 1110, facial expression change model 1120, talking head model 1130, and gesture model 1140 that can be adopted by a person skilled in the art can be used in the real-time video content generation method. .

FIG. 12 is a diagram illustrating an example of generating a video 1210 including a voice 1012 and a view 1022 and transmitting it to a viewer 1220 according to an embodiment of the present disclosure. The information processing system may transmit a video 1210 including a voice 1012 speaking the script and a view 1022 in a real-time streaming manner. According to one embodiment, the video 1210 may further include sounds other than the voice 1012 uttering the script. For example, the image 1210 may further include sounds such as background music, sound effects, and music included in the song list of a music broadcast.

The viewer 1220 can watch the video 1210 transmitted in a streaming manner through the viewer terminal. Additionally, the viewer 1220 can respond to the video 1210 by entering real-time chat through the viewer terminal. Here, real-time chat may include various types of chat such as image chat, video chat, and sound chat as well as text chat. Additionally, real-time chat may include a donation chat that includes donations sent to a user account associated with video 1210 (eg, a user account associated with an administrator) and a general chat that does not include donations.

Figure 13 is an exemplary diagram showing an artificial neural network model 1300 according to an embodiment of the present disclosure. The artificial neural network model 1300 is an example of a machine learning model, and in machine learning technology and cognitive science, it is a statistical learning algorithm implemented based on the structure of a biological neural network or a structure that executes the algorithm.

According to one embodiment, the artificial neural network model 1300 has nodes, which are artificial neurons that form a network by combining synapses, as in a biological neural network, repeatedly adjusting the weights of synapses to determine the correct input corresponding to a specific input. By learning to reduce the error between the output and the inferred output, a machine learning model with problem-solving capabilities can be represented. For example, the artificial neural network model 1300 may include a random probability model, a neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

According to one embodiment, the above-described content generator (e.g., story generation model, summary model, image recognition model, speech recognition model, image recognition model, etc.), hate speech filter, similarity measurement model, content chatbot, speech synthesizer, A view synthesizer (eg, emotion prediction model, facial expression change model, talking head model, gesture model, etc.) may be implemented in the form of an artificial neural network model 1300.

The artificial neural network model 1300 may be composed of nodes of one or more layers and connections between them. The artificial neural network model 1300 according to this embodiment can be implemented using one of various artificial neural network model structures including MLP. As shown in FIG. 13, the artificial neural network model 1300 includes an input layer 1320 that receives an input signal or data 1310 from the outside, and an output layer that outputs an output signal or data 1250 corresponding to the input data. (1340), located between the input layer 1320 and the output layer 1340, receives a signal from the input layer 1320, extracts the characteristics, and transmits it to the output layer 1340. n (where n is a positive integer) It consists of hidden layers (1330_1 to 1330_n). Here, the output layer 1340 receives signals from the hidden layers 1330_1 to 1330_n and outputs them to the outside.

The learning method of the artificial neural network model 1300 includes a supervised learning method that learns to optimize problem solving by inputting a teacher signal (correct answer), and an unsupervised learning method that does not require a teacher signal. ) There is a way. The artificial neural network model 1300 may be learned using the same/similar method to the learning method described above with reference to FIGS. 5 to 8, 10, and 11.

According to one embodiment, when the artificial neural network model 1300 is a story generation model, the input variable may include a vector representing or characterizing a broadcast outline in text format. In this way, when the above-described input variable is input through the input layer 1320, the output variable output from the output layer 1340 may be a vector representing or characterizing text content.

Additionally, when the artificial neural network model 1300 is a voice synthesizer, the input variable may include a vector representing or characterizing the script. In this way, when the above-described input variables are input through the input layer 1320, the output variable output from the output layer 1340 of the artificial neural network model 1300 may be a vector representing or characterizing the synthesized voice.

In this way, a plurality of input variables and a plurality of output variables corresponding to the input layer 1320 and the output layer 1340 of the artificial neural network model 1300 are matched, respectively, and the input layer 1320, the hidden layers 1330_1 to 1330_n, and By adjusting the synapse values between nodes included in the output layer 1340, learning can be done so that the correct output corresponding to a specific input can be extracted. Through this learning process, the characteristics hidden in the input variables of the artificial neural network model 1300 can be identified, and the nodes of the artificial neural network model 1300 can be used to reduce the error between the output variable calculated based on the input variable and the target output. You can adjust the synapse values (or weights) between them. Using the artificial neural network model 1300 learned in this way, text content can be generated, reaction text can be generated, or voices and views can be synthesized.

FIG. 14 is a flowchart illustrating an example of a method 1400 for generating real-time video content according to an embodiment of the present disclosure. According to one embodiment, the real-time video content generation method 1400 may be initiated by a processor (eg, at least one processor of an information processing system) generating text content based on a broadcast outline (S1410).

The processor may generate text content based on broadcast outlines of various modalities. For example, the processor may generate text content based on a broadcast outline in text format, a broadcast outline in sound format, a broadcast outline in image format, a broadcast outline in video format, etc.

For example, the processor may use a story generation model to generate text content based on a broadcast outline in text format. As another example, the broadcast outline is a song list including a plurality of song names, and the processor normalizes the plurality of song names and uses a search engine to search for a plurality of songs included in the song list based on the normalized plurality of song names. Relevant information can be collected. Then, text content can be generated by summarizing the collected information using the summary model and changing the style of the summarized information using the style conversion model. As another example, the processor may use an image recognition model to generate text content based on a broadcast outline in the form of an image, use a speech recognition model to generate text content based on a broadcast outline in the form of a sound, or use an image recognition model to generate text content based on a broadcast outline in the form of a sound. Using this, text content can be created based on the broadcast outline in video format.

According to one embodiment, the broadcast outline may be given by a user (eg, an administrator) or may be generated based on information collection results using the web (eg, crawling results). For example, the broadcast outline may be created based on the results of collecting popular search term information using the web that provides at least one of popular search term or trend services. As another example, the broadcast summary may be created based on the results of collecting popular news information using a web that provides news. The process of collecting popular search word information or popular news information using the web and/or creating a broadcast outline based on this may be performed by an information processing system and/or a user terminal (for example, an administrator terminal). there is.

Then, the processor may generate a first voice based on the text content (S1420). Here, the first voice may include a voice that utters the generated text content. For example, the processor may generate the first speech from text content using any speech synthesis model.

Additionally, the processor may generate a first view based on at least one of text content or the first voice (S1430). According to one embodiment, the first view may include at least one of a virtual streamer in the form of a character, a virtual streamer in the form of a virtual person, or a background view associated with text content.

When the first view includes a virtual streamer, the first view may include a change in the virtual streamer's facial expression, a change in the virtual streamer's mouth shape, and/or a gesture of the virtual streamer. According to one embodiment, the change in the virtual streamer's facial expression may be a change in the virtual streamer's facial expression that reflects the emotion associated with the text content, and can be generated using an emotion prediction model and an facial expression change model. Additionally or alternatively, the virtual streamer's mouth shape change may be a mouth shape change of the virtual streamer uttering the first voice, which may be generated using a talking head model. Additionally or alternatively, the virtual streamer's gestures may be generated using a gesture model.

The processor may transmit the first video including the first voice and the first view in real-time streaming (S1440). The viewer can use the viewer terminal to watch the first video broadcast in real-time streaming and transmit real-time chat about the first video. The processor may obtain real-time chat for the first video from a plurality of viewer terminals.

According to one embodiment, the processor may generate reaction text for at least part of the real-time chat using a chatbot model. For example, the processor may use a chatbot model to generate reaction text associated with broadcast content for at least a portion of the real-time chat based on the broadcast outline or text content.

According to one embodiment, the processor may generate reaction text for chatting that is highly related to broadcast content during real-time chatting. For example, the processor may use a similarity measurement model to select one or more real-time chats associated with broadcast content among the real-time chats. The processor may then use the chatbot model to generate reaction text for one or more selected live chats.

According to one embodiment, the viewer's real-time chat may include chat in various modalities (e.g., text chat, image chat, sound chat, and/or video chat), and the processor uses a chatbot model to chat in various modalities. You can create reaction text for chat.

According to one embodiment, the processor may filter the real-time chat using a hate speech detection model configured to determine the harmfulness of the input text before generating reaction text for the real-time chat.

The processor may then generate a second voice based on the text content and the reaction text. For example, the processor may generate the second voice using a delivery priority associated with the text content and a delivery priority associated with the reaction text. The second voice may include a voice that utters text content and reaction text according to transmission priority. Additionally, the processor may generate a second view based on at least one of text content or a second voice, and transmit a second video including the second voice and the second view in real-time streaming.

In one embodiment, when a reaction text is generated for a viewer's real-time chat, the processor may preferentially read the reaction text. That is, according to one embodiment, the transmission priority associated with the reaction text may precede the transmission priority associated with the text content.

Additionally or alternatively, the viewer's live chat may include donation chat associated with donations and general chat not associated with donations. In this case, the processor may preferentially read the reaction text for the donation chat related to the donation. In other words, among reaction texts, the transmission priority associated with the reaction text for donation chat may be ahead of the transmission priority associated with the reaction text for general chat.

According to one embodiment, the processor may modify broadcast content based on a viewer's real-time chatting about the first video transmitted through real-time streaming. For example, a broadcast synopsis may be modified based on at least part of the live chat. The processor may use the story generation model to generate modified text content based on the modified broadcast outline, generate a third voice and a third view based on the modified text content, and generate a third voice and a third view. The third video including the view can be transmitted in real-time streaming.

The flowchart included in the drawings and the above description are only examples and the scope of the present disclosure is not limited thereto. For example, according to another embodiment, some steps may be added/changed/deleted, and the order of each step may be changed.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. Media may be used to continuously store executable programs on a computer, or may be temporarily stored for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and There may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various other software, or servers.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

When implemented in software, the techniques may be stored on or transmitted through a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or the desired program code in the form of instructions or data structures. It can be used to transfer or store data and can include any other media that can be accessed by a computer. Any connection is also properly termed a computer-readable medium.

For example, if the Software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, disk and disk include CD, laser disk, optical disk, digital versatile disc (DVD), floppy disk, and Blu-ray disk, where disks are usually magnetic. It reproduces data optically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known. An exemplary storage medium may be coupled to the processor such that the processor may read information from or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage medium may reside within an ASIC. ASIC may exist within the user terminal. Alternatively, the processor and storage medium may exist as separate components in the user terminal.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: Broadcast overview
120: Content Generator
130: Stream controller
140: Voice and view generator
150: video
160: Viewer
170: Hate Speech Filter
180: Content chatbot

Claims (23)

In a method of generating real-time video content, performed by at least one processor,
Generating text content based on the broadcast outline;
Based on the text content, generating a first voice;
generating a first view based on at least one of the text content or the first voice; and
Transmitting the first video including the first voice and the first view in real-time streaming
A method of generating real-time video content, including.
According to paragraph 1,
Obtaining a viewer's real-time chat about the first video transmitted through the real-time streaming method; and
Using a chatbot model, generating reaction text for at least some of the real-time chatting
A method of generating real-time video content, further comprising:
According to paragraph 2,
generating a second voice based on the text content and the reaction text;
generating a second view based on at least one of the text content or the second voice; and
Transmitting the second video including the second voice and the second view in real-time streaming
A method of generating real-time video content, further comprising:
According to paragraph 2,
The step of generating reaction text for at least some of the real-time chatting includes:
Using the chatbot model, generating reaction text associated with broadcast content for at least some of the real-time chat based on the broadcast outline or the text content
A method of generating real-time video content, including.
According to paragraph 2,
The step of generating reaction text for at least some of the real-time chatting includes:
Using a similarity measurement model, selecting one or more real-time chats related to broadcast content among the real-time chats; and
Using the chatbot model, generating a reaction text for the selected one or more real-time chats
A method of generating real-time video content, including.
According to paragraph 2,
Filtering the real-time chat using a hate-speech detection model configured to determine the harmfulness of input text.
A method of generating real-time video content, further comprising:
According to paragraph 2,
The viewer's real-time chat is a method of generating real-time video content, including at least one of text chat, image chat, sound chat, and video chat.
According to paragraph 3,
Generating a second voice based on the text content and the reaction text includes:
Generating the second voice using a transmission priority associated with the text content and a transmission priority associated with the reaction text.
A method of generating real-time video content, including.
According to clause 8,
The viewer's real-time chat includes donation chat related to donations and general chat not related to donations,
The transmission priority associated with the reaction text is ahead of the transmission priority associated with the text content,
Among the reaction texts, the transmission priority associated with the reaction text for the donation chat is ahead of the transmission priority associated with the reaction text for the general chat.
According to paragraph 1,
The step of generating text content based on the broadcast outline is,
Using a story creation model, generating the text content based on a broadcast outline in text format
A method of generating real-time video content, including.
According to paragraph 1,
The broadcast outline is a song list including a plurality of song names,
The step of generating text content based on the broadcast outline is,
normalizing the plurality of song names;
Using a search engine, collecting information associated with a plurality of songs included in the song list based on the plurality of normalized song names;
summarizing the collected information using a summary model; and
Changing the style of the summarized information using a style conversion model.
A method of generating real-time video content, including.
According to paragraph 1,
The broadcast outline is generated based on a collection result of collecting popular search word information using a web that provides at least one of a popular search term or trend service.
According to paragraph 1,
The broadcast outline is generated based on the results of collecting popular news information using a web that provides news. A method of generating real-time video content.
According to paragraph 1,
The step of generating text content based on the broadcast outline is,
Using an image recognition model, generating the text content based on the broadcast outline in image format
A method of generating real-time video content, including.
According to paragraph 1,
The step of generating text content based on the broadcast outline is,
Using a speech recognition model, generating the text content based on the broadcast outline in sound format
A method of generating real-time video content, including.
According to paragraph 1,
The step of generating text content based on the broadcast outline is,
Using a video recognition model, generating the text content based on a broadcast outline in video format
A method of generating real-time video content, including.
According to paragraph 1,
Obtaining a viewer's real-time chat about the first video transmitted through the real-time streaming method;
modifying the broadcast outline based on at least some of the real-time chat;
Using a story generation model, generating modified text content based on the modified broadcast outline;
generating a third voice and a third view based on the modified text content; and
Transmitting the third video including the third voice and the third view in real-time streaming
A method of generating real-time video content, further comprising:
According to paragraph 1,
The first view includes at least one of a virtual streamer in the form of a character, a virtual streamer in the form of a virtual person, or a background view associated with the text content.
According to paragraph 1,
The first view includes a virtual streamer in the form of a character or virtual person,
Generating a first view based on at least one of the text content or the first voice includes:
Using an emotion prediction model and an expression change model, generating a first view including a change in the facial expression of the virtual streamer reflecting the emotion associated with the text content.
A method of generating real-time video content, including.
According to paragraph 1,
The first view includes a virtual streamer in the form of a character or virtual person,
Generating a first view based on at least one of the text content or the first voice includes:
Using a talking head model, generating a first view including a change in mouth shape of the virtual streamer uttering the first voice.
A method of generating real-time video content, including.
According to paragraph 1,
The first view includes a virtual streamer in the form of a character or virtual person,
Generating a first view based on at least one of the text content or the first voice includes:
Using a gesture model, generating a first view including a gesture of the virtual streamer
A method of generating real-time video content, including.
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 21 on a computer.
As an information processing system,
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
Generate text content based on a broadcast outline, generate a first voice based on the text content, generate a first view based on at least one of the text content or the first voice, and generate the first voice and the first voice. An information processing system comprising instructions for transmitting a first image including the first view in a real-time streaming manner.

Images