KR102692085B1 - Automatic recommendation music support system in streaming broadcasting - Google Patents

Abstract

The present invention is a streaming system that automatically recommends and plays background music or sound effects while broadcasting to viewers in real time through online streaming, transmits and receives chat data with the viewer terminal in real time, shoots the broadcast, and transmits broadcast data. A broadcasting unit, an emotion recognition unit that recognizes emotions from broadcast data transmitted from the streaming broadcast unit, an artificial intelligence data processing unit that recommends music corresponding to the emotion recognized by the emotion recognition unit, and a music system recommended by the artificial intelligence data processing unit. By providing an automatic recommended music support system for streaming broadcasts that includes a recommended music support unit that plays music, there is an effect of quickly and automatically supporting background music or broadcast effects appropriate for the broadcast situation without a separate system administrator.

Description

Automatic recommendation music support system in streaming broadcasting}

The present invention relates to an automatically recommended music support system in streaming broadcasts, and more specifically, to an automatically recommended music support system in streaming broadcasts that automatically recommends and plays sound effects and background music suitable for broadcast situations or viewer reactions. will be.

User satisfaction is increasing due to the increase in Internet speed, that is, transmission bandwidth. The rapid development of Internet-related technology has made it possible to provide Internet services that allow users to view and listen to multimedia content combining vivid audio and video through online searches. I did it. A representative example is multimedia streaming service. Streaming is the process of executing large-sized audio or video data, rather than downloading and executing it in its entirety, but dividing it into several files and transmitting them one after another in real time.

Regardless of age or gender, many people own smartphones and high-speed Internet has become commercialized. Personal broadcasting (or one-person broadcasting) services that allow anyone to broadcast anytime, anywhere are becoming widespread, and these personal broadcasts are known as streaming broadcasts. is being implemented.

As shown in Figure 1, such personal broadcasting is performed when an individual who wishes to broadcast the desired content broadcasts the desired content through a terminal such as a PC or smartphone that the individual uses individually, and a large number of viewers who want to view the content broadcast the content on their own. This is done by accessing the terminal hosting the personal broadcast using a personal terminal and watching the content. In this way, as viewers watch the content, they create their viewing impressions of the content using their personal terminals, and the generated viewing impressions are transmitted in real time to the terminal hosting the personal broadcast. Therefore, when the broadcasting terminal receives viewer opinions from the viewer terminal, the viewer opinions move from the bottom to the top of the chat window according to the reception time and are exposed to the broadcasting terminal and the viewer terminal.

For example, in Publication Patent No. 10-2012-0024233 (publication date: March 14, 2012), which is cited herein as a prior document, an example of the screen 320 on which each channel information is displayed is shown in Figure 2. As described above, it provides a menu that categorizes each channel by topic, such as the latest music channel, popular pop channel, popular pop channel, emotional theme channel, and weather theme channel. Information on each channel of the currently selected menu is displayed on the right side of the screen. For each channel, the channel number, title of the song currently being broadcast on that channel, and singer name are displayed, allowing the user to search for the desired channel and access the channel by clicking the listen to channel button 323.

However, these prior literature are only personal broadcasts in the form of radio broadcasts, and recently, broadcasting formats that enable two-way communication between users and viewers are supported, and in order to increase broadcast effectiveness, weather, season, season, and specific anniversaries are supported. It further enhances the broadcasting effect by playing background music that suits the broadcaster's situation, providing sound effects related to emotions or situations such as surprise, sadness, laughter, and tension, or chatting situations exchanged in real time with viewers. there is.

However, in order to provide appropriate background music or sound effects according to these situations, it is necessary to have personnel to operate the system in addition to the broadcaster appearing on the broadcast, and music recommendations are also made within the scope of the individual's experience or knowledge, making it difficult to obtain sufficient effect. There was a problem.

No. 10-2012-0024233 (Publication date: March 14, 2012)

The present invention was created to solve the above-mentioned problems, and its purpose is to provide an automatic recommended music support system in streaming broadcasting that can automatically recommend and play background music or sound effects suitable for the situation.

In addition, the purpose of the present invention is to provide an automatic recommendation music support system in streaming broadcasting that can automatically recommend and play background music requested by a viewer or sound effects corresponding to the viewer's chat through a chat window.

The technical problem to be achieved by this embodiment is not limited to the technical problems described above, and other technical problems may exist.

In a preferred embodiment of the present invention, in a system that automatically recommends and plays background music or sound effects while broadcasting to viewers in real time through online streaming, chat data is transmitted and received in real time with the viewer terminal, and the broadcast is filmed and broadcast. A streaming broadcasting unit that transmits data, an emotion recognition unit that recognizes emotions from broadcast data transmitted by the streaming broadcasting unit, an artificial intelligence data processing unit that recommends music corresponding to the emotion recognized by the emotion recognition unit, and the artificial intelligence An automatic recommended music support system for streaming broadcasting, including a recommended music support unit that plays music recommended by a data processing unit, is provided.

In addition, in a preferred embodiment of the present invention, in the above-described embodiment, a data collection unit that scores emotion data recognized from the emotion recognition unit in at least one streaming broadcasting unit, and the emotions collected by the data collection unit An automatic recommended music support system for streaming broadcasting, further comprising an artificial intelligence learning unit that learns recommended music from data, wherein the artificial intelligence data processing unit updates learning results from the artificial intelligence learning unit in preset units. provides.

Additionally, in a preferred embodiment of the present invention, in the above-described embodiment, the emotion recognition processing unit includes a streamer emotion recognition unit that recognizes the emotion of the streamer conducting the broadcast, and a streamer emotion recognition unit that recognizes the emotion of the viewer from the viewer's chat data. Provides an automatic recommendation music support system in streaming broadcasting, which includes a chat data emotion recognition unit that recognizes.

In addition, in a preferred embodiment of the present invention, in the above-described embodiment, the streamer emotion recognition unit includes a face recognition unit that recognizes the face of the streamer, and an emotion recognition unit that recognizes emotions from the recognized face data. Provides an automatic recommendation music support system for streaming broadcasting, which includes:

In addition, in a preferred embodiment of the present invention, in the above-described embodiment, the streamer emotion recognition unit extracts emotions through clip leveling or analyzes emotions through context translation. Provides an automatic recommended music support system.

In addition, in a preferred embodiment of the present invention, in the above-described embodiment, the artificial intelligence data processing unit sets the type of music, music playback volume, and playback time differently for each situation and emotion. Provides an automatic recommended music support system for broadcasting.

In addition, in a preferred embodiment of the present invention, an automatic recommendation music support system in streaming broadcasting is provided, characterized in that in the above-described embodiment, the recommended music is background music or sound effects.

According to the automatic recommended music support system for streaming broadcasting according to a preferred embodiment of the present invention, accuracy is improved through artificial intelligence learning and automatically recommends and plays background music or sound effects suitable for the situation, Broadcasting effectiveness can be improved.

In addition, according to the automatic recommendation music support system in streaming broadcasting according to a preferred embodiment of the present invention, by providing effect music or background sounds corresponding to viewers' chatting, it increases communication with viewers and induces viewer participation more effectively. There is an effect that can be done.

According to the automatic recommended music support system for streaming broadcasting according to a preferred embodiment of the present invention, suitable background music or sound effects can be played at precise timing without a separate operator for playing background music or sound effects.

Figure 1 is a system configuration diagram showing the configuration of a general real-time personal broadcasting system.
Figure 2 is an exemplary diagram showing an example of a screen displaying information on each channel in a real-time music broadcasting system according to prior literature.
Figure 3 is a block diagram showing an automatic recommended music support system for streaming broadcasting according to a preferred embodiment of the present invention.
Figure 4 is a block diagram showing a streamer emotion recognition process in an automatic recommendation music support system for streaming broadcasting according to a preferred embodiment of the present invention.
Figure 5 is a block diagram showing a chat emotion recognition process in the automatic recommendation music support system for streaming broadcasting according to a preferred embodiment of the present invention.
Figure 6 is a block diagram showing a new user group recommendation process in the automatic recommendation music support system for streaming broadcasting according to a preferred embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with an element in between. . In addition, when a part is said to "include" a certain component, this means that it does not exclude other components, but may further include other components, unless specifically stated to the contrary, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The terms “about,” “substantially,” and the like used throughout the specification are used to mean at or close to that value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used to enhance the understanding of the present invention. Precise or absolute figures are used to assist in preventing unscrupulous infringers from taking unfair advantage of stated disclosures. The term “step of” or “step of” as used throughout the specification of the present invention does not mean “step for.”

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed on a server connected to the terminal, apparatus, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is identifying data of the terminal. It can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

Figure 3 is a block diagram showing an automatic recommended music support system for streaming broadcasting according to a preferred embodiment of the present invention.

Referring to FIG. 3, the automatic recommendation music support system for streaming broadcasting according to a preferred embodiment of the present invention includes a streaming broadcasting unit 100, an emotion recognition unit 200, a data collection unit 300, and an artificial intelligence learning unit. (400) and a recommended music support unit (500).

Here, each component of FIG. 3 is generally connected through a network. For example, the streaming broadcasting unit 100, the emotion recognition unit 200, the data collection unit 300, the artificial intelligence learning unit 400, and the recommended music support unit 500 may be connected by wire within one system. If necessary, each component may be remotely supported by being connected to a separate system or server through a wired or wireless network.

Here, the network refers to a connection structure that allows information exchange between each node, such as a plurality of terminals and servers. Examples of such networks include a local area network (LAN) and a wide area network (WAN). Wide Area Network, Internet (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), and Wi-Fi. , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC ( It includes, but is not limited to, Near-Field Communication (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network.

In the following, the term at least one is defined as a term including singular and plural, and even if the term at least one does not exist, each component may exist in singular or plural, and may mean singular or plural. This should be self-explanatory. In addition, whether each component is provided in singular or plural form may be changed depending on the embodiment.

At least one system may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

First, the streaming broadcasting unit 100 captures a video of a streamer (broadcaster, also called BJ) captured through a camera, captures the streamer's voice through a microphone, and transmits it to a broadcasting platform (e.g., broadcasting data) as broadcasting data. Afreeca TV, etc.) or transmit to the viewer's individual terminal through a real-time streaming platform. At this time, the viewer's individual terminal may also be configured with the above-described computer.

In addition, the streaming broadcasting unit 100 transmits the text that the streamer inputs by typing or using a mouse to the viewer's individual terminal as chat data, collects the chat data received from the viewer's individual terminal, and opens a separate chat window on the broadcast screen. Displayed sequentially.

The emotion recognition unit 200 recognizes the streamer's emotions from broadcast data transmitted from the streaming broadcasting unit 100, recognizes the streamer's or viewer's emotions from chat data, and selects recommended music.

This emotion recognition unit 200 may largely include a streamer emotion recognition unit that recognizes the emotions of the streamer conducting the broadcast, and a chat data emotion recognition unit that recognizes the viewer's emotions from the viewer's chat data.

Recognizing the emotions of a streamer who is broadcasting involves recognizing the face of the streamer, learning the emotion from the recognized face data, and creating a model according to the face, as shown in Figure 4. After modeling emotions, the recognition of the streamer's face that changes during a streaming broadcast can be predicted through comparison with a previously learned model.

Additionally, in another preferred embodiment of the present invention, in addition to visually recognizing the streamer's emotions as described above, the emotions can also be recognized audibly from the streamer's voice waveform based on a pre-modeled emotion model.

In particular, more accurate emotion recognition is possible by complementing visual emotion recognition through face recognition and auditory emotion recognition through voice recognition.

Meanwhile, the chat data emotion recognition unit extracts situations and emotions based on chat data of multiple users. After excluding data related to situations and emotions, the chat situation and corresponding emotions are identified by a combination of keywords related to situations and emotions. It can be recognized. Similar to the streamer's emotion recognition, as shown in Figure 5, the emotion of each word or sentence from existing streaming chat is learned and modeled by artificial intelligence, and the emotion of the streaming chat is derived from a similar model among the previously learned models. By making predictions, you can recognize the emotions of chat data.

The artificial intelligence data processing unit selects recommended music based on either the emotion recognized by the streamer emotion recognition unit or the emotion recognized by the chat data emotion recognition unit, or both when both occur simultaneously. In other words, background music or effect music, which is recommended music that has already been learned, is selected in response to the relevant situation or emotion.

In another embodiment of the present invention, this process includes a data collection unit 300 that collects emotion data recognized from an emotion recognition unit in at least one streaming broadcasting unit, and emotion data collected in the data collection unit 300. Through the artificial intelligence learning unit 400 that learns recommended music, recommended music according to emotions can be continuously learned and updated.

For example, as shown in Figure 6, the streamer's playing music, downloaded music, preferences, reviews, etc. are classified into multiple user groups, a recommendation model for each preferred group is created from the classified user groups, and each user Recommended music can be selected by estimating from the recommendation model for each cluster. Through repetition of this process, artificial intelligence is learned, and effective background music or sound effects can be recommended to new streamers through the above process.

The system for automatically recommending music in streaming broadcasting according to a preferred embodiment of the present invention may run a service program in a streamer terminal, that is, a personal broadcasting system, using a script executed in a web browser, where the web browser is a program that allows users to use web (WWW: World Wide Web) services. It refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape and Explorer. ), Chrome, etc. Additionally, an application refers to an application on a terminal and includes, for example, an app running on a mobile terminal (smartphone).

The recommended music support unit 500 synthesizes recommended music selected by the artificial intelligence data processing unit of the emotion recognition unit 200 with voice data of streaming broadcast data. In other words, it is included in the broadcast data as background music or sound effects during a streamer's streaming broadcast. At this time, the recommended music support unit 500 reproduces the type of music, music playback volume, and playback time differently depending on the situation and emotion according to the song selection made by the artificial intelligence data processing unit. For example, for intense emotions such as surprise or cheer, the playback volume can be set to high and played briefly, for sad emotions it can be played long at a low volume, and for tense scenes, it can be played briefly and repeatedly, etc. Different playback methods can be set and provided.

This method has the effect of further maximizing the effect of background music or sound effects on broadcasting.

The system for supporting automatically recommended music in streaming broadcasts described above can also be implemented in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

In addition, the system for automatically recommending music in streaming broadcasts described above can be executed by an application that is installed by default on the terminal (this may include programs included in the platform or operating system, etc. that are installed by default in the terminal), and the user can use the application. It may also be executed by an application (i.e. program) installed directly on the master terminal through an application providing server such as a store server, application, or web server related to the service. In this sense, the automatic recommendation music support system in the above-described streaming broadcast according to an embodiment of the present invention is implemented as an application (i.e., program) installed by default in the terminal or directly installed by the user, and is implemented as an application (i.e., program) installed by default in the terminal, etc. It may be recorded on a computer-readable recording medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (7)

In a system that automatically recommends and plays background music or sound effects while broadcasting to viewers in real time through online streaming,
A streaming broadcasting unit that transmits and receives chat data with the viewer terminal in real time, films the broadcast, and transmits the broadcast data;
an emotion recognition unit that recognizes emotions from broadcast data transmitted from the streaming broadcasting unit;
a data collection unit that collects emotion data recognized from the emotion recognition unit in the streaming broadcasting unit;
an artificial intelligence learning unit that learns recommended music from the emotional data collected by the data collection unit;
an artificial intelligence data processing unit that recommends music corresponding to the emotion recognized by the emotion recognition unit and updates learning results from the artificial intelligence learning unit in set units; and
An automatic recommended music support system for streaming broadcasting, including a recommended music support unit that plays music recommended by the artificial intelligence data processing unit.
delete According to claim 1,
The emotion recognition processing unit,
A streamer emotion recognition unit that recognizes the emotions of the streamer conducting the broadcast,
An automatic recommendation music support system for streaming broadcasting, comprising a chat data emotion recognition unit that recognizes the viewer's emotions from the viewer's chat data.
According to claim 3,
The streamer emotion recognition unit,
A face recognition unit that recognizes the face of the streamer,
Emotion recognition unit that recognizes emotions from the recognized facial data
An automatic recommendation music support system for streaming broadcasts, comprising:
According to claim 3,
The streamer emotion recognition unit,
An automatic recommendation music support system for streaming broadcasting, characterized in that it recognizes emotions from the streamer's voice waveform based on a pre-modeled emotion model.
According to claim 3,
The streamer emotion recognition unit,
An automatic recommendation music support system for streaming broadcasts, characterized by extracting emotions through clip leveling or analyzing emotions through context translation.
According to claim 1,
The artificial intelligence data processing unit,
An automatic recommendation music support system for streaming broadcasting, characterized by setting the type of music, music playback volume, and playback time differently depending on the situation and emotion.