KR102608935B1 - Method and apparatus for providing real-time audio mixing service based on user information - Google Patents

Abstract

The method of providing an audio mixing service is a method of providing an audio mixing service to a user terminal through a processor, and includes an information receiving step of receiving sound source information and mixing information about the sound source information, and is included in the sound source information based on the mixing information. An audio generation step of selecting an audio block corresponding to at least one stem item and combining audio information included in the selected audio block to generate one session audio, and the session It may include a mixing sound source reproduction step of generating a mixed sound source based on audio and playing the generated mixed sound source.

Description

Method and apparatus for providing real-time audio mixing service based on user information}

The present invention relates to a method and device for providing a real-time audio mixing service based on user information. More specifically, the present invention relates to a method and device for providing a real-time audio mixing service based on user information. More specifically, the present invention utilizes user information including information input from the user or information about the user's surrounding environment as mixing information, and accordingly, the system This invention relates to technology that selects and mixes data to provide real-time mixed audio.

With the advancement of technology, it has become possible to digitize existing large-capacity media and convert it into low-capacity media. Today, users can store various types of media on portable user terminal devices and easily select and enjoy the media they want on the go. It became possible. In addition, media digitized through digital compression technology is explosively revitalizing online media services by enabling media sharing between users on a network, and many applications or programs related to this are being developed.

Among the vast amounts of media provided, music accounts for a significant portion. Compared to other types of media, it has a lower capacity and communication load, so it is easy to support real-time streaming services, resulting in high satisfaction for both service providers and users. Accordingly, services that provide online music to users in various ways are currently emerging.

Existing online music services simply provide music in real time to users connected online by providing music to the user's terminal device or providing streaming services, but recently, they have utilized big data or artificial intelligence technology. We provide a service that recommends highly preferred media to users.

However, the current recommendation method in online music services is to create a music chart by simply counting the number of music sources purchased, listened to, or searched by the user and make recommendations based on this. This recommendation method is a statistical method based on the simple number of accesses. It is a method of recommending music based on standards and ignores the diversity and volatility of the user's preferences. In addition, since this recommendation method recommends music based on the accumulated number of accesses, the volatility of the music chart is low, and most of the previously recommended music overlaps with the currently recommended music, greatly reducing its effectiveness.

In addition, even if it is the same music, there are cases where users want to listen to various versions depending on their preferences. Currently, unless the music distribution company distributes the sound source in a different version, the user cannot listen to a version of the music with a different feel. I can't hear it.

Korean Patent Publication No. 10-2015-0084133 (published on July 22, 2015 - 'Pitch recognition using sound interference phenomenon and scale notation method using the same') Korean Patent No. 10-1696555 (2019.06.05.) - 'Text location search system and method through voice recognition in image or geographic information'

Therefore, the method and device for providing a real-time audio mixing service based on user information according to an embodiment are inventions designed to solve the problems described above, and mix sound sources according to the user's taste or mix sound sources based on the user's surrounding environment. The purpose is to provide a method and device for creating new mixing sound sources.

More specifically, an audio system that generates mixing information based on music characteristic information and music tag information input from the user and the user's surrounding environment information collected from sensors, and allows free mixing of audio using the generated mixing information. The purpose is to provide mixing services.

A method of providing an audio mixing service according to an embodiment is a method of providing an audio mixing service to a user terminal through a processor, comprising an information receiving step of receiving sound source information and mixing information related to the sound source information, and mixing information based on the mixing information. Audio that selects an audio block corresponding to at least one stem item included in the sound source information and combines the audio information included in the selected audio block to create one session audio. It may include a generation step and a mixing sound source reproduction step of generating a mixed sound source based on the session audio and playing the generated mixed sound source.

The mixing information may include at least one of music characteristic information and music tag information input by the user.

The music tag information may include at least one of genre information, mood information, and brightness information.

The music tag information may be input directly from the user, or may be information generated based on the user's voice information and conversation information with a chatbot.

The mixing information may include information generated based on information about the user's surrounding environment collected from sensors of the user terminal.

The user's surrounding environment information may include at least one of the user's location information, weather information, temperature information, and movement information.

The audio generating step includes generating a plurality of session audio, arranging the order of the plurality of session audio, and adjusting at least one of the volume, speed, composition, effect, and playback start point of the plurality of session audio. It may further include generating the plurality of session audios for which the steps and adjustments have been completed as one mixing sound source.

An audio mixing service providing device according to an embodiment includes a communication module that receives sound source information and mixing information about the sound source information, and at least one stem item preset for at least one audio version of the sound source information. a memory module in which corresponding audio block information is stored, a mixing information generation module that generates user information including at least one of information input from the user and information about the user's surroundings collected from sensors as the mixing information, and A mixing audio generation module that selects an audio block based on mixing information, combines the audio information contained in the selected audio block to create one session audio, and generates a mixing sound source based on the session audio. may include.

The mixing audio generation module generates a plurality of session audio, arranges the order of the plurality of session audio, and adjusts at least one of the volume, speed, composition, effect, and playback start point of the plurality of session audio, The plurality of session audios that have been adjusted can be created as one mixing sound source.

The user's surrounding environment information may include at least one of the user's location information, weather information, temperature information, and movement information.

A method and device for providing a real-time audio mixing service based on user information according to an embodiment not only actively mixes and produces audio of the user's desired taste, but also utilizes music tag information input by the user as mixing information. There is an advantage in providing an audio streaming service with increased user convenience.

The method and device for providing a real-time audio mixing service based on user information according to an embodiment have the advantage that the user can adjust the sound source even while the sound source is being played, and the adjusted mixed sound source can be transmitted in real time.

The method and device for providing a real-time audio mixing service based on user information according to an embodiment has the advantage of providing an audio streaming service that reflects the user's tastes in a more diverse manner by allowing the user to directly adjust various musical characteristics of the session audio. exist.

FIG. 1 is a diagram illustrating a partial configuration of a system for providing a real-time audio mixing service based on user information according to an embodiment of the disclosed invention.
FIG. 2 is a diagram illustrating a partial configuration of an apparatus for providing a real-time audio mixing service based on user information according to an embodiment of the disclosed invention.
Figure 3 is a diagram illustrating a method of providing a real-time audio mixing service based on user information according to an embodiment of the disclosed invention.
FIG. 4 illustrates a method of generating mixing information from music characteristic information input by a user and generating a mixing sound source based on the generated mixing information in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention. It is a drawing.
FIG. 5 illustrates a method in which music tag information input by a user is generated as mixing information and a mixing sound source is generated based on the generated mixing information in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention. It is a drawing.
FIG. 6 is a diagram illustrating a method in which information on the user's surrounding environment is generated as mixing information and a mixing sound source is generated based on the generated mixing information in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention. .
FIG. 7 is a diagram illustrating a method of generating a real-time mixing sound source by individually adjusting session audio in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention.
FIG. 8 is a diagram illustrating a multi-client management method of a real-time audio mixing service providing device according to an embodiment of the disclosed invention.
Figure 9 is a diagram illustrating an interface screen displayed on a user terminal when providing a mixing audio service actually implemented by applying the disclosed invention.

Hereinafter, embodiments according to the present invention will be described with reference to the attached drawings. When adding reference signs to components in each drawing, it should be noted that the same components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, when describing embodiments of the present invention, if detailed descriptions of related known configurations or functions are judged to impede understanding of the embodiments of the present invention, the detailed descriptions will be omitted. In addition, embodiments of the present invention will be described below, but the technical idea of the present invention is not limited or limited thereto and may be modified and implemented in various ways by those skilled in the art.

Additionally, the terms used in this specification are used to describe embodiments and are not intended to limit and/or limit the disclosed invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, terms such as “comprise,” “provide,” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. It does not exclude in advance the existence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

In addition, throughout the specification, when a part is said to be “connected” to another part, this refers not only to the case where it is “directly connected” but also to the case where it is “indirectly connected” with another element in between. Terms including ordinal numbers, such as “first” and “second,” used in this specification may be used to describe various components, but the components are not limited by the terms.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted.

Meanwhile, the title of the present invention is 'method and device for providing real-time audio mixing service based on user information', but for convenience of explanation below, 'device for providing real-time audio mixing service based on user information' is referred to as 'device for providing audio mixing service'. ', and 'method of providing real-time audio mixing service based on user information' will be abbreviated and explained as 'method of providing audio mixing service'.

FIG. 1 is a diagram illustrating a partial configuration of a system for providing a real-time audio mixing service based on user information according to an embodiment of the disclosed invention.

Referring to FIG. 1, an audio mixing service providing system according to an embodiment includes an audio mixing service providing device 200 that provides an audio mixing service to user terminals 300A, 300B, and 300C, and an audio mixing service providing device ( It may include user terminals (300A, 300B, 300C) that display the audio mixing interface received from 200) on the displays of the user terminals (300A, 300B, 300C).

As shown in the figure, the user terminals 300A, 300B, and 300C may include a plurality of user terminals 300A, 300B, and 300C.

The audio mixing service providing device 200 allows the user to customize audio stored in the user terminal (300A, 300B, 300C) or an external server (not shown) linked to the user terminal (300A, 300B, 300C) to the user's taste. An interface for mixing and editing can be created, the created interface can be provided to the user through the user terminals 300A, 300B, and 300C, user information can be input, and mixed audio can be generated based on the user information. Specific details regarding this will be described later.

The audio mixing service providing device 200 creates an audio mixing interface, transmits the generated audio mixing interface to the user terminals 300A, 300B, and 300C, and mixes the audio and mixes the generated audio to the user terminals 300A and 300B. , 300C) can be implemented as a server.

In the present invention, a server refers to a typical server. A server is computer hardware on which a program is running, and monitors or controls the entire network, such as printer control or file management, or other functions through a main frame or public network. It can support the sharing of software resources such as network connections, data, programs, and files, or hardware resources such as modems, faxes, shared printers, and other equipment. The user terminals 300A, 300B, and 300C provide the audio mixing service provided by the audio mixing service providing device 200 using a specific program or application installed on the user terminals 300A, 300B, and 300C. 300B, 300C) can be displayed on the display.

Meanwhile, in Figure 1, the audio mixing service providing device 200 is implemented as a server and is described based on receiving an interface that allows the user to mix and edit audio from the server. However, the audio mixing service providing device 200 according to the present invention ( The embodiment of the present invention is not limited to the fact that 200) is implemented as a server, and the audio mixing service providing device 200 may be implemented as user terminals 300A, 300B, and 300C.

When the audio mixing service providing device 200 is implemented as a user terminal (300A, 300B, 300C), the processor included in the user terminal (300A, 300B, 300C) directly generates the audio mixing interface screen and creates the interface. The screen may be displayed on the display of the user terminal (300A, 300B, 300C).

Specifically, the user terminals 300A, 300B, and 300C include a processor capable of generating a mixing interface screen provided by the audio mixing service providing device 200, and the processor generates an audio mixing interface screen and , the generated screen can be provided to the user through the display of the user terminal (300A, 300B, 300C). Therefore, the user inputs at least one of the music characteristic information edited to suit the user's taste for the audio to be mixed and music tag information related to the genre and mood of the sound source through the audio mixing interface screen, and provides the audio mixing service providing device 200. It can be sent to .

Accordingly, the user terminals 300A, 300B, and 300C may be implemented with several terminal devices including processors so that these algorithms can be realized. For example, as shown in FIG. 1, a personal computer (PC) 300A, a smart It can be implemented as a pad (300B) or a note book (300C). In addition, although not shown in the drawing, the user device 300 includes a Personal Digital Assistant (PDA) terminal, a Wireless Broadband Internet (Wibro) terminal, a smartphone, a tablet PC, a smart watch, and smart glasses. It can be implemented with all types of handheld-based wireless communication devices, such as glass and wearable devices.

FIG. 2 is a diagram illustrating a partial configuration of an apparatus 200 for providing a real-time audio mixing service based on user information according to an embodiment of the disclosed invention.

Referring to FIG. 2, the audio mixing service providing device 200 according to an embodiment includes a communication module 210, an audio mixing screen generation module 220, a mixing information generation module 230, and a mixing audio generation module 240. , may include a play list creation module 250 and a memory module 260.

Meanwhile, in FIG. 2, for convenience of explanation, a communication module 210, an audio mixing screen creation module 220, a mixing information creation module 230, a mixing audio creation module 240, and a play list creation module 250 are shown. Although each is indicated separately, the embodiment of the present invention is not limited to these independent configurations, and includes a communication module 210, an audio mixing screen generation module 220, a mixing information generation module 230, a mixing audio generation module 240, The play list creation module 250 may be implemented by consisting of one processing module that functions as a processor.

When the audio mixing service providing device 200 is implemented as a server-like device, the communication module 210 wirelessly communicates with the user terminals 300A, 300B, 300C and an external server (not shown) storing audio data, etc. Can be performed, audio mixing generated by the audio mixing screen generation module 220 and the mixing audio generation module 240 based on audio data received from at least one of the user terminal (300A, 300B, 300C) and an external server. The interface can be transmitted to user terminals (300A, 300B, 300C).

In addition, in another embodiment of the present invention, when the audio mixing service providing device 200 is implemented as a user terminal (300A, 300B, 300C), the communication module 210 of the audio mixing service providing device 200 allows the user to Data pre-stored on an external server or audio data pre-stored by a company operating an external server may be received, and the received audio data may be stored in the memory module 260.

The audio mixing screen creation module 220 can generate various screens or panels displayed on the display of the user terminal (300A, 300B, 300C), and display the generated screen on the display of the user terminal (300A, 300B, 300C). there is. In the present invention, a panel refers to a portion of an interface divided according to the nature of the content displayed on the display screen. Accordingly, a plurality of panels can be created depending on the nature of the content, and the plurality of generated panels can be displayed simultaneously on the display screen.

Additionally, the size of the panel can be automatically adjusted according to the number of panels created, and can be made smaller or larger depending on the user's manipulation.

The audio mixing screen generation module 220 according to the present invention can generate screens with different characteristics and display the generated screens on the displays of the user terminals 300A, 300B, and 300C.

Specifically, when audio to be mixed is executed by the user, the audio mixing screen creation module 220 loads at least one audio version pre-stored for the executed audio, and preset each of the loaded audio versions. An audio block corresponding to at least one stem item can be created, and an audio block screen including the generated audio block can be displayed on the display of the user device.

Audio here refers to music data that includes all the songs and accompaniments that we commonly listen to, and the stem classifies each audio track that makes up one piece of music, taking into account the sound range and function. This refers to data composed of one audio track.

Specifically, the sound source that makes up the audio is composed of human vocals and the sounds of various instruments combined to form a single result, and the stem here refers to data about a single item that makes up the sound source. For example, types of stems may include rhythm stems, bass stems, mid stems, high stems, FX stems, and melody stems.

In addition, if a selection block selected by the user exists among the audio blocks displayed on the audio mixing screen or a selection block selected by the mixing audio creation module 240 exists, the audio mixing screen creation module 220 selects the selected block as another audio. Blocks can be displayed in a different shade, checkmarks can be displayed only on selected blocks, and generated waveform information can be displayed on the audio mixing screen.

The mixing information generation module 230 may generate user information including at least one of information input from the user and information about the user's surrounding environment collected from a sensor as mixing information.

The mixing audio generation module 240 selects an audio block corresponding to at least one stem item included in the sound source information based on the mixing information generated by the mixing information generation module 230, The audio information contained in the selected audio block can be combined to create one session audio.

When the user's audio block selection is completed or the artificial intelligence server's audio block selection is completed based on user information, the mixing audio generation module 240 combines the audio information included in the selected audio block to create one session. ) can be created as audio.

Session audio as used in the present invention may refer to one part of one audio divided by a certain time unit. The standard for dividing session audio can be based on equal time units, but it can also be divided based on sections where the characteristics of the audio change by considering the average flow of the entire audio.

Accordingly, the standard for dividing session audio may be based on information preset and stored by the music producer, or may be freely changed based on the user's operation.

Additionally, reverberation may be placed between the session audios set in this way so that the music between the session audios can be connected naturally.

Additionally, the mixing audio generation module 240 may generate a plurality of session audios and generate the plurality of session audios as one mixing sound source. Mixing audio and mixing sound source described in the present invention have the same meaning.

Data related to the mixed audio or each session audio created in this way may be stored in the memory module 260.

The play list creation module 250 may create a list of mixed audios based on user information and then play the audio that exists in the list.

The playlist generated by the playlist creation module 250 may include mixed audio obtained by mixing audio blocks based on user information, and sound sources with similar characteristics to the mixed audio may be included in the playlist. The playlist creation module 250 may search for audio similar to an audio block containing a specific stem in the mixed audio mixed with the audio block based on user information and include it in the playlist, or may search for audio similar to the audio block containing a specific stem and include it in the playlist. Audio created by randomly mixing audio blocks can be included in the playlist. Detailed information regarding this will be described later.

The memory module 260 refers to a module in which data regarding sound sources and mixed sound sources previously stored by the user can be stored.

When the audio mixing service providing device 200 is implemented as a user terminal (300A, 300B, 300C), the memory module 260 is not included in the user terminal (300A, 300B, 300C) and is included in the memory module 260. Various data that can be stored can be stored on an external server. Therefore, in this case, the user terminals 300A, 300B, and 300C can receive data about various types of audio from an external server using the communication module 210.

Figure 3 is a diagram illustrating a method of providing a real-time audio mixing service based on user information according to an embodiment of the disclosed invention.

Referring to FIG. 3, the present invention includes an information reception step (S100) of receiving sound source information and mixing information about the sound source information, and based on the mixing information, an information corresponding to at least one stem item included in the sound source information is provided. An audio generation step (S110) in which an audio block is selected and the audio information contained in the selected audio block is combined to create one session audio, and a mixing sound source is created based on the session audio. It may include a mixing sound source reproduction step (S120) of playing the mixed sound source.

The audio generation step (S110) of the method for providing an audio mixing service according to an embodiment selects audio blocks included in sound source information based on user information, and combines the audio information included in the selected audio blocks to form one audio block. It can be created as session audio.

Additionally, step S110 may include generating a plurality of session audios by repeating the process of generating one session audio, and generating a mixed sound source by individually adjusting the characteristics of the plurality of session audios.

In addition, the present invention provides a play method that, when a mixing sound source is being played, considers the music characteristics of the mixing sound source, determines the similarity between the data stored in the memory module 260 and the mixed sound source being played, and selects the sound source to be played next. It may include a list creation step (S130).

In the playlist creation step (S130), a playlist may be created by selecting an audio block or session audio similar to the mixed audio generated through the audio block selected based on user information. This process of determining similarity and selecting a sound source to be played next may be performed by the play list creation module 250.

Hereinafter, the information reception step (S100) and the audio generation step (S110) of the method for providing real-time audio mixing services will be described in detail.

The audio mixing service providing device 200 according to an embodiment may receive sound source information and mixing information about the sound source information. At this time, the sound source information may be information included in the music characteristic information shown in FIG. 3.

In the information reception step (S100), in relation to the step where the audio mixing service providing device 200 receives sound source information, the user selects music to be played using the audio mixing interface displayed on the user terminals 300A, 300B, and 300C. By selecting , sound source information can be transmitted to the audio mixing service providing device 200.

Alternatively, when the user inputs voice information into the user terminal (300A, 300B, 300C), the processor of the user terminal (300A, 300B, 300C) processes the natural language included in the input voice information to select music to be used for playback. and the selected sound source information can be transmitted to the audio mixing service providing device 200.

Through this method, the audio mixing service providing device 200 can receive sound source information to be played in the user terminals 300A, 300B, and 300C from the user.

Thereafter, the audio mixing service providing device 200 may transmit playback sound source information to the user terminals 300A, 300B, and 300C so that audio corresponding to the input sound source information is played in the user terminals 300A, 300B, and 300C. . Playback sound source information input to the user terminals 300A, 300B, and 300C may be displayed on the audio mixing interface of the user terminals 300A, 300B, and 300C.

Next, in the information receiving step (S100), the audio mixing service providing device 200 receives mixing information. When the audio mixing service providing device 200 receives music characteristic information from the user, the input music Characteristic information can be created as mixing information.

Specifically, the user can mix audio by selecting audio blocks according to the user's taste using the audio mixing interface displayed on the user terminals 300A, 300B, and 300C. In other words, the user can input music characteristic information to the audio mixing service providing device 200 by turning on/off various audio blocks by clicking and turning on only the audio block that suits the user. Thereafter, the audio mixing service providing device 200 may generate mixing information based on music characteristic information input from the user.

Additionally, the audio mixing service providing device 200 may receive music tag information from the user and generate the input music tag information as mixing information.

Specifically, the music tag information input by the user may be music tag information directly selected by the user using the audio mixing interface displayed on the user terminals 300A, 300B, and 300C.

In other words, music tag information input from the user may be information input directly from the user. Accordingly, the user can selectively click on tag information displayed on the audio mixing interface and input music tag information suited to the user's taste into the audio mixing service providing device 200. Afterwards, the audio mixing service providing device 200 may generate mixing information using music tag information input from the user.

Alternatively, the music tag information input by the user may be information generated based on the user's voice information and conversation information with the chatbot.

Specifically, the user can input the user's voice information and conversation information with the chatbot into the user terminals 300A, 300B, and 300C using the audio mixing interface displayed on the user terminals 300A, 300B, and 300C. The user terminals (300A, 300B, 300C) transmit the input user's voice information and conversation information with the chatbot to the audio mixing service providing device 200, and the audio mixing service providing device 200 transmits the input user's voice information and conversation information with the chatbot to the user terminals (300A, 300B). , 300C), mixing information can be generated based on the user's voice information and conversation information with the chatbot.

As an example, the mixing information generation module 230 of the audio mixing service providing device 200 processes the user's voice information and conversation information with the chatbot, maps it to music tag information, and generates mixing information based on the mapped music tag information. can be created.

Additionally, the audio mixing service providing device 200 may generate mixing information using the user's surrounding environment information collected from the sensors of the user terminals 300A, 300B, and 300C.

Specifically, the user's surrounding environment information may include at least one of the user's location information, weather information, temperature information, and movement information. The audio mixing service providing device 200 may generate mixing information based on at least one of the user's location information, weather information, temperature information, and movement information.

As an example, the mixing information generation module 230 of the audio mixing service providing device 200 processes at least one of the user's location information, weather information, temperature information, and movement information, maps it to music tag information, and maps the mapped information to music tag information. Mixing information can be generated based on music tag information.

However, the user surrounding environment information is not limited to the above-described examples, and any information that can be collected from sensors attached to the user terminals 300A, 300B, and 300C may belong to the user surrounding environment information.

The audio mixing service providing device 200 according to an embodiment of the disclosed invention receives information in various ways as described above, and adds information to at least one stem item included in the sound source information based on the mixing information generated through this. You can select the corresponding audio block and combine the audio information contained in the selected audio block to create one session audio.

Therefore, the step of generating mixing information based on user information and selecting an audio block based on the mixing information in the method of providing an audio mixing service according to an embodiment of the disclosed invention involves receiving music characteristic information from the user and providing the information to the user. There is a method in which the audio block is directly selected (S111), or the audio block is automatically selected (S112) by applying artificial intelligence technology.

Additionally, the audio mixing service providing device 200 may generate a plurality of session audio belonging to different sections within one sound source based on the input mixing information and adjust the plurality of session audio (S113).

Specifically, the audio mixing service providing device 200 may arrange the order of a plurality of session audios. In addition, the audio mixing service providing device 200 can adjust at least one of the volume, speed, composition, effect, and playback start point of a plurality of session audios, and generate a plurality of adjusted session audios as one mixing sound source. . The process of arranging the order of the plurality of session audios and the process of adjusting the individual characteristics of the plurality of session audios may be performed directly by the user, or may be performed automatically by applying artificial intelligence technology. Details regarding this will be described later.

When adjustment is completed, the audio mixing service providing device 200 can complete audio generation by generating a mixing sound source based on the adjusted session audio.

When the audio mixing service providing device 200 completes the generation of the mixing sound source, it can output the generated mixing sound source. Specifically, the audio mixing service providing device 200 transmits information about the finally generated mixing sound source to the user terminals 300A, 300B, and 300C, and the user terminals 300A, 300B, and 300C play the mixing sound source. Completely mixed audio can be transmitted to the user.

FIG. 4 illustrates a method in which music characteristic information input by a user is generated as mixing information and a mixing sound source is generated based on the generated mixing information in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention. It is a drawing.

Referring to FIG. 4, the audio mixing service providing device 200 according to the present invention can generate one completed sound source by dividing it into several versions. Specifically, the audio mixing service providing device 200 of the present invention can generate a plurality of session audio in which a plurality of audio blocks corresponding to a plurality of stem items are divided and combined for each version of one sound source. there is.

Through this, users can create new mixed audio by recombining audio blocks to suit their taste.

Session audio described in the present invention refers to data in the form of a collection of various stem data constituting each sound source for several versions of the sound source created for one sound source.

Meanwhile, a completed sound source within the meaning of the present invention may be audio produced from the beginning in a block format, or may be audio that has already been released but produced in a block format with the consent of the composer.

The audio mixing service providing device 200 of the present invention can display a list of playable sound sources on the user terminals 300A, 300B, and 300C through the audio mixing screen creation module (S210).

Through this, the user can directly select a sound source to be played through the audio mixing interface displayed on the screen of the user terminal (300A, 300B, 300C). Therefore, the audio mixing service providing device 200 of the present invention can receive sound source information input by the user from the user terminal (S220).

Additionally, users can use an artificial intelligence-based music recommendation service by selecting the music tag for the mood they want to listen to and entering music tag information, or by entering voice information and conversation information using natural language or a chatbot.

The user's voice information and conversation information are information input from the user through the input unit of the user terminal 300, and the input information may be processed as text information in the user terminal 300. For example, the user's voice information is in response to the chatbot's question 'What BPM is good?' in response to the sentence 'Recommend music that is good to listen to while walking', 'Recommend music with a BPM of 100 or less.' It may be information containing the sentence answered '. In this case, the audio mixing service providing device 200 of the present invention may select a sound source to be played using data stored in the memory module based on at least one of music tag information, voice information, and conversation information input from the user. Accordingly, the audio mixing service providing device 200 of the present invention can receive selected sound source information using artificial intelligence (S220).

The user can input music characteristic information about the sound source played through the user terminals 300A, 300B, and 300C. In other words, the audio mixing service providing device 200 of the present invention can receive music characteristic information from the user (S230).

Specifically, the present invention can display a mixing interface screen on the user terminals 300A, 300B, and 300C, and audio blocks corresponding to stem data of the sound source currently being played can be displayed on the mixing interface screen. . Accordingly, the user terminals 300A, 300B, and 300C can generate music characteristic information based on the audio block selected by the user and transmit the generated music characteristic information to the audio mixing service providing device 200 of the present invention. .

The audio mixing service providing device 200 of the present invention can generate mixing information from music characteristic information input by the user and generate session audio based on the generated mixing information (S240). More specifically, the audio mixing service providing device 200 of the present invention can generate a plurality of session audio.

Session audio as used in the present invention refers to one part of one audio divided by a certain time unit. The standard for dividing session audio can be divided into equal time units, but considering the overall flow of audio, it can also be divided based on sections where audio characteristics change. The criteria for dividing sessions in this way are set and stored in advance by the music producer, or can be freely changed by user manipulation.

When the generation of session audio is completed, the audio mixing service providing device 200 of the present invention can output the mixing sound source, and thus the mixing sound source can be played in the user terminals 300A, 300B, and 300C.

Afterwards, the audio mixing service providing device 200 of the present invention can adjust the session audio according to the user's control (S250).

Specifically, the audio mixing service providing device 200 of the present invention arranges the order of a plurality of session audios and adjusts at least one of the volume, speed, composition, effect, and playback start point of the plurality of session audios to create a new mixing sound source. can be created.

When the adjustment of the session audio is completed, the audio mixing service providing device 200 of the present invention generates a plurality of adjusted session audio as one mixing sound source, transmits the generated mixing sound source to the user terminal, and accordingly, the user Terminals (300A, 300B, 300C) can play mixed sound sources in real time (S260). Details regarding this are described later in FIG. 7 .

FIG. 5 illustrates a method in which music tag information input by a user is generated as mixing information and a mixing sound source is generated based on the generated mixing information in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention. It is a drawing.

Referring to FIG. 5, the audio mixing service providing device 200 of the present invention can display a list of sound sources that can be played on the user terminals 300A, 300B, and 300C through the audio mixing screen creation module (S310).

Through this, the user can directly select a sound source to be played through the audio mixing interface displayed on the screen of the user terminal (300A, 300B, 300C). Therefore, through this method, the audio mixing service providing device 200 of the present invention can receive sound source information from the user (S320).

Additionally, users can use an artificial intelligence-based music recommendation service by selecting the music tag for the mood they want to listen to and entering music tag information, or by entering voice information and conversation information using natural language or a chatbot. In this case, the audio mixing service providing device 200 of the present invention can select a sound source to be played using data stored in the memory module based on at least one of music tag information, voice information, and conversation information received from the user terminal. Accordingly, the audio mixing service providing device 200 of the present invention can receive selected sound source information using artificial intelligence (S320).

The user can input music tag information for the sound source played through the user terminals 300A, 300B, and 300C. In other words, the audio mixing service providing device 200 of the present invention can receive music tag information from the user (S330).

Specifically, the present invention can display a mixing interface screen on the user terminals (300A, 300B, and 300C), and music tags including genre information, mood information, brightness information, etc. of the sound source can be displayed on the mixing interface screen. .

Accordingly, the user can generate music tag information by selecting a tag displayed on the screen of the user terminal (300A, 300B, 300C) and input the generated music tag information into the audio mixing service providing device 200 of the present invention.

Alternatively, when the user inputs voice information and conversation information with the chatbot into the audio mixing service providing device 200, the audio mixing service providing device 200 mixes based on the input user's voice information and conversation information with the chatbot. Information can be generated.

The user's voice information and conversation information are information input from the user through the input unit of the user terminal 300, and the input information may be processed as text information in the user terminal 300. For example, the user's voice information may include the sentence 'Recommend music that is good to listen to while walking,' and the conversation information with the chatbot may include the chatbot's response to the user's voice information, 'What is the BPM?' This may be information containing a sentence in which the user responds to the question 'Is this good?' by saying 'Recommend music with a BPM of 100 or less.'

Thereafter, the audio mixing service providing device 200 may analyze and process the input user's voice information and conversation information with the chatbot to map corresponding music tag information. Accordingly, the audio mixing service providing device 200 can generate music tag information through mapping.

For example, the audio mixing service providing device 200 may generate music tag information such as Lo-fi or Jazz based on the user's voice information requesting recommendation of good music to listen to while taking a walk.

Afterwards, the audio mixing service providing device 200 of the present invention generates music tag information generated based on voice information input from the user and conversation information with the chatbot as mixing information, and generates session audio based on the generated mixing information. Can be created (S340). More specifically, the audio mixing service providing device 200 of the present invention can generate a plurality of session audio.

Session audio as used in the present invention refers to one part of one audio divided by a certain time unit. The standard for dividing session audio can be divided into equal time units, but considering the overall flow of audio, it can also be divided based on sections where audio characteristics change. The criteria for dividing sessions in this way are set and stored in advance by the music producer, or can be freely changed by user manipulation.

When the generation of session audio is completed, the audio mixing service providing device 200 of the present invention can output the mixing sound source, and accordingly, the mixing sound source can be played in the user terminals 300A, 300B, and 300C.

Afterwards, the audio mixing service providing device 200 of the present invention can adjust the session audio according to the user's control (S350).

Specifically, the audio mixing service providing device 200 of the present invention arranges the order of a plurality of session audios and adjusts at least one of the volume, speed, composition, effect, and playback start point of the plurality of session audios to create a new mixing sound source. can be created.

When the adjustment of the session audio is completed, the audio mixing service providing device 200 of the present invention generates a plurality of adjusted session audio into one mixing sound source and sends the generated mixing sound source to the user terminals 300A, 300B, and 300C. It is transmitted to, and according to this, the user terminals (300A, 300B, 300C) can play the mixed sound source in real time (S360).

FIG. 6 is a diagram illustrating a method in which information on the user's surrounding environment is generated as mixing information and a mixing sound source is generated based on the generated mixing information in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention. .

Referring to FIG. 6, the audio mixing service providing device 200 of the present invention can display a list of sound sources that can be played on the user terminal (300A, 300B, 300C) on the user terminal through the audio mixing screen creation module (S410) ).

Through this, the user can directly select a sound source to be played through the audio mixing interface displayed on the screen of the user terminal (300A, 300B, 300C). Therefore, the audio mixing service providing device 200 of the present invention can receive sound source information from the user (S420).

Additionally, users can use an artificial intelligence-based music recommendation service by entering music tag information by selecting a music tag for the mood they want to listen to, or by entering voice information and conversation information using natural language or a chatbot. In this case, the audio mixing service providing device 200 of the present invention may select a sound source to be played using data stored in the memory module based on at least one of music tag information, voice information, and conversation information input from the user. Accordingly, the audio mixing service providing device 200 of the present invention can receive selected sound source information using artificial intelligence (S420).

Afterwards, the audio mixing service providing device 200 of the present invention can receive surrounding environment information collected from the sensor (S430).

Specifically, in the present invention, a button for using a mixing service based on the user's surrounding environment may be displayed on the mixing interface screen of the user terminals 300A, 300B, and 300C.

Therefore, the user can provide information on the user's surrounding environment to the audio mixing service providing device 200 of the present invention by selecting the surrounding environment-based mixing service button displayed on the screen of the user terminal (300A, 300B, 300C), and provide information on the user's surrounding environment to the audio mixing service providing device 200 of the present invention. Environmental information can be input to the audio mixing service providing device 200 of the present invention.

In addition, the audio mixing service providing device 200 of the present invention allows the user to directly input surrounding environment information into the audio mixing service providing device 200, as described above. It can also automatically receive information about the user's surrounding environment collected by sensors.

The user's surrounding environment information may include at least one of the user's location information, weather information, temperature information, and movement information.

When the audio mixing service providing device 200 of the present invention receives information about the user's surrounding environment through a sensor, the audio mixing service providing device 200 can generate mixing information based on the input information about the user's surrounding environment. there is.

More specifically, the audio mixing service providing device 200 may analyze and process the input user's surrounding environment information and generate corresponding music tag information through mapping. For example, when the weather in the user's surrounding environment is clear and the movement speed is slow, the audio mixing service providing device 200 maps corresponding music tag information such as Lo-fi and Jazz. can be created.

Through this, the audio mixing service providing device 200 can generate music tag information through mapping. Afterwards, the audio mixing service providing device 200 of the present invention generates mixing information from the surrounding environment information collected from the sensor and , Session audio can be created based on the generated mixing information (S440). More specifically, the audio mixing service providing device 200 of the present invention can generate a plurality of session audio.

Session audio as used in the present invention refers to one part of one audio divided by a certain time unit. The standard for dividing session audio can be divided into equal time units, but considering the overall flow of audio, it can also be divided based on sections where audio characteristics change. The criteria for dividing sessions in this way are set and stored in advance by the music producer, or can be freely changed by user manipulation.

When the generation of session audio is completed, the audio mixing service providing device 200 of the present invention can output the mixing sound source, and thus the mixing sound source can be played in the user terminals 300A, 300B, and 300C.

Afterwards, the audio mixing service providing device 200 of the present invention can adjust the session audio according to the user's control (S450).

Specifically, the audio mixing service providing device 200 of the present invention arranges the order of a plurality of session audios and adjusts at least one of the volume, speed, composition, effect, and playback start point of the plurality of session audios to create a new mixing sound source. can be created.

When the adjustment of the session audio is completed, the audio mixing service providing device 200 of the present invention generates a plurality of adjusted session audio into one mixing sound source and sends the generated mixing sound source to the user terminals 300A, 300B, and 300C. It is transmitted to, and according to this, the user terminals (300A, 300B, 300C) can play the mixed sound source in real time (S460).

FIG. 7 is a diagram illustrating a method of generating a real-time mixing sound source by individually adjusting session audio in a method of providing a real-time audio mixing service according to an embodiment of the disclosed invention.

Referring to FIG. 7, the user can change the playback start point, audio block, and arrangement order of session audio using the mixing interface displayed on the user terminals 300A, 300B, and 300C. As an example, as shown in FIG. 7, the user can change the playback order of the session by clicking on session audio containing at least one audio block and then moving it to a desired location.

Additionally, the user can click on the session audio to adjust at least one of the volume, speed, composition, and effects of the session audio.

Additionally, the user may select an audio block corresponding to at least one stem item included in the session audio and individually change the volume, speed, composition, and effect of the corresponding audio block.

Additionally, the real-time audio mixing service providing device 200 according to an embodiment can automatically adjust the playback start point, length, position, arrangement, etc. of session audio.

For example, the audio mixing service providing device 200 may automatically adjust the length of each of the plurality of session audios and the arrangement or position between the plurality of session audios based on the input mixing information. Therefore, the audio mixing service providing device 200 according to the present invention can provide the user with various mixing sound sources that meet the user's needs even without user input.

In the process of adjusting the session audio according to the user's session audio control and the session audio control of the audio mixing service providing device, the present invention provides real-time audio streaming to the user terminals (300A, 300B, 300C) based on WebRTC (Web Real-Time Communication) technology. Mixed sound sources can be transmitted.

Web RTC is a technology that allows web applications and sites to exchange arbitrary data by capturing audio or video media between browsers without an intermediary and streaming it at will. Data is exchanged through a P2P connection between web browsers without installing a separate driver or plug-in. It is a technology that allows exchange to take place.

WebRTC has a short latency, so it can provide near real-time services with almost no delay. This WebRTC can be mainly used in media transmission and reception services where real-time information processing is important.

Therefore, in the case of the conventional music streaming service that uses the mixdown method to stream multiple tracks simultaneously, there is a disadvantage that it is difficult to transmit the mixed sound source in real time to the user terminal (300A, 300B, 300C) even if the mixing information is used to create the mixed sound source. In the case of the present invention, it is possible to provide a service that can mix and play multiple audios that interactively change every hour in real time by utilizing WebRTC technology, which is designed to enable communication between web browsers without the help of plug-ins.

Accordingly, the audio mixing service providing device 200 according to the present invention adjusts at least one of the volume, speed, composition, effect, and playback start point for each session audio, or adjusts the audio block corresponding to each stem constituting the session audio. By adjusting at least one of the volume, speed, composition, effect, and playback start point, session audio can be customized in real time to suit the user's needs, and the mixed sound source created by automatically adjusting the session audio can be recommended to the user. .

Sound sources including a plurality of session audios adjusted by the user or the mixing audio service providing device 200 may be mixed down into one sound source and a global effect may be applied to ultimately create a mixed sound source.

FIG. 8 is a diagram illustrating a multi-client management method of the real-time audio mixing service providing device 200 according to an embodiment of the disclosed invention.

Referring to FIG. 8, the audio mixing service providing device 200 of the present invention may include a plurality of mixing servers so that the mixing servers can be matched for each user.

Through this, when a user uses an audio mixing service, the final generated mixed sound source can be stored in a mixing server that corresponds 1:1 with the user. Therefore, the present invention can utilize mixing sound source data created by a user in the past as basic data when providing an audio mixing service.

The mixing server shown in FIG. 8 may correspond to the memory module of the audio mixing service providing device 200 of the present invention, but is not limited to this and may be an external server provided separately from the audio mixing service providing device 200.

Figure 9 is a diagram illustrating an interface screen displayed on a user terminal when providing a mixing audio service actually implemented by applying the disclosed invention.

Referring to FIG. 9, when a sound source is played by a streaming service based on sound source information, a cover image of the corresponding audio may be displayed on the interface screen, as shown in the left screen of FIG. 9. In this state, when the user clicks on the audio cover, the audio block screen containing audio block information in the current section of the audio being played appears on the interface screen of the user terminals 300A, 300B, and 300C, as shown in the right screen of FIG. 9. It can be displayed in .

Through this, the user can intuitively understand the currently playing audio block information, and if the user wants to change and select an audio block, he or she can select another audio block displayed on the screen and send the changed music characteristic information to the audio mixing service providing device (200). ) can be entered.

Additionally, as shown on the left side of FIG. 9, the user can select various music tags displayed on the interface screen and input music tag information used for mixing into the audio mixing service providing device 200.

In addition, as shown on the left side of FIG. 9, when the user selects AI Mix on the interface screen, the audio mixing service providing device 200 generates mixing information based on the user's surrounding environment information, and the generated mixing information By selecting the corresponding audio block, an artificial intelligence-based mixing service can be provided.

So far, we have looked in detail at the method and device for providing a real-time audio mixing service based on user information according to the present invention.

A method and device for providing a real-time audio mixing service based on user information according to an embodiment not only actively mixes and produces audio of the user's desired taste, but also utilizes music tag information input by the user as mixing information. There is an advantage in providing an audio streaming service with increased user convenience.

The method and device for providing a real-time audio mixing service based on user information according to an embodiment have the advantage that the user can adjust the sound source even while the sound source is being played, and the adjusted mixed sound source can be transmitted in real time.

The method and device for providing a real-time audio mixing service based on user information according to an embodiment has the advantage of providing an audio streaming service that reflects the user's tastes in a more diverse manner by allowing the user to directly adjust various musical characteristics of the session audio. exist.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may perform an operating system (OS) and one or more software applications that run on the operating system.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent. Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

200; Audio mixing service provider
300A, 300B, 300C; user terminal
210; communication module
220; Audio mixing screen creation module
230; Mixing information generation module
240; Mixing Audio Generation Module
250; Playlist creation module
260; memory module

Claims (10)

In a method of providing an audio mixing service to a user terminal through a processor,
An information receiving step of receiving sound source information and mixing information about the sound source information;
Based on the mixing information, an audio block corresponding to at least one stem item included in the sound source information is selected, and the audio information included in the selected audio block is combined to create one session ( session) An audio generation step that generates audio; and
A mixing sound source reproduction step of generating a mixed sound source based on the session audio and playing the generated mixed sound source,
The mixing information is
A method of providing an audio mixing service, including information generated based on information about the user's surrounding environment collected from sensors of the user terminal. According to paragraph 1,
The mixing information is
A method of providing an audio mixing service, including at least one of music characteristic information and music tag information input by a user. According to paragraph 2,
The music tag information is
A method of providing an audio mixing service including at least one of genre information, mood information, and brightness information. According to paragraph 2,
The music tag information is
A method of providing an audio mixing service, which is information received directly from the user or generated based on the user's voice information and conversation information with a chatbot. delete In paragraph 1
The user's surrounding environment information is,
A method of providing an audio mixing service including at least one of the user's location information, weather information, temperature information, and movement information. According to paragraph 1,
The audio generation step is,
generating a plurality of the session audio;
arranging the order of the plurality of session audios;
adjusting at least one of volume, speed, composition, effect, and playback start point of the plurality of session audios; and
A method of providing an audio mixing service further comprising: generating the plurality of adjusted session audio into one mixing sound source. A communication module that receives sound source information and mixing information about the sound source information;
a memory module storing audio block information corresponding to at least one stem item preset for at least one audio version of the sound source information;
a mixing information generation module that generates user information including at least one of information input from the user and information about the user's surrounding environment collected from a sensor as the mixing information; and
Mixing audio generation that selects an audio block based on the mixing information, combines the audio information contained in the selected audio block to create one session audio, and generates a mixing sound source based on the session audio. Contains a module;
The mixing information is
An audio mixing service providing device including information generated based on information about the user's surrounding environment collected from sensors of the user's terminal. According to clause 8,
The mixing audio generation module is
After generating a plurality of session audios, arranging the order of the plurality of session audios, adjusting at least one of the volume, speed, composition, effect, and playback start point of the plurality of session audios, and adjusting the plurality of session audios for which the adjustment has been completed. An audio mixing service provider that generates session audio into a single mixing sound source. delete