JP7442979B2 - karaoke system - Google Patents

Description

The present invention relates to a karaoke system.

Services and platforms (hereinafter referred to as live streaming services) that allow anyone to easily perform live streaming using user terminals such as smartphones are becoming widespread. Among such live distribution services, there are also those that can use sound sources for karaoke (see Non-Patent Documents 1 and 2; hereinafter referred to as karaoke distribution compatible services).

A user of a karaoke distribution compatible service performs karaoke singing along with the performance sound emitted from his or her user terminal, and photographs himself or herself singing karaoke using his or her user terminal. The user terminal generates a singing video using the captured video and transmits it to the server device by streaming. On the other hand, other users can freely view the streaming singing video by accessing the server device using their own user terminals.

At this time, the user who performed the karaoke singing can receive comments and gifts (such as virtual currency that can be used within the karaoke distribution compatible service) from the users who watched the singing video.

Incidentally, in live concerts by artists and various events, audience participation type production methods are sometimes adopted. For example, a large number of audience members can individually move their portable light-emitting devices (e.g., penlights or wristwatch-shaped light-emitting devices) in time with the performance or production of a song, increasing the performance effect and creating a sense of unity between the artist and the audience. be.

Patent Document 1 discloses a light-emitting device for effect that uses a penlight that can emit light in multiple colors. The light emitting device for performance disclosed in Patent Document 1 controls the emitted light color of a plurality of portable light emitting devices (penlights) by one wirelessly connected light emitting state control device, so that arbitrary light can be generated in the audience seats. It is possible to express characters, images, and patterns.

JP2003-036981A

"SHOWROOM", Daiichi Kosho Co., Ltd., SHOWROOM Co., Ltd., [searched on May 6, 2021], Internet, <URL: https://prtimes.jp/main/html/rd/p/000000434.000014607. html＞

"LINE LIVE", Xing Co., Ltd., LINE Co., Ltd., [Searched on May 6, 2021], Internet, <URL: https://prtimes.jp/main/html/rd/p/000001318.000001594.html>

An object of the present invention is to provide a karaoke system that can give viewers of live-streamed singing videos a feeling similar to the sense of unity between an artist and audience at a live concert.

One invention for achieving the above object is a karaoke system including a plurality of user terminals and a server device capable of communicating with the user terminals, wherein a singer who performs karaoke singing among the user terminals The singer terminal operated by the singer terminal includes a performance processing section that performs karaoke performance based on the performance data of the song, the performance sound of the karaoke performance performed by the performance processing section, and the singing voice of the karaoke singing collected by the sound collection means. and a transfer unit that generates a singing video based on a video obtained by photographing the singer performing karaoke singing using a photographing means, and transmits the video by streaming to the server device, Among them, the plurality of viewer terminals operated by the plurality of viewers who view the singing video include a display control unit that displays the singing video live-distributed by the server device on a display means, and a display control unit that detects the movement of the viewer terminal. and a detection unit that transmits motion data to the server device, and the server device includes at least one sensor that simulates the movement of a penlight that is actually used, based on each of the plurality of received motion data. a generation unit that generates one penlight video; a distribution unit that combines the plurality of generated penlight videos with respect to the singing video streamed by the singer terminal and delivers it live to the viewer terminal; This is a karaoke system with
Other features of the present invention will become clear from the description and drawings described below.

According to the present invention, it is possible to provide viewers of a live singing video with a feeling similar to the sense of unity between an artist and audience at a live concert.

1 is a diagram schematically showing a karaoke system according to an embodiment. It is a figure showing a singer terminal concerning an embodiment. FIG. 2 is a diagram showing a viewer terminal according to an embodiment. FIG. 1 is a diagram showing a server device according to an embodiment. It is a flow chart showing processing of a karaoke system concerning an embodiment.

A karaoke system according to a first embodiment will be described with reference to FIGS. 1 to 5.

==Karaoke system==
Karaoke system 1 is a system for live streaming singing videos. A singing video is a video obtained by photographing a singer performing karaoke singing, and synthesizes the performance sound and singing voice of the karaoke performance. As shown in FIG. 1, the karaoke system 1 according to this embodiment includes a plurality of user terminals and a server device S.

The user terminal is a smartphone or a personal computer owned by a user of the karaoke system 1. Dedicated application software (hereinafter referred to as "live distribution application") for using the karaoke system 1 is installed on the user terminal according to the present embodiment. The live distribution application can be obtained by downloading from the server device S or a website provided by the server device S.

The live distribution application has a mode (generation mode) for generating a singing video for performing live distribution, and a mode (viewing mode) for viewing the live distributed singing video.

In the generation mode, a karaoke performance can be performed based on the performance data for karaoke performance received from the server device S. Further, the singing videos generated in the generation mode can be sequentially streamed and transmitted to the server device S. That is, by executing the live distribution application in this embodiment, the user can receive a karaoke distribution compatible service. Details of the generation mode and viewing mode will be described later.

In the example of FIG. 1, user U1 photographs his own karaoke singing using user terminal M1 that he owns. The user terminal M1 generates a singing video based on karaoke singing and transmits it to the server device S by streaming. Further, an unspecified number of users U2 to Un use their own user terminals M2 to Mn to view the singing videos live distributed by the server device S. That is, the user U1 corresponds to a "singer", and the users U2 to Un correspond to "viewers". Further, the user terminal M1 corresponds to a "singer terminal", and the user terminals M2 to Mn correspond to "viewer terminals". Among the user terminals, the singer terminal is a terminal operated by a singer who performs karaoke singing. The viewer terminal is a terminal among user terminals that is operated by a viewer who views a singing video.

The server device S is a computer that manages various information and delivers live singing videos to each user terminal.

==Singer terminal==
With reference to FIG. 2, the configuration of the singer terminal will be described. Hereinafter, the user terminal M1 in FIG. 1 will be described as an example of the singer terminal. As shown in FIG. 2, the user terminal M1 includes a storage means 10, a communication means 11, a display means 12, an input means 13, a sound collection means 14, a photographing means 15, a sound emitting means 16, and a control means 17. Each component is connected to bus B via an interface (not shown).

[Storage means, communication means, display means, input means, sound collection means, photographing means, sound emission means]
The storage means 10 is a large capacity storage device that stores various data. The storage means 10 stores song names, song identification information, and terminal identification information of songs that can be performed in karaoke using the live distribution application. The song identification information is information unique to each song, such as a song ID for identifying the song. The terminal identification information is information unique to each user terminal, such as a terminal ID for identifying the user terminal.

The communication means 11 provides an interface for connection with the server device S. The display means 12 is a display that displays various information. The input means 13 is configured for the user to input various instructions. Note that in a typical smartphone, the display means 12 is configured in the form of a touch panel, and functions as the input means 13.

The sound collecting means 14 of the user terminal M1 is a microphone for collecting the singing voice of the singer (user U1). The photographing means 15 is a camera installed in the user terminal. The photographing means 15 of the user terminal M1 photographs a singer (user U1) performing karaoke singing. The sound emitting means 16 is an earphone or the like for emitting sound.

Normally, when using a karaoke distribution compatible service, the singer fixes the user terminal on a stand, etc., and sings with a wireless microphone while listening to the karaoke performance sound with wireless earphones at a slightly distant location. The photographing means 15 is caused to photograph a moving image. The singing voice collected by the wireless microphone is emitted from the wireless earphones along with the karaoke performance sound. It is also possible to add reverberation effects such as echo to the singing voice. Singing sounds and the like can be obtained in the same manner in the karaoke distribution compatible service in this embodiment. That is, the sound collecting means 14 includes a separate member such as a wireless microphone that can be used in the user terminal M1. Further, the sound emitting means 16 includes another member such as a wireless earphone that can be used in the user terminal M1.

[Control means]
The control means 17 performs various controls on the user terminal M1. The control means 17 includes a CPU and a memory (both not shown). The CPU implements various functions by executing programs stored in memory.

When the live distribution application is started and the generation mode is selected (when the user desires to use a karaoke distribution compatible service), the control means 17 functions as a performance processing section 17a and a transfer section 17b.

(Performance processing section)
The performance processing section 17a performs karaoke performance based on the performance data of the music. Specifically, the performance processing section 17a generates a performance sound based on the performance data, and causes the sound emitting means 16 to emit the sound.

The performance data can be obtained from the server device S, for example. As a specific example, it is assumed that the user U1 selects a song X for which he desires live distribution of karaoke singing via the user terminal M1. In this case, the performance processing unit 17a transmits the song ID of the song X to the server device S. The server device S reads the performance data of the song X based on the song ID received from the user terminal M1, and transmits it to the user terminal M1. The performance processing unit 17a generates performance sounds based on the received performance data of the song X. Note that the storage means 10 may store performance data of a plurality of songs in advance. In this case, the performance processing unit 17a reads the performance data of the music piece selected by the user U1, and generates the performance sound based on the performance data.

User U1 performs karaoke singing along with the performance sound. The sound collecting means 14 collects the singing voice of the user U1 singing at karaoke. Further, the photographing means 15 photographs the figure (specifically, the face and upper body) of the user U1 at least while the user U1 is singing karaoke. That is, the user U1 takes a self-portrait of a moving image.

Note that, in general, for songs that can be sung at karaoke, lyrics caption data for displaying the lyrics of the song is prepared together with the performance data. The performance processing section 17a can generate lyrics based on the lyrics telop data and display the lyrics on the display means 12 in synchronization with the performance sound. Lyrics telop data can be obtained in the same manner as performance data.

(transfer section)
The transfer unit 17b obtains the performance sound of the karaoke performance performed by the performance processing unit 17a, the singing voice of the karaoke singing collected by the sound collecting means 14, and the photographing of the singer performing the karaoke singing by the photographing means 15. A singing video is generated based on the video image and streamed to the server device S.

The transfer unit 17b synthesizes the performance sound, the singing voice, and the moving image of the singer to generate a singing moving image. A known method can be used to synthesize each sound and a moving image. The transfer unit 17b sequentially transmits the generated singing videos to the server device S via the communication means 11 by streaming.

==Viewer terminal==
Next, the configuration of the viewer terminal will be described with reference to FIG. 3. Hereinafter, the user terminal M2 in FIG. 1 will be described as an example of the viewer terminal. Note that the user terminals M3 to Mn also have similar configurations.

As shown in FIG. 3, the hardware configuration of the user terminal M2 is similar to that of the user terminal M1. On the other hand, when the live distribution application is started and the viewing mode is selected, the control means 17 functions as a display control section 17c, a detection section 17d, and a setting section 17e.

(Display control section)
The display control unit 17c causes the display unit 12 to display the singing video live-distributed by the server device S.

For example, after selecting the viewing mode, the user U2 selects a singing video that he/she wishes to view on the search screen displayed on the display means 12. When the user U1 selects the singing video being streamed, the display control unit 17c accesses the server device S, receives the live streaming of the singing video being streamed by the user U1, and displays it on the display means 12. . Therefore, the user U2 can view the singing video that the user U1 is currently streaming. Moreover, the user U2 who views the singing video can input a comment using the input means 13, or can provide a gift to the user U1.

(Detection unit)
The detection unit 17d detects the movement of the viewer terminal and transmits it to the server device S as motion data.

Detection of the movement of the viewer terminal can be performed by a known technique using an acceleration sensor (not shown) built into the viewer terminal.

For example, when a live distribution application is started and a viewing mode is selected, a selection screen for selecting whether to synthesize a penlight video is displayed on the display unit 12. After the user U2 selects "to synthesize" via the input means 13, the user U2 repeatedly shakes the user terminal M2 left and right and up and down in accordance with the karaoke singing of the singing video.

In this case, the detection unit 17d detects the movement of the user terminal M2 and transmits it to the server device S as motion data MD.

Note that the action of shaking the viewer terminal may be performed only for a certain period of time (for example, 5 seconds or 2 bars of the song being played at karaoke). After that, the viewer can watch the singing video on the viewer terminal.

(setting section)
The setting unit 17e sets light emission information for determining the light emission mode of the penlight image.

The penlight video is a video generated by the server device S (details will be described later). The light emission mode includes the light emission color in the penlight image (color of the penlight image), brightness, lighting pattern (always on, blinking, etc.).

For example, when a live distribution application is started and a viewing mode is selected, a light emission mode selection screen is displayed on the display means 12. The user U2 selects an arbitrary light emission mode (e.g., light emission color: red) from among the plurality of light emission modes via the input means 13. The setting unit 17e sets the selected light emission mode as light emission information corresponding to the motion data MD of the user terminal M2.

The detection unit 17d transmits the set light emission information to the server device S together with the motion data MD.

==Server device==
FIG. 4 is a diagram showing an example of the hardware configuration of the server device S. As shown in FIG. The server device S includes a storage means 20, a communication means 21, and a control means 22. Each component is connected to bus B via an interface (not shown).

The storage means 20 is a large capacity storage device that stores various data. For example, the storage means 20 stores music data regarding a plurality of songs for performing karaoke. The song data is given song identification information. The music data includes performance data, reference data, lyrics telop data, and the like. The communication means 21 provides an interface for connecting the server device S and each user terminal.

The control means 22 performs various controls in the server device S. As shown in FIG. 4, the control means 22 according to this embodiment functions as a generation section 22a and a distribution section 22b.

(Generation part)
The generation unit 22a generates at least one penlight image based on the received motion data. Furthermore, in the present embodiment, the generation unit 22a sets the light emission mode of the plurality of penlight images based on the received light emission information.

The penlight video is a video that simulates the movement of a penlight that is actually used.

For example, assume that motion data MD is received from the user terminal M2. In this case, the generation unit 22a generates an image in which the penlight image moves with the movement indicated by the motion data MD. A known CG technique can be used to generate the penlight image.

Further, it is assumed that the generation unit 22a receives the light emission information "light emission color: red" together with the motion data MD. In this case, the generation unit 22a generates an image in which the penlight moves while emitting red light.

Note that the generation unit 22a may generate a plurality of penlight images based on one received motion data. In this case, the plurality of penlight images generated based on one motion data are all the same image.

(Distribution Department)
The distribution unit 22b synthesizes the generated penlight video with the singing video streamed by the singer terminal and delivers it live to the viewer terminal.

A publicly known method can be used to live distribute the singing video. For example, as described in the above example, it is assumed that the user U2 selects a singing video that is being streamed and transmitted from the user terminal M1 of the user U1. In this case, the distribution unit 22b performs live distribution of the singing video being streamed by the user terminal M1 to the user terminal M2 owned by the user U2.

When the user U2 selects the combination of penlight images, the generation unit 22a generates a penlight image. The distribution unit 22b combines the singing video streamed by the user terminal M1 with the penlight video generated by the generation unit 22a, and performs live distribution to the user terminal M2. The display control unit 17c of the user terminal M2 causes the display unit 12 to display the singing video in which the penlight video has been synthesized.

Note that when a plurality of users U2 to Un are viewing the singing video of the user U1, a plurality of motion data may be transmitted from a plurality of user terminals M2 to Mn. In this case, the generation unit 22a generates a plurality of penlight images based on each of the plurality of motion data. Further, the distribution unit 22b synthesizes the plurality of generated penlight videos into a singing video and live-distributes the video to each viewer terminal. Conversely, the generation unit 22a may generate the penlight video using only some (however, two or more) of the plurality of motion data. Further, the distribution unit 22b may synthesize only a portion of the plurality of generated penlight videos into a singing video and live-distribute the video.

==About processing in the karaoke system==
Next, processing in the karaoke system 1 according to this embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing processing in the karaoke system 1. In this example, an example will be described in which a singing video streamed from a singer terminal is viewed on one viewer terminal.

The performance processing unit 17a of the singer terminal performs karaoke performance based on the performance data of the song (starts karaoke performance. Step 10). The sound collecting means 14 collects singing voices of karaoke singing. Further, the photographing means 15 photographs a singer performing karaoke singing to obtain a moving image.

The transfer unit 17b generates a singing video based on the performance sound of the karaoke performance, the singing voice, and the moving image, and transmits the singing video by streaming to the server device S (streaming transmits the singing video. Step 11).

The distribution unit 22b of the server device S live-distributes the singing video being streamed by the singer terminal to the viewer terminal (live-distributes the singing video. Step 12).

The display control unit 17c of the viewer terminal causes the display means 12 to display the singing video live-distributed by the server device S (displays the singing video. Step 13). Viewers can watch videos of singers singing.

Here, the viewer selects an arbitrary light emission mode on the light emission mode selection screen displayed on the display means 12 of the viewer terminal. The setting unit 17e sets the selected light emission mode as light emission information on the viewer terminal (setting of light emission information. Step 14).

After that, the viewer shakes the viewer terminal in accordance with the performance sound of the singing video. In this case, the detection unit 17d detects the movement of the viewer terminal, and transmits the motion data and the light emission information set in step 14 together to the server device S (transmission of motion data and light emission information. Step 15).

The generation unit 22a of the server device S generates a penlight image that simulates the movement of a penlight that is actually used, based on the received motion data MD and light emission information (generates a penlight image. Step 16).

The distribution unit 22b combines the penlight video generated in step 16 with the singing video streamed by the singer terminal and delivers it live to the viewer terminal (synthesizes the penlight video and delivers it live. Step 17) .

The display control unit 17c of the viewer terminal causes the display unit 12 to display the singing video in which the penlight video that is live-distributed by the server device S is synthesized (displays the singing video in which the penlight video is synthesized. Step 18).

As is clear from the above, the karaoke system 1 according to the present embodiment includes a plurality of user terminals and a server device S that can communicate with the user terminals. Among the user terminals, the singer terminal operated by the singer who performs karaoke singing includes a performance processing unit 17a that performs karaoke performance based on the performance data of the song, a performance sound of the karaoke performance performed by the performance processing unit 17a, A singing video is generated based on the singing voice of karaoke singing collected by the sound collecting means 14 and a moving image obtained by photographing the singer performing karaoke singing by the photographing means 15, and is streamed to the server device S. and a transfer unit 17b. Among the user terminals, the plurality of viewer terminals operated by the plurality of viewers who view the singing video have a display control unit 17c that causes the display means 12 to display the singing video that is live-distributed by the server device S, and a viewer terminal. and a detection unit 17d that detects the movement of and transmits it to the server device S as motion data. The server device S includes a generation unit 22a that generates at least one video of a penlight that simulates the movement of a penlight that is actually used based on each of the plurality of received motion data, and a singer terminal that performs streaming transmission. It has a distribution unit 22b that combines a plurality of generated penlight videos with the singing video and delivers the composite live to the viewer terminal.

According to the karaoke system 1, a plurality of light penlight videos are generated based on motion data corresponding to the movements of a plurality of viewer terminals, and the plurality of light penlight videos are combined into a singing video and displayed. can be done. In this way, viewers who watch a singing video that simulates the performance using a penlight can have the same feeling as actually waving a penlight at a concert venue. That is, according to the karaoke system 1 according to the present embodiment, viewers of live-streamed singing videos are provided with a feeling similar to the sense of unity between the artist and the audience at a live concert (i.e., a sense of unity with the singer). can be given.

Furthermore, the viewer terminal has a setting unit 17e that sets light emission information for determining the light emission mode of the penlight image, and the detection unit 17d sends the set light emission information to the server device S together with the motion data. The generating unit 22a may set the light emitting mode of the penlight image based on the received light emitting information. According to the karaoke system 1 as described above, it is possible to generate a penlight image according to the light emission mode desired by the viewer.

Note that the setting section 17e is not an essential component. For example, it is also possible to store a plurality of pieces of light emission information in advance on the server device S side. In this case, the generation unit 22a selects one light emission information when generating the penlight image. Then, the generation unit 22a determines the light emission mode of the penlight image based on the selected light emission information.

Further, the generation unit 22a may generate a plurality of penlight images based on one received motion data. According to such karaoke system 1, a plurality of penlight images generated based on one motion data can be displayed. Therefore, even when the number of viewers is small, it is possible to simulate the appearance of many penlights being waved.

<Others>
In the above embodiment, an example has been described in which the viewer shakes the viewer terminal while viewing a live-distributed singing video. In this case, while the viewer is shaking the viewer terminal, only the audio of the singing video can be heard. On the other hand, it is also possible to use predetermined motion data stored in the storage means 10 in advance. In this case, the detection unit 17d reads predetermined motion data stored in advance in the storage means 10, and transmits it to the server device S as motion data corresponding to the movement of the viewer terminal. By using motion data stored in advance in this way, the viewer can concentrate on watching the singing video.

As the predetermined motion data, for example, one pattern from several predetermined patterns can be adopted. At this time, the adopted pattern may be corrected based on tempo information of the song to be performed in karaoke, and may be used as predetermined motion data.

Alternatively, the motion data acquired by the detection unit 17d can be stored in the storage means 10 in advance, and the motion data can be employed as the predetermined motion data. In this case, it is possible to display a moving penlight image that matches the viewer's preference.

Further, the distribution unit 22b may live-distribute the singing video synthesized with the penlight video to the singer terminal. In this case, the singer can view the singing video in which the video of the penlight has been synthesized, so they can experience a feeling similar to the sense of unity between the artist and the audience at a live concert (i.e., a sense of unity with the audience). I can do it.

The above embodiments are presented as examples and do not limit the scope of the invention. The above configurations can be implemented in appropriate combinations, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. The above-described embodiments and their modifications are included within the scope and gist of the invention as well as within the scope of the invention described in the claims and its equivalents.

1 Karaoke system 17a Performance processing unit 17b Transfer unit 17c Display control unit 17d Detection unit 17e Setting unit 22a Generation unit 22b Distribution unit M1 User terminal (singer terminal)
M2 User terminal (viewer terminal)
S server device

Claims (5)

A karaoke system including a plurality of user terminals and a server device capable of communicating with the user terminals,
Among the user terminals, the singer terminal operated by the singer who performs karaoke singing is
a performance processing unit that performs karaoke performance based on the performance data of the song;
Based on the performance sound of the karaoke performance performed by the performance processing section, the singing voice of karaoke singing collected by the sound collecting means, and the moving image obtained by photographing the singer performing karaoke singing by the photographing means. a transfer unit that generates a singing video and streams it to the server device;
has
Among the user terminals, a plurality of viewer terminals operated by a plurality of viewers who watch the singing video are:
a display control unit that causes a display unit to display a singing video that is live distributed by the server device;
a detection unit that detects a movement of the viewer terminal and transmits it as motion data to the server device;
has
The server device includes:
a generation unit that generates at least one penlight image simulating the movement of a penlight that is actually used, based on each of the plurality of received motion data;
a distribution unit that combines the plurality of generated penlight videos with the singing video streamed by the singer terminal and delivers the result live to the viewer terminal;
A karaoke system with The viewer terminal has a setting unit that sets light emission information for determining a light emission mode of the penlight image,
The detection unit transmits the set light emission information together with the motion data to the server device,
The karaoke system according to claim 1, wherein the generation unit sets a light emission mode of the penlight image based on the received light emission information. The karaoke system according to claim 1 or 2, wherein the generation unit generates a plurality of penlight images based on the received one of the motion data. The karaoke system according to any one of claims 1 to 3, wherein the detection unit reads predetermined motion data stored in advance in a storage means and transmits it to the server device as the motion data. . 5. The karaoke system according to claim 4, wherein the detection unit stores motion data based on the movement of the viewer terminal detected by the detection unit as the predetermined motion data.

Images