JP6548499B2 - Karaoke system - Google Patents

Description

  The present invention relates to a karaoke system having a function of recording a singing video of a singer.

  In particular, the karaoke apparatus used by a plurality of customers such as a karaoke room and a night shop is required to have an entertainment property to enliven the place, and various techniques for meeting the needs have been proposed. Technology to display background images on displays etc. along with karaoke performances is one of them. In particular, with the recent increase in capacity of storage devices, it has become possible to produce and store background images corresponding to each piece of music. There is. For example, there is a service that records live video of an artist who sings a song and plays it as a background video when the singer (user in the karaoke room) sings the song, and the singer has a live presence feeling You can sing comfortably while tasting On the other hand, it is also desired to generalize a service that records singing by singers, and to provide a recording that reflects the live feeling of singing by singers in order to improve service to users.

  Conventionally, as one of the service functions for the user of the karaoke apparatus, for example, a head mount display (hereinafter referred to as "HDM") is connected to the karaoke apparatus to display lyrics so that the position of the singer can be freely set. There is known a technique for increasing the degree and solving the problem that the lyrics are not visible due to an obstacle or the like (Patent Document 1). In addition to this, there is also known a technology capable of reproducing omnidirectional images and sounds by HMD and stereo headphones (Patent Document 2).

  On the other hand, as another service provided in the karaoke room, a service for recording a singing video is in widespread use. For example, there is also known a technique of recording a singer's song, uploading it to a server, and making it publicly viewable (Patent Document 3).

JP 2001-022362 A Japanese Patent Application Laid-Open No. 2002-171460 JP, 2011-059619, A JP 2012-008451 A JP, 2014-085796, A

  By the way, by using the conventional techniques as described in Patent Documents 1 to 3 above, when a singer sings, a live image of the artist's line of sight is displayed on the HMD, and the posture of the singer is used as a singing image. Although you can record, but you can feel the presence of live by HMD display at the time of singing, the singing video that can be recorded and browsed is simply the video that is singing in the karaoke room, and a great sense of satisfaction obtained There is a problem that can not be reproduced.

  On the other hand, Patent Document 4 and the like disclose a technique for combining a singing video in a karaoke room with another background video, and it is conceivable to use a stage image as a background video and to insert a singing video there. Even if the singing image is embedded in the stage image unrelated to the image displayed on the singer's HMD at a time, the audience reaction is not linked with the progress of the music even if the appearance is good, and the problem of eventually losing interest There is a problem that the viewer feels uncomfortable because the image is being sung while the singer wears the HMD.

  Therefore, the present invention has been made in view of the above problems, and it is an object of the present invention to provide a karaoke system which reflects a live singing feeling of a singer and provides a viewer with a recorded video without a sense of discomfort.

In order to solve the above-mentioned problems, the invention according to claim 1 is a karaoke system providing a video sung by a singer, and music data storage means for storing a plurality of music data for karaoke performance; A video displayed on a head mounted display worn by a singer singing along with a karaoke performance, and a stage viewpoint video based on a view when the music is sung on an actual live stage with the music data of the music A stage viewpoint video storage means linked and stored, and an image sharing the same time and space as the stage viewpoint video, and a audience seat viewpoint video in which only the configuration of the scene on the stage is viewed from the audience seat, and audience perspective image storage means to be stored the stage view picture and cord attached is in, Calao of the song by the singer person or other singer Among a plurality of singing videos sung in advance according to the performance, singing video acquisition means for acquiring a singing video desired by the singer, singing video of the music acquired by the singing video acquisition means a stage view image editing means for editing the stage view image of the music on the basis of, at least, the image display control means for displaying a stage view image edited by the stage view image editing means in accordance with the karaoke performance to the head-mounted display If, taking the figure of singer is singing in accordance with the karaoke playing a singing video recording means for recording as recording singing video, reads out the recording singing video recorded by the singing video recording means, said on the recording singing video edit incorporates singing image and the audience view images acquired by the singing image acquisition unit A configuration with, and singing video editing means.

In the invention of claims 2 and 3, "the face image data storage means for storing the image data of the face of the singer is further provided, and the singing picture editing means comprises a head mounted display worn by the singer of the recording and singing picture. "Correct the image area including at least using the face image data",
"An accessory image data storage means for storing image data of an accessory worn by the singer on the head is further provided, and the song video editing means is an image including a head mount display worn by the singer of the recorded song video The region is corrected with at least the image data of the decorative item.

According to the invention of claim 1, a plurality of music data, a stage viewpoint image from an actual live stage displayed on a head mounted display worn by a singer, a spectator seat where the configuration on the stage is viewed from the spectator seat stores the viewpoint video, which the singer is recorded as a record singing image by taking the figure to be singing together in karaoke performance, editing by incorporating the audience point of view the video to record singing the video that has been read after singing The song video of the singer who has been sung in accordance with the karaoke performance of the music is stored in advance for publication, the song video is acquired when performing the karaoke performance of the music, and the song video is obtained based on the song video A head mount display worn by a singer who edits a stage viewpoint video and sings the edited stage viewpoint video according to the karaoke performance Displays Lee, also reads the recorded singing video after the singing, with the configuration of editing incorporate the acquired singing video and audience view image on the recording singing image is displayed on the HMD when singing The scene from the singing live will be incorporated into the recorded recording video as well as the scene from the spectator seat, so that the live presence felt at the time of singing can be reproduced also at the time of reproduction of the recording video, the singer Can reflect the lively singing feeling of the music video and provide viewers with a recorded video without discomfort, and edit the display on the HMD or recorded video for multiple songs regardless of the person or others It can be done.

  According to the second and third aspects of the present invention, the image data of the face of the singer is stored, and the image area including the head mounted display worn by the singer of the recording and singing image is at least An image area including a head mount display worn by the singer in the singing image editing means, which corrects using image data and stores the image data of the ornament worn by the singer of the recorded song video on the head. Correction by replacing at least the image data of the ornament with the image of the singer's face or the image of the ornament such as sunglasses by correcting the HMD in the recorded recorded song video Thus, it is possible to eliminate the discomfort that the singer wears the HMD and sings on the stage.

It is a block block diagram of 1st Embodiment in the karaoke system which concerns on this invention. It is structure explanatory drawing of the server of FIG. It is each block block diagram of the image | video display control means of FIG. 1, a song imaging | video production | generation means, and a song imaging | video editing means. It is a processing flowchart of the image display control means of FIG. It is a processing flowchart of the song imaging | video production | generation means of FIG. It is a processing flowchart of the song imaging | video editing means of FIG. It is a partial block diagram of 2nd Embodiment in the karaoke system which concerns on this invention. It is structure explanatory drawing of the server in 2nd Embodiment of the karaoke system which concerns on this invention. It is a processing flowchart of the stage viewpoint video editing means of FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows a block diagram of a first embodiment of the karaoke system according to the present invention, and FIG. 2 shows a schematic diagram of the server of FIG. In FIG. 1, the karaoke terminal 11 is externally connected to the karaoke body 12 as a main device by wire or wireless, and includes a display unit 13, a mixing amplifier 14, a microphone 15, a speaker 16, a remote input / output device 17, and a head. A mounted display (HMD) 18 and an imaging unit 19 as an imaging unit are connected.

  The display unit 13 displays normal music selection display, lyrics at the time of karaoke performance, background images, etc. For example, a liquid crystal display (LCD), a plasma display (PDP), and various other displays can be adopted. . The mixing amplifier 14 mixes the audio signal input from the microphone 15 in a wired or wireless manner with the music performance signal sent from the karaoke body 12, amplifies it, and amplifies and outputs it from the speaker 16.

  The remote input / output device 17 transmits / receives data to / from the karaoke body 12 by using a wired system or a wireless system (IR system, piconet connection system of Bluetooth (registered trademark) mechanism, etc.) by a terminal transmission / reception unit (not shown) To that end, at least login / logout processing means 17A and music selection music registration means 17B which is music search means are suitably provided.

  The login / logout processing means 17A acquires a user name and a user ID, which is identification information, from the user in response to a login operation by the user, performs login processing, and terminates use processing in response to the logout operation. Is a program that

  The music selection music registration means 17B searches the music desired by the user on the basis of a table comprising data such as music ID, music title, artist name, singing part information and the like by the GUI function of the remote control display unit (not shown). In this case, it is set whether or not to record the singing video (here, setting is made to record the singing video), the music ID of the selected music is attached to the user ID, and the karaoke body 12 is It is a program to send. The transmitted song ID is registered in the reservation queue 41 of the RAM 23 in the karaoke body 12.

  The HMD 18 is a liquid crystal display (LCD) prepared to be worn by a singer and displaying an image (to be described later) from the image display control means 24 described later, which is known to make the wearer visually recognize a sense of presence It is a thing. The HMD 18 has a head tracking function, and a head tracking signal corresponding to, for example, the front, right, left, or back is input to the head tracking signal input unit 32 of the karaoke body 12. The singer may wear headphones along with the HMD 18, and sound input to the headphones may be connected by wire or radio from the mixing amplifier 14.

  The image pickup unit 19 is a digital video camera provided with an image pickup element, photographs a manner in which a singer sings in accordance with a karaoke performance, and outputs video data to the singing video generating means 33 of the karaoke body 12. As the format of the video data, for example, the format adopted by a general-purpose video camera can be applied, and it is preferable to use the same format as the data of the audience seat viewpoint video to be described later. The imaging unit 19 and the singing picture generating means 33 described later constitute a singing picture recording means. The technique described in Patent Document 3 can be used for shooting and recording of a singer.

  The karaoke body 12 includes a bus 20, a central control unit 21, a ROM 22, a RAM 23, an image display control unit 24, a reproduction unit 25, an audio signal input unit 26, transmission / reception units 27A and 27B, a storage unit 28, and user information acquisition unit 31. And a head tracking signal input unit 32, a singing image generating unit 33, and a singing image editing unit 34.

  In the RAM 23, storage areas for the reservation queue 41, the user information 42, and the recording song video 43 are formed. Further, the storage unit 28 includes a music database (music DB) 51, a background video database (background video DB) 52, a viewpoint-specific video database (view-specific video DB) 53, and an ornament image database which is a decorative item image data storage unit The (decorative product image DB) 54 is stored. The viewpoint-specific video DB 53 constitutes stage viewpoint video storage means and audience seat viewpoint video storage means, and stores the stage viewpoint video 53A and the audience seat viewpoint video 53B (described later).

  In addition, in order to show that most of the above-described configurations are applicable to the conventional karaoke system, even if they are component parts not directly related to the subject matter of the present invention, the entire components will be described.

  The central control unit 21 is a physical CPU that centrally controls processing of this system, and performs algorithm processing based on a program stored in the ROM 22. The RAM 23 serves as a work area for developing and executing the various programs, in addition to storage areas for the reservation queue 41, the user information 42, and the recording song video 43. It is a concept that includes semiconductor memory and is virtually constructed on a hard disk.

  The above-mentioned video display control means 24 outputs the lyric display data based on the background video extracted from the background video DB 52 and the lyric data of the music extracted from the music DB 51 to the display unit 13 during performance. It is a program or an electronic circuit that outputs the viewpoint image 53A and the above-mentioned lyric display data. Details will be described with reference to FIGS. 3 and 4.

  The reproduction means 25 includes, for example, a sequence program and a sound source (synthesizer) for karaoke music, drives the sound source (synthesizer) according to the note data of the music data, and outputs it to the mixing amplifier 14 as a performance signal. That is, the music data of the music (music ID) registered in the reservation queue 41 of the RAM 23 is extracted from the music DB 51 based on the instruction of the central control unit 21 and reproduced.

  The audio signal input means 26 is an electronic circuit for A / D converting an audio signal input from the microphone 15, and the A / D converted audio signal is provided as an audio signal to the singing video generating means 33 described later. Ru.

  The transmission / reception unit 27A exchanges data with the remote input / output device 17 by using a wired system or a wireless system (IR system, piconet connection system of Bluetooth (registered trademark) mechanism, etc.). Electronic circuits and programs. The transmission / reception unit 27B is configured by software such as a physical communication circuit or platform for the karaoke body 12 and the server 62 to exchange data via the communication network 61.

  The music DB 51 stored in the storage unit 28 stores note data, singing lyric data (including color change information) and the like for each music. With regard to the performance, specifically, there is a music table associated with music ID, music title and artist ID (artist name), and for each music, performance data of karaoke music of a predetermined data format managed by music ID (for example, It is a database in which the music piece ID is stored as a file name for music data (file) composed of musical note data for each note of a melody part of MIDI (registered trademark) format or the like. The background video DB 52 stored in the storage unit 28 is a database in which background music data corresponding to each music is stored with the music ID as a file name. The background video here is a live video (a video viewed from the audience seat) including the artist that the original music artist is singing.

  The viewpoint-specific video DB 53 stored in the storage unit 28 includes the stage viewpoint video 53A and the audience seat viewpoint video 53B. The stage viewpoint video 53A is a video displayed on the HMD 18 worn by a singer who sings in accordance with the karaoke performance, and the music of the stage viewpoint video based on the view when the music is sung on the actual live stage is Image data associated with the music data (music ID). Specifically, it is an image of the entire circumference (planar all angles) of the artist's line of sight when the original music artist sings live for the predetermined music. Note that the technology described in Patent Document 2 can be used for recording of all-round video and display on the HMD 18. Also, the entire circumference image of the stage viewpoint image 53A displayed on the HMD 18 is not necessarily limited to the image of the artist's line of sight, but it may be an image of the appropriate viewpoint on the stage, but in that case the artist who sings is not shown Is desirable.

  The audience viewpoint video 53B is a video sharing the same time and space as the stage viewpoint video 53A, and the audience seat viewpoint video of a scene on an actual live stage viewed from the audience seat is the same as the stage viewpoint video 53A. It is linked and stored. The audience seat viewpoint video 53 </ b> B is the same video as the original music artist of the background video of the music stored in the background video DB.

  The accessory image DB 54 stored in the storage unit 28 is an image such as an accessory worn by the singer on the face or clothes worn by the person other than the face, for example, specified as an accessory worn on the face There are kinds of glasses, predetermined types of sunglasses and predetermined types of hats.

  The user information acquisition unit 31 acquires user information such as age and gender and a user face image (described in FIG. 2) from the server 62 described later based on the user ID acquired in the login process. It is a program that is attached to the user ID in the RAM 23 described later and stored as user information 42.

  The head tracking signal input means 32 inputs the head tracking signal sent from the HMD 18 according to the face angle angle of the singer wearing the HMD 18 and supplies it to the video display control means 24 and the singing video generation means 33. It is an electronic circuit or program.

  The above-mentioned singing video generation means 33 is a program for acquiring a video of the imaging unit 19 in which the singing person shoots a posture to sing along with the karaoke performance and recording it as a recorded singing video, and the recorded singing video is recorded in the RAM 23 The image 43 is stored. The technique described in Patent Document 3 can be used for shooting and recording of a singer. Details will be described with reference to FIGS. 3 and 5.

  The singing video editing means 34 reads out the recorded singing video 43 which is recorded by the singing video generating means 33 and stored in the RAM 23, and the musical composition of the music which is stored in the viewpoint-based video DB 53 of the storage unit 28 in the recorded singing video. It is a program that incorporates the audience seat viewpoint video 53B and edits the audience seat video into the video sung by the singer. In addition, the singing video editing means 34 has a program for correcting the image area including the HMD 18 worn by the singer using the image related to the user acquired in advance from the user who is the singer.

  For example, image data of all or part (here, part) of the face is stored from the user as will be described later, and correction is performed using the image data. Further, the image area including the HMD 18 worn by the singer is corrected using the image data of the accessory (which may be the image data including the accessory). The image area including the HMD 18 worn by the singer may be corrected by the image data to the accessory, or the face area may be further corrected after the image area including the HMD 18 worn by the singer is corrected to the image data of the face. You may correct | amend with the image data of a decoration item. In this embodiment, details of the latter correction will be described with reference to FIG. 3 and FIG.

  On the other hand, in FIG. 2A, the server 62 manages a predetermined number of karaoke terminals 11 (11A to 11N) as terminals via a communication network 61 such as the Internet, and distributes music data to the respective karaoke terminals 11. These devices transmit and receive various data, transmit face image data of the user in response to a request from at least one of the karaoke terminals 11, and transmit and receive upload singing videos.

  The server 62 has a function of releasing a song image recorded when the user sings an arbitrary music song so as to be downloadable, but as shown in FIG. 2 (B), at least the control unit as a basic configuration 71, network transmission / reception unit 72, user information extraction unit 73, upload video transmission / reception unit 74, user database (user DB) 75 as storage unit, user face image database (face image storage unit (user face image) DB) 76, upload song video database (upload song video DB) 77 is provided.

  The user DB 75 is the same as the conventional configuration, and the description thereof is omitted. Further, the processing means for uploading and downloading for release is also the same as the configuration of the conventional release server in the upload singing video database 77, so the configuration and description will be omitted.

  The control unit 71 is a physical CPU that controls the server 62 in an integrated manner, and executes an integrated processing by executing a program stored in a ROM (not shown). The network transmission / reception unit 72 is, for example, a physical communication circuit or platform for achieving consistency with the communication system of the communication network 61 in order to perform communication (data exchange) with each karaoke terminal 11 (11A to 11N). It consists of software such as

  The user information extraction means 73 extracts the various bibliographic information from the user DB 75 when there is a request for information of the user based on the user ID from the predetermined karaoke terminal 11, and the user The face image data of the user based on the user ID is extracted from the face image DB 76 and transmitted to the corresponding karaoke terminal 11 as part of the user information.

  The upload video transmission / reception means 74 extracts the song video from the upload song video DB 77 and sends it to the corresponding karaoke terminal 11 when there is a request for selection and download of a song video for upload from the predetermined karaoke terminal 11. It is a program.

  The user face image DB 76 stores the face image data of the user in advance in association with the user ID as shown in FIG. 2C, and as an example, the user is diagonally behind the right 45 °, right 45 ° forward, 45 ° front, left 45 ° right, 45 ° right back, right back each still image data, for example 1 / 5th of head part face image It is stored as data.

  Therefore, FIG. 3 shows a block diagram of each of the video display control means, the singing video generation means and the singing video editing means of FIG. 1, and FIG. 4 shows the process flowchart of the video display control means of FIG. FIG. 6 shows a processing flow chart of the singing image generation means of FIG. 3, and FIG. 6 shows a processing flow chart of the singing image editing means of FIG.

  In FIG. 3A, the video display control unit 24 includes a lyric display control unit 81, a background video display control unit 82, and an HMD video display control unit 83. The lyric display control unit 81 is a program for acquiring lyric data from the music DB 51 for a musical piece to be played and performing color change display along the musical performance to make lyric display data. The background image display control unit 82 acquires a background image (song image from the audience seat viewpoint sung by the original artist) from the background image DB 52 based on the music ID of the music to be played, and lyrics the background image. It is a program which adds the lyric display data produced | generated by the display control part 81, and outputs it along with a performance to the display part 13 as a background image.

  The HMD video display control unit 83 acquires the target stage viewpoint video 53A from the viewpoint-based video DB 53 based on the music ID of the music to be played, and acquires the head tracking signal from the HMD 18 while singing, and the stage viewpoint It is a program that adds the above-mentioned lyric display data to the video portion in the viewpoint direction of the singer according to the head tracking signal in the video 53A and outputs the data along the performance to the HMD 18.

  That is, as shown in FIG. 4, for the reproduced music, the lyrics data is acquired from the music DB 51, the background video data is acquired from the background video DB 52, and the head tracking signals from the stage viewpoint video 53A and the HMD 18 from the viewpoint video DB 53. Are obtained from the head tracking signal input means 32 (step 1 (S1)).

  When the music is reproduced, the lyric display control unit 81 sequentially generates the lyric display data, and the background image display control unit 82 adds the lyric display data to the background image in synchronization with the reproduction of the music, and the display is performed. The HMD video display control 83 adds the lyric display data to the stage viewpoint video of the viewpoint angle according to the head tracking signal based on the stage viewpoint video data 53A and outputs it to the HMD 18 (S2). For example, if a singer wearing the HMD 18 faces the front, a spectator seat is displayed, if the user directs the right, a bassist on the stage and a guitarist is displayed the left. Then, the process is continued until the reproduction of the music is completed (S3).

  In FIG. 3 (B), the song video generation unit 33 includes an imaging data input unit 91 and a song video synchronization unit 92. The said imaging data input part 91 is a program which inputs the imaging data which imaged the singer who is singing on the stage in the karaoke room from the imaging part 19, for example. The song video synchronization unit 92 synchronizes the singing voice signal of the singer obtained from the voice signal input unit 26 with the imaging data input from the imaging unit 19 and attaches incidental information obtained by synchronizing the head tracking signal. This is a program for generating a recording song video and storing it as the recording song video 43 in the RAM 23. The head tracking signal is attached to the recorded song video in order to specify the viewpoint direction (face direction) of the singer along the song when the singer is subsequently combined with the audience seat viewpoint video.

  That is, as shown in FIG. 5, for the reproduced music, the imaging data is acquired from the imaging unit 19 in the imaging data input unit 91, and the singing video synchronization unit 92 acquires a voice singing signal from the voice signal input unit 26; The head tracking signal from the HMD 18 is acquired from the head tracking signal input means 32 (S11).

  The song video synchronization unit 92 synchronizes the singing voice signal with the acquired imaging data, and synchronizes the acquired head tracking signal and sequentially records it as attached information (S12). Then, when the music reproduction is finished (S13), the recorded singing video is stored in the RAM 23 as the recording video 43 (S14).

  In FIG. 3C, the singing video editing unit 34 includes a synthesizing unit 101, an HMD recognizing unit 102, a face angle specifying unit 103, a replacing unit 104, an eye recognizing unit 105, and a processing unit 106. The synthesizing unit 101 acquires the recording song video 43 from the RAM 23 and acquires the audience seat viewpoint video 53 B of the reproduction target music from the viewpoint-based video DB 53, and at least the area other than the singer of the recording singing video 43 This is a program for synthesizing the video 53B.

  The HMD recognition unit 102 is a program for recognizing the image area of the HMD 18 worn by the singer in the synthesized video. The face angle identification unit 103 is a program for identifying the face angle of the singer in the synthesized video based on the head tracking signal attached to the recorded song video 43, and, for example, the viewpoint direction of the singer to be acquired The direction in which the user is facing is specified in correspondence with the angle of the face image data.

  The substitution unit 104 reads face image data contained in the user information 42 of the RAM 23, erases the image area of the HMD 18 located on the head of the singer recognized by the HMD recognition unit 102, and recognizes the face angle The program is a program for replacing the face image data corresponding to the viewpoint angle of the singer recognized by the speaker 103 and the surrounding background image for correction.

  The both-eyes recognition unit 105 is a program for recognizing the both-eyes area in the singer's face image data corrected by the substitution unit 104. The processing unit 106 is a program for acquiring desired accessory image data from the accessory image DB 54 and incorporating the image data of the acquired accessory into the two-eye area recognized by the both-eye recognition unit 105 for correction. Note that the technology described in Patent Document 4 can be applied to the synthesis in the processing in the song video editing means 34, and the technology described in Patent Document 5 is used for detection of an image area, substitution with other data, and correction. It can be used.

  That is, as shown in FIG. 6, for the reproduced music after completion of the reproduced music, the face image data of the singing person in the recorded / song video 43 and the user information 42 of the singer is acquired from the RAM 23 The audience seat viewpoint video 53B is acquired from the video DB 53, and desired accessory image data (for example, image data of sunglasses) is acquired from the accessory image DB 54 (S21).

  The synthesizing unit 101 synthesizes the recording / song video 43 of the singer with the audience seat viewpoint video 53B (S22). Subsequently, the HMD recognition unit 102 detects the HMD image area based on the composite image, and the face angle specification unit 103 sequentially specifies the face angle, and deletes the HMD image area detected by the replacement unit 104 and sings. Face image data corresponding to the identified face angle of the person is incorporated, and the surrounding background image is corrected (S23).

  Further, both eyes area of the face image incorporated by both eyes recognition unit 105 is recognized, and a singing picture is generated by adding the accessory image data (for example, the decoration image data of sunglasses) acquired on the both eyes area by processing unit 106 (S24). Then, the song video is transmitted to the server 62 as an upload song video (S25). In the server 62, the upload video transmission / reception unit 74 shown in FIG. 2 (B) receives and stores it in the upload song video DB 77.

  In this way, the scene of the live song displayed on the HMD 18 during singing will be incorporated into the recorded song video 43 as the scene from the spectator seat, and when playing back the recorded song video (upload song video) It is also possible to reproduce the live presence that you tasted at the time of singing. In addition, since the HMD 18 in the recorded song video 43 recorded while reflecting the live singer's lively feeling of the singer is replaced with the image of the singer's own face or the image of the ornament such as sunglasses, the singer Can eliminate the sense of discomfort that the user wears the HMD 18 and sings on a virtual live stage.

  Next, FIG. 7 shows a partial block diagram of a second embodiment of the karaoke system according to the present invention, and FIG. 8 shows a configuration explanatory view of a server in the second embodiment of the karaoke system according to the present invention. In the second embodiment according to the present invention, as shown in FIG. 7A, the remote input / output device 17 of FIG. 1 is provided with a duet target image selection acquisition unit 17C as a singing image acquisition unit. The video duet duet is a form of multiple singing.

  The above-mentioned duet target video selection acquiring means 17C acquires a singing video of a singer who has been sung and stored in advance in accordance with the karaoke performance of the music, and specifically, the user selects, for example, a music selection With respect to a duet music upload video extracting means (78) described later of the server 62, a desired duet target video is searched and selected from the upload song video DB 77, and a corresponding duet target video is downloaded from the server 62. Is a program stored as a duet target video 44 in the RAM 23 shown in FIG. Note that the duet target video may be a singing video of the user himself / herself uploaded previously without any other person (for example, a single time difference duet).

  Further, the karaoke body 12 is provided with stage viewpoint video editing means 35, and a storage area for the duet HMD display video 45 is provided in the RAM 23. The stage viewpoint video editing means 35 is a program for editing the stage viewpoint video 53A of the music based on the duet target video 44 of the music acquired from the server 62 and stored in the RAM 23. Specifically, as shown in FIG. , A stage viewpoint video acquisition unit 111, a duet target video acquisition unit 112, and a synthesis unit 113.

  The stage viewpoint video acquisition unit 111 is a program for acquiring the stage viewpoint video 53A of the corresponding music from the viewpoint-specific video DB 53. The duet target video acquisition unit 112 is a program for acquiring the duet target video 44 stored in the RAM 23. The combining unit 113 is a program that combines the duet target video 44 with the stage viewpoint video 53 A and stores the result as the duet HMD display video 45.

  On the other hand, in FIG. 8A, the server 62 is newly provided with a duet music upload video extracting means 78. The duet music upload video extracting means 78 searches a desired duet target video from the upload song video DB 77 according to the request from the user of the predetermined karaoke terminal 11 as described above, and corresponds the selected duet target video Is a program to be sent to the karaoke terminal. For example, the upload video of the selection target duet as shown in FIG. 8B is extracted, and a singing video based on the selected singing video ID is extracted from the upload singing video DB 77 and transmitted to the corresponding karaoke terminal 11 Ru.

  Therefore, FIG. 9 shows a processing flowchart of the stage viewpoint video editing means of FIG. The processing by the stage viewpoint video editing means 35 may be immediately after the music selected by the user is registered in the reservation queue 41 of the RAM 23, or may be immediately before the music is reproduced. That is, in FIG. 9, the stage viewpoint video acquisition unit 111 acquires the stage viewpoint video 53A from the viewpoint video DB 53, and the duet target video acquisition unit 112 reads out the duet target video 44 from the RAM 23 and acquires it. (S31).

  The combining unit 113 combines the duet target video 44 with the stage viewpoint video 53A (S32). For example, in the stage viewpoint video of the whole circumference, the audience seat is displayed when facing the front, and the guitarist is displayed when the singer of the duet opponent on the stage is facing the left when facing the right. In addition, although the image of the singer of the duet target is basically a front image, the image when the singer turns to the right is also an image, but if, for example, the left side image is also photographed in addition to the front It can be solved.

  Then, the composite video is stored in the RAM 23 as a duet HMD display video 45 (S33). The video display control means 24 displays the edited stage viewpoint video (the duet HMD display video 45) on the HMD 18 worn by a singer who sings in accordance with the karaoke performance.

  On the other hand, after singing, the singing video editing means 34 reads out the recorded singing video 43 stored in the RAM 23, and the recording / song video 43 is subjected to the duet target video selection acquiring means 17C of the remote input / output device 17 which is the singing video acquiring means. Edits by incorporating the duet target image 44 acquired in the RAM 23 and stored in the RAM 23 and the audience seat viewpoint image 53B acquired from the viewpoint-specific image DB 53, replacing face image data, processing ornament image data, and uploading It is transmitted to the server 62 as a singing video.

  In this manner, the display on the HMD 18 and the recorded video can be edited for a plurality of songs regardless of the person or others. More specifically, for "Collaboration Singing Video (refer to Patent Document 3)", which sings according to the singing video of another singing person and creates a new singing image, duet singing is performed with a sense of live presence, It is possible to reproduce the presence of a duet song in live even during playback.

  The karaoke system of the present invention can be used in industries such as manufacture, use, and sale of a device provided with the basic functions of karaoke.

11 karaoke terminal 12 karaoke main unit 13 display section 17 remote input / output device 18 head mounted display (HMD)
19 image pickup unit 24 image display control means 26 audio signal input means 28 storage unit 31 user information acquisition means 32 head tracking signal input means 33 singing video generation means 34 singing video editing means 35 stage viewpoint video editing means 42 user information 43 recording Singing image 44 duet target image 45 duet music HMD display image 53 viewpoint-based image database (image database according to viewpoint)
54 Ornamental Product Image Database (Decorative Product Image DB)
62 server 73 user information extraction means 74 upload video transmission / reception means 75 user database (user DB)
76 User Face Image Database (User Face Image DB)
77 upload singing video database (uploading singing video DB)
78 duet music upload video extraction means 81 lyrics display control unit 83 HMD image display control unit 91 imaging data input unit 92 singing video synchronization unit 101 synthesis unit 102 HMD recognition unit 103 face angle identification unit 104 replacement unit 105 second eye recognition unit 106 processing unit 111 Stage viewpoint video acquisition unit 112 Duet target video acquisition unit 113 Synthesis unit

Claims (3)

It is a karaoke system that provides a video of a singer singing.
Music data storage means for storing a plurality of music data for karaoke performance;
A video displayed on a head mounted display worn by a singer singing along with a karaoke performance, and a stage viewpoint video based on a view when the music is sung on an actual live stage with the music data of the music Stage viewpoint video storage means which are stored in association with each other;
An audience seat that shares the same time and space as the stage viewpoint video, and that is a audience seat viewpoint video in which only the configuration of the scene on the stage is viewed from the audience seat in association with the stage viewpoint video Viewpoint video storage means,
Singing video acquisition means for acquiring a singing video desired by the singer among a plurality of singing videos sung and presented in advance according to the karaoke performance of the music by the singer himself or another singer;
Stage viewpoint video editing means for editing the stage viewpoint video of the music based on the singing video of the music acquired by the singing video acquiring means;
At least a video display control unit for displaying on the head mounted display a stage viewpoint video edited by the stage viewpoint video editing unit in accordance with a karaoke performance;
Singing video recording means for photographing the manner in which the singer sings in accordance with the karaoke performance and recording it as a recorded singing image;
Singing video editing means for reading out the recorded singing video recorded by the singing video recording means, and incorporating and editing the singing video acquired by the singing video acquiring means and the audience seat viewpoint video into the recorded singing video;
Karaoke system characterized by having. The karaoke system according to claim 1, wherein
A face image data storage unit for storing image data of a singer's face;
The karaoke system, wherein the singing picture editing means corrects an image area including a head mounted display worn by a singer of the recording and singing picture using at least image data of a face. The karaoke system according to claim 1 or 2, wherein
An accessory image data storage means for storing image data of an accessory worn by the singer on the head;
The karaoke system, wherein the singing video editing means corrects an image area including a head mounted display worn by a singer of the recorded singing video using at least image data of a decorative item.

Images