JP2014235301A - Command input discrimination system using gesture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To discriminate a command gesture from a singing posture of a singer to replace the command gesture with another video, or to enable discrimination of a command gesture to enhance interest in an image disclosure.SOLUTION: An input discrimination system includes: a command database 45c for managing a given command correlated with an action of a singer; command determination means 48 for analyzing an action video of the singer and determining a command gesture if the action of the singer exists in the command database 45c; command input section identification means 49 for identifying a pickup section in which a command gesture is performed from the singing posture of the singer; and motion image data edit means 50 capable of editing motion image in the identified pickup section to edit as a serial piece of motion image data not including a gesture for command input, or discriminating a gesture for command input.

Description

  The present invention relates to a command input identification system using gestures. For example, in a karaoke system capable of executing various commands using gestures, the present invention relates to a system capable of distinguishing gestures for command input and choreography during singing. is there.

  In recent years, as a natural user interface, a technology capable of recognizing various commands by imaging a user's action (gesture) and performing three-dimensional analysis or the like has been developed. By using this technique to link a user's action (gesture) and a command, the user can input a command by the action (gesture) without operating the karaoke remote control device or the like. .

  For example, a technique capable of controlling an automatic performance, an effect on a singing voice, and the like by a user's operation (gesture) is disclosed (see Patent Document 1). The technique described in Patent Document 1 is based on the figure of a person obtained from the imaging means and the three-dimensional position information of the person obtained from the three-dimensional position measurement device. Matches and detects human gestures. Then, the image processing device transmits a control event to the musical tone processing device and the video display device based on the detected gesture. The musical tone processing device and the video display device are configured to control automatic performance, effects on the singing voice, and the like based on the received control event.

JP-A-11-175061

  By the way, when enjoying karaoke, there is a demand not only to sing but also to imitate the choreography of the artist's dance, and to ask others to see the singing appearance of such a singer. Therefore, some currently popular karaoke systems have a function of taking a video of a singer's singing state and recording it on a recording medium such as a DVD or publishing it on a website. A karaoke system having such a moving image publishing function may employ a function of inputting a command by the above-described user operation (gesture).

  However, when shooting a singer's singing appearance, if not only dance choreography but also movements for command input (gestures) are included, movements unrelated to choreography will be recorded, For users who publish videos and viewers of videos, there is a risk of losing interest in publishing videos.

  The present invention has been proposed in view of the above-described circumstances, and identifies a karaoke system having a function of capturing a video of a singer's singing state and a function of recognizing a singer's operation and inputting a command. Then, when an operation for command input is performed, the operation is replaced with another video, or information that can identify that the operation is for command input is added to the video, An object of the present invention is to provide a command input identification system using gestures that can enhance public interest.

  The command input identification system by gesture according to the present invention has been proposed in view of the above-described circumstances, and has the following features. That is, the command input identification system by gesture according to the present invention is a karaoke system having a function of recording a singer's singing state and a function of recognizing a singer's operation and inputting a command. And a moving image data creating means, a command database, a command determining means, a command input section specifying means, and a moving image data editing means.

  The photographing means is means for photographing the singing state of the singer. The voice input means is means for inputting a singer's singing voice. The moving image data creating means is a means for creating and storing moving image data using the singing appearance of the singer photographed by the photographing means and the singing voice inputted by the voice input means. The command database is a database for associating and managing a predetermined singer's action and a predetermined command corresponding to the singer's action.

  The command determination means is a means for analyzing the action video of the singer and determining that the action of the singer is a gesture for inputting a command when the action of the singer exists in the command database. . The command input section specifying means determines the shooting section in which the command input is performed from the singing state of the singer that has been filmed when the operation of the singer is determined to be a gesture for command input. It is a means for specifying. The video data editing means edits the video data of the shooting section identified as command input, and edits as a series of video data not including a gesture for inputting the command, or the command input It is a means for making the gesture for identifiable.

  In addition, the motion video to be determined by the command determination unit is any of the singing state of the singer photographed by the photographing unit and the singing state of the singer included in the moving image data created by the moving image data creating unit. Either can be used.

  Further, the moving image data editing means is configured to, based on the command database, the moving image data of a predetermined section including a shooting section identified as a command input being performed, and a singer's face not including a gesture for inputting the command. It is possible to replace it with video data that has been digitally zoomed up into a video centered on the.

  In addition, by including video data corresponding to the content of the command in the command database, the moving image data editing means, based on the command database, in the moving image data of the shooting section identified as being command input, It is possible to synthesize video data in accordance with the contents so that a gesture for inputting a command can be identified.

  In the command input identification system based on the gesture having such a configuration, the singing voice of the singer is input by the function of the voice input means (for example, the microphone and its auxiliary equipment), and the photographing means (for example, the video camera and its auxiliary equipment) With this function, the singing state of the singer is photographed. Then, by the function of the moving image data creating means, moving image data is created using the photographed singing appearance and the input singing voice, and is stored in a predetermined recording device (for example, HDD). The stored moving image data can be recorded on a recording medium such as a DVD or uploaded to a website or the like for the purpose of moving image release.

  Further, in the present invention, a database for managing a predetermined singer's motion and a predetermined command corresponding to the singer's motion is provided and managed by the function of the command determination means. The motion video of the singer is analyzed, and when the singer's motion is present in the command database, the singer's motion is determined to be a gesture for inputting a command. When it is determined that the singer's action is a gesture for inputting a command, the karaoke performance device is operated based on the command. Such commands include, for example, increase / decrease in volume, generation of sound effects (applause, cheer, etc.), setting of echo strength, change of performance keys, change of performance tempo, and the like.

  Furthermore, when it is determined that the singer's action is a gesture for inputting a command, the shooting section in which the command is input is specified by the function of the command input section specifying means. Then, the function of the moving image data editing means edits the moving image data of the predetermined section including the shooting section identified as the command input being performed as a series of moving image data not including the gesture for inputting the command. Alternatively, a process for enabling identification of a gesture for inputting the command is performed. Thus, a series of moving image data that does not include a gesture for inputting a command, or moving image data that can identify that a singer's action is a gesture for inputting a command is created. A series of video data that does not include gestures for command input includes, for example, video data that has been digitally zoomed into a video centered on the singer's face, a singer's stop motion image, a mood image prepared in advance, karaoke Videos of other users who are enjoying together.

  In addition, if the command database includes video data (command descriptive character data, command content display data) according to the command content, the video of the shooting section identified as the command input being performed , It is possible to identify that the gesture is an operation for inputting a command.

  According to the command input identification system based on the gesture of the present invention, in the karaoke system having the function of recording the singer's singing state and the function of recognizing the operation of the singer and inputting the command, the singing that is actually photographed When an action for command input is shown in the video of the user or stored video data, the action is identified as a gesture for inputting the command, and the gesture is replaced with another video. Alternatively, it is possible to identify that the gesture is an operation for inputting a command.

  Therefore, when a singing video is recorded and played back later, or when the singing video is released, no extra motion is included in the singing video, and even if the extra motion is included Since the operation itself can be identified as an operation (gesture) for command input, it is possible to enhance the interest of the singer and the viewer.

The block diagram which shows the structure of the command input identification system by the gesture which concerns on embodiment of this invention. The flowchart which shows the procedure of the edit process of moving image data. Explanatory drawing explaining replacement | exchange of an image | video. The schematic diagram of the video processing which makes it possible to identify the gesture for command input.

  With reference to the drawings, an embodiment of a command input identification system (hereinafter abbreviated as a command input identification system) using a gesture according to the present invention will be described. 1 to 4 show a command input identification system according to an embodiment of the present invention. FIG. 1 is a block diagram showing a configuration of the command input identification system. FIG. 2 is a flowchart showing a procedure of editing processing of moving image data. FIG. 3 is an explanatory diagram for explaining video replacement, and FIG. 4 is a schematic diagram of video processing that makes it possible to identify a gesture for inputting a command.

<Overview of command input identification system>
The command input identification system 10 according to the embodiment of the present invention is a technique applied to a karaoke system having a function of recording a singer's singing state and a function of recognizing a singer's operation and inputting a command. As main components, as shown in FIG. 1, photographing means (video camera 32 and its associated devices), audio input means (microphone 34 and A / D converter 53), moving image data creation means 51, command database 45c, command determining means 48, command input section specifying means 49, and moving image data editing means 50. Each means is realized as a function means of the karaoke performance terminal 30 and its associated devices.

  In the following description, a program is a logic that can be stored in a RAM or the like and executed by hardware such as a CPU, so that not only software that exhibits the function but also an equivalent function can be achieved. It is a concept that includes a circuit.

<Karaoke performance terminal>
The karaoke performance terminal 30 to which the command input identification system 10 according to the embodiment of the present invention is applied includes a karaoke main body 40, a karaoke remote control device 31, a video camera 32, a speaker 33, a microphone 34, and a display device 35, as shown in FIG. A mixing amplifier 36 is provided. In addition, the karaoke performance terminal 30 of the present embodiment is connected to the management server 60 via the router 20 and the data transmission / reception line 70.

<Management server>
The management server 60 is a server for managing member information, distributing music data and the like to the karaoke performance terminal 30, and publishing karaoke videos uploaded by users. The data transmission / reception line 70 may be any line depending on the environment, such as an Internet line, a digital communication line, an analog communication line, a wireless communication line, a LAN, etc. It is preferable to use a VPN constituted by the Internet in that it is difficult to hear and tamper with the network and is an inexpensive communication network without monopolizing the bandwidth.

  Instead of realizing the above-described plurality of functions by the single management server 60, a server specialized for each function may be provided, and each function may be realized by each server. At this time, one server can have a plurality of functions by using a virtualization technique.

<Karaoke remote control device>
The karaoke remote control device 31 has a user interface function, and transmits / receives data to / from the local transmission / reception means 46 of the karaoke main body 40 by a wired method or a wireless method. The karaoke remote control device 31 includes a program functioning as a music search means 31a, a music index database 31b, a data storage unit 31c for storing various data, an input / output display unit 31d for inputting / outputting data, and the like. ing. A music selection operation or the like is performed by operating switches attached to the karaoke remote controller 31 or various icons displayed on the input / output display unit 31d.

<Music search means / music index database>
The music search means 31a is composed of a program for searching for music by referring to the music index database 31b based on a user instruction. The song index database 31b is a database describing attribute information of karaoke songs used for performance at the karaoke performance terminal 30. For example, the song number, song name, singer name, lyrics of the singing part, popular season, music This includes various attribute information such as genre classification and whether or not it is a duet song.

<Microphone>
The microphone 34 is a device for inputting singing voice. The singing voice signal input from the microphone 34 is mixed and amplified by the mixing amplifier 36 with the performance voice signal sent from the music reproduction control means 52 and output to the speaker 33. Note that the audio input signal from the microphone 34 is digitally converted by the A / D converter 53 and used for creating a moving image in the moving image data creating means 51 and scoring in a singing scoring means (not shown). In the present embodiment, the microphone 34 and the A / D converter 53 function as sound input means.

<Video camera>
The video camera 32 is a device that functions as a photographing unit for photographing a user's singing state, and includes an imaging lens and an imaging element as main components, and includes a focusing mechanism, a zoom mechanism, a pan / tilt mechanism, and the like. May be. When shooting is performed by the video camera 32, a video signal (video signal) is input. This video signal is used for creating a moving image in the moving image creating means 51, analyzing an action video of the singer in the command determining means 48, and the like. In the example shown in FIG. 1, two video cameras 32 are provided. This is because not only the user's singing state but also the case where the state of the audience is photographed as the replacement video data 44c is considered.

<Display device>
The display device 35 is a device for displaying a background video, lyrics telop, and the like related to karaoke music, and is configured by a liquid crystal display, for example.

<Karaoke body>
The karaoke main body 40 includes a network transmission / reception means 41, a central control means 42, a ROM 43, a RAM 44, an HDD 45, a local transmission / reception means 46, a reservation management means 47, a command determination means 48, a command input section specifying means 49, a moving picture data editing means 50, a moving picture. Data creation means 51, music reproduction control means 52, A / D converter 53, and video reproduction control means 54 are provided.

<Central control means>
The central control means 42 is a means for comprehensively controlling the karaoke main body 40 and is constituted by, for example, a CPU and its peripheral devices, and is controlled by the CPU or the like operating according to a program stored in the ROM 43 or the like. The function can be demonstrated.

<ROM / RAM>
The ROM 43 is a device for storing program data and numerical data for controlling each device constituting the karaoke main body 40, and is composed of, for example, a semiconductor memory. The RAM 44 functions as a temporary storage area for temporarily storing programs and various data, and is constituted by, for example, a semiconductor memory. Instead of configuring the RAM 44 with a physical semiconductor memory, the virtual RAM 44 may be configured using a hard disk storage device or the like.

  In the present embodiment, a reservation queue 44a, moving image data 44b, and replacement video data 44c are stored in the RAM 44. The reservation queue 44a is a data table in which music IDs are arranged in order of performance for karaoke music reserved for music selection, and other identification data such as a user ID of a music selector and a flag for performing video shooting. May be linked.

<Movie data>
The moving image data 44 b is synchronized in the moving image data creation means 51 with image data captured by the video camera 32 and input as a video signal, and singing voice data input from the microphone 34 and digitally converted by the A / D converter 53. Data. The moving image data 44b can be recorded on a recording medium such as a DVD, or uploaded to the management server 60 or the like and released.

<Replacement video data>
The replacement video data 44c is video data material for replacing the moving image data 44b in the shooting section specified as a gesture for inputting a command. For example, the mood video and karaoke prepared in advance are enjoyed together. Consists of images of other users.

<HDD>
The HDD 45 functions as a mass storage device, and stores at least a music database 45a, a video database 45b, and a command database 45c. Note that a large-capacity storage device such as a DVD capable of rewriting data may be used instead of the HDD 45 or together with the HDD 45.

<Music database / video database>
The music database 45a is a database configured by associating music control data (MIDI standard data) and lyrics rendering data in synchronization with music IDs. The performance control data is digital data for controlling the performance of each musical piece, and the lyric rendering data includes display timing data and color change data of lyric characters synchronized with the performance. The video database 45b is a database in which a predetermined number of background videos corresponding to the karaoke music to be played are stored as video files corresponding to the music ID of the karaoke music.

<Command database>
The command database 45c is a database for associating and managing a predetermined singer's action and a predetermined command corresponding to the singer's action. The commands managed in the command database 45c relate to the operation of the karaoke performance terminal 30. For example, increase / decrease in volume, generation of sound effects (applause, cheer, etc.), setting of echo strength, change of performance keys, performance For example, changing the tempo. Gestures associated with these commands include various actions such as swinging out the left or right hand, raising the left or right hand, raising both hands, and combining these actions in succession. can do. Further, video data corresponding to the content of the command may be included in the command database 45c.

  As will be described in detail later, when the action photographed by the photographing means is a gesture for inputting a command, a command input associated with the gesture is performed, and the video data editing means 50 functions to Editing is done. For command input by gesture, for example, the technique described in Patent Document 1 (Japanese Patent Laid-Open No. 11-175061) described above can be applied.

<Transmitting / receiving means>
The local transmission / reception means 46 includes an electronic circuit and a program for transmitting / receiving data between the karaoke main body 40 and the karaoke remote control device 31. In the present embodiment, data transmission / reception is performed between the karaoke main body 40 and the karaoke remote control device 31 by infrared communication. In the present embodiment, network transmission / reception means 41 for transmitting / receiving data to / from the router 20 is provided.

<Reservation management means>
The reservation management means 47 creates a reservation queue 44a by arranging the song IDs selected by the user using the function of the music search means 31a in the order of performance, and stores this reservation queue 44a in the RAM 44 for management. Consists of programs.

<Music playback control means>
The music reproduction control means 52 is an electronic circuit for digitally reproducing the sound source data based on the performance control data extracted from the music database 45a based on the music ID and converting it to analog and outputting it to the mixing amplifier 36. As described above, the mixing amplifier 36 mixes the singer's singing voice signal input from the microphone 34 and the performance voice signal sent from the music reproduction control means 52, and amplifies it by the amplifier function to be amplified by the speaker 33. It is a device for outputting more.

<Video playback control means>
The video reproduction control means 54 uses the video data extracted from the video database 45b and the lyric characters created based on the lyric rendering data included in the music database 45a during the performance of the karaoke music, and the music data of the karaoke music. And a program for outputting to the display device 35 in synchronization with the.

<Video data creation means>
The moving image data creating means 51 uses the singing appearance of the singer photographed by the photographing means (the video camera 32 and its associated devices) and the singing voice inputted from the microphone 34 and digitally converted by the A / D converter 53. It consists of an electronic device and a program for creating and storing 44b. As described above, the moving image data 44b created by the moving image data creating unit 51 is stored in the HDD 45 or the like, edited in the moving image data editing unit 50, recorded on a recording medium such as a DVD, and singing moving image in the management server 60. It is used for the public release.

<Command determination means>
The command determination means 48 analyzes a singer's motion video, and when the singer's motion exists in the command database 45c, a program for determining that the singer's motion is a gesture for command input Consists of. As described above, the motion video of the singer is captured by the video camera 32 and input to the command determination means 48 as digital data. In the command determination means 48, whether or not the singer's action is a gesture for inputting a command is analyzed by a known analysis technique such as three-dimensional analysis and compared with the action (gesture) existing in the command database 45c. Determine whether. When it is determined that the singer's action is a gesture for inputting a command, the central control unit 33 controls the music reproduction control unit etc. to increase / decrease the volume, generate a sound effect, set the strength of the echo, etc. Give instructions.

<Command input section specifying means>
The command input section specifying means 49, when the function of the command determining means 48 determines that the singer's action is a gesture for command input, the command input section specifying means 49 It consists of a program for specifying the shooting section in which an input is made. That is, the command input section specifying means 49 specifies the shooting section in which a gesture for command input is performed with respect to the singing state of the singer currently shot or the moving image data 44b stored in the HDD 45 or the like. .

<Video data editing means>
The moving image data editing means 50 edits moving image data of a shooting section identified as command input, and edits it as a series of moving image data that does not include a gesture for command input. It consists of a program that makes it possible to identify gestures for.

  For example, by the function of the command input section specifying means 49, the shooting section that is specified as the command input is performed on the singing state of the singer who is actually photographed or the moving image data 44b stored in the HDD 45 or the like. If present, the moving image data 44b stored in the HDD 45 or the like is replaced with other video data (replacement video data 44c), and edited as a series of moving image data 44b that does not include a gesture for inputting the command.

  In addition, if there is a shooting section that is specified as a command input in the singing state of the singer currently being shot or the moving image data 44b stored in the HDD 45 or the like, in the video of the shooting section, Instead of replacing with another video, an explanation display for identifying that the gesture is an operation for inputting a command may be added to the moving image data 44b.

  In this embodiment, replacement video data 44 c to be replaced with a gesture for inputting a command is stored in the RAM 44. The replacement video data 44c includes, for example, a mood video prepared in advance, a video of another user who enjoys karaoke together, and the like. Alternatively, the moving image data 44b may be directly processed and replaced with video data that has been digitally zoomed into a video centered on the singer's face, a singer's stop motion image, or the like.

  The editing of the moving image data 44b (replacement with the replacement data 44c) is preferably performed for a predetermined shooting section before and after the shooting section specified as the command input is performed. That is, when only the shooting section identified as command input is replaced with other video data, the connection of moving images may become unnatural. Therefore, by setting the singing phrase breaks as the starting and ending points of the shooting section for editing the video, not only the gestures for inputting commands unnecessary for the video are replaced with other videos, but also the entire video. Connection becomes natural, and it is possible to further enhance the interest of the singer and viewer.

<Editing gesture videos / Displaying gestures that can be identified>
Next, with reference to FIG. 2 to FIG. 4, a procedure for replacing a gesture for command input with another video or making a gesture identifiable will be described.

  In the gesture moving image editing process, as shown in FIG. 2, an operation image of the singer currently photographed or an operation image of the singer in the stored moving image data 44b is analyzed (S1). Subsequently, it is determined whether or not the operation of the singer exists in the command database 45c (S2). When the operation of the singer exists in the command database 45c, the operation of the singer is a gesture for inputting a command. (S3).

  Here, if it is determined that the singer's action is a gesture for inputting a command, the command is input from the action video of the singer who is currently shooting or from the stored video data 44b. A shooting section is specified (S4). On the other hand, if there is no gesture for command input, the shooting section is not specified.

  Subsequently, the moving image data of the specified shooting section is replaced with other video data (video obtained by directly processing the moving image data 44b or replacement video data 44c), or it is possible to identify a gesture for inputting a command ( S5). Then, in all of the specified shooting sections, it is determined whether or not the process for replacing or identifying the moving image data has been completed (whether or not the specified shooting section further exists) (S6). If there is a further shooting section, a process for making it possible to identify a gesture for image replacement or command input is sequentially performed (S5). Then, when the process of making it possible to identify an image replacement or a command input gesture for all the shooting sections is finished, the editing process of the moving image data 44b is finished.

  As shown in FIG. 3, in the image replacement process, the moving image data 44b of the gesture is replaced with other video data (replacement video) for a predetermined section including a shooting section identified as a gesture for inputting a command. In place of the data 44c), the image data is edited such as displaying a zoom-in of the singer's face image.

  In addition, as shown in FIG. 4, the process for making it possible to identify a gesture for inputting a command is a description for explaining the contents of the command with respect to a video in which a gesture for inputting a command is performed. Just synthesize the video.

<Other embodiments>
The devices and means constituting the system of the present invention and its peripheral devices are not limited to those described above, and only the necessary devices and means are configured according to the purpose of use, or other devices and means are appropriately added. Can be. Further, each unit may be configured as a unit in which a plurality of functions are integrated, instead of being configured separately.

DESCRIPTION OF SYMBOLS 10 Command input identification system 20 Router 30 Karaoke performance terminal 31 Karaoke remote control apparatus 31a Music search means 31b Music index database 31c Data storage part 31d Input / output display part 32 Video camera 33 Speaker 34 Microphone 35 Display apparatus 36 Mixing amplifier 40 Karaoke main body 41 Network Transmission / reception means 42 Central control means 43 ROM
44 RAM
44a Reservation queue 44b Movie data 44c Replacement video data 45 HDD
45a Music database 45b Video database 45c Command database 46 Local transmission / reception means 47 Reservation management means 48 Command determination means 49 Command input section identification means 50 Video data editing means 51 Video data creation means 52 Music playback control means 53 A / D converter 54 Video playback Control means 60 Management server 70 Data transmission / reception line

Claims (4)

In a karaoke system having a function of recording a singer's singing appearance and a function of recognizing the singer's operation and inputting a command,
Photography means for photographing the singer's singing appearance,
Voice input means for inputting the singing voice of the singer;
Movie data creation means for creating and storing movie data using the singing appearance of the singer photographed by the photographing means and the input singing voice;
A command database for managing a predetermined singer's motion and a predetermined command corresponding to the singer's motion,
A command determination unit that analyzes a motion video of the singer and determines that the singer's motion is a gesture for command input when the singer's motion is present in the command database;
A command input section that specifies a shooting section in which the command input is performed from the singing state of the singer that has been filmed when the action of the singer is determined to be a gesture for command input. Specific means,
Edit the moving image data of a predetermined section including the shooting section identified as the command input being performed, and edit as a series of moving image data not including the gesture for the command input, or the command input Video data editing means for identifying gestures for
A command input identification system using gestures characterized by comprising:   The motion video to be determined by the command determination means is the singing appearance of the singer who is photographing by the photographing means, and the singing appearance of the singer included in the video data created by the moving picture data creation means. The command input identification system by gesture according to claim 1, wherein the system is one of the following.   The moving image data editing means, based on the command database, the moving image data of the shooting section identified as the command input being performed, focusing on the face of the singer who does not include a gesture for the command input. 3. The command input identification system by gesture according to claim 1 or 2, wherein the image is replaced with video data that has been digitally zoomed up. The command database includes video data corresponding to the content of the command,
The moving image data editing means combines the moving image data corresponding to the content with the moving image data of the shooting section identified as the command input being performed based on the command database, and The gesture command input identification system according to claim 1 or 2, wherein a gesture for identification can be identified.

Images