JP2014217627A - Physical action evaluation device, karaoke system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an environment of enjoying choreography and interjection to a user including a singer in an environment of karaoke, etc., and to provide new evaluation functions to the choreography and the interjection, etc.SOLUTION: A physical action evaluation device includes: an example data memory unit 106 for storing multiple example data defining examples of physical movement in conformity to music along with information of reproduction position of corresponding music, respectively; an example data image reproduction unit 102 for reproducing images of the multiple example data in synchronization with reproduction of the music; a physical action acquisition unit 110 for acquiring the physical action of the user in association with the reproduction position of the music; a matching degree specifying unit 112 for specifying the matching degree of the physical action to the example data from a matching degree between the example data and each of the acquired physical action; and a physical action evaluation value calculation unit 116 for calculating an evaluation value of the physical action of the user by using the specified matching degree corresponding to each of the multiple example data.

Description

  The present invention relates to a body motion evaluation apparatus, a karaoke system, and a program.

  In karaoke or the like that accompanies a singer, various additional functions are provided to the singer and other users. For example, the performance of music, the video display that changes in accordance with the singing of the singer, adjustment of keys, tempo, etc.

  Moreover, the scoring function for singing is one of important additional functions in karaoke. By allowing the karaoke system to perform such evaluation on singing, an evaluation value as a more objective numerical value can be provided.

  In today's karaoke, the elements that entertain the user are not limited to the singing of the singer. That is, the body movement (choreography) by a singer, a user, etc., body movements such as gestures, and various actions (hands) such as shout and chorus are performed. For example, the physical movement (choreography) of a singer has become an important factor in enjoying music in today's days when many of the hit songs have been choreographed. In addition, since the user (viewer) can become an active participant in the music from a passive listener, the enjoyment in karaoke can be further enhanced.

  There is a technique for providing a karaoke play that displays the timing at which a listener puts a “hand of hand”. That is, a time signature display memory that stores a series of image data for a time signature display image that sequentially changes its display mode in order to assist the progress of singing, timing detection means that detects timing information included in the acoustic control data, Based on the timing information detected by the timing detection means, the timing determination means for determining the timing for switching a plurality of image data, and the display form differs from the time signature display memory in synchronization with the timing determined by the timing determination means. Video signal generating means for sequentially reading out the image data of the time signature display image and generating a video signal for displaying the displayed image at a predetermined position of the display (see, for example, Patent Document 1).

  Further, a sensor for detecting the motion of at least a part of the user's body as an evaluation target, a motion determination means for determining the motion of the evaluation target based on the detection result by the sensor, and the motion determination means It is possible to evaluate physical movements such as choreography in the performance of a performance song, including movement evaluation means for evaluating the movement of the evaluation object by comparing the determination result with the predetermined model movement of the evaluation object (See, for example, Patent Document 2).

  In addition, a 3D image of a model character is displayed on the second divided screen of the monitor device, and a continuous photograph-like auxiliary image indicating the flow of dance is displayed on the auxiliary screen below the monitor device. The player recognizes the flow of the dance with this auxiliary image and dances by imitating the 3D image of the character. This figure is captured by the front camera device and displayed on the first divided screen 11a. A plurality of infrared lasers and infrared sensors provided on the stage 1 detect the step position and step timing of the player. The rear camera device captures a rear image of the player. This rear image is matched with the character image. There is a technique for comprehensively evaluating the player's dance based on the detection output from the infrared sensor and the matching error (see, for example, Patent Document 3).

  The movement of the motion detection marker provided on the surface of the singer's body is detected by the camera, and the singer's motion data such as choreography based on the detection information is created and output in real time according to the karaoke performance. There is a technique for reflecting the user's own operation of the karaoke system in various forms on the function on the apparatus side by utilizing the outputted singer operation data (see, for example, Patent Document 4).

  Further, the storage means for storing a given questioning action for the player, the questioning action display means for displaying the questioning action, the action detecting means for detecting the action of the player, and detected by the action detecting means Similarity determination means for determining similarity by comparing the player action display means for displaying the action of the player, the questioning action, and the action of the player detected by the action detection means And a game calculation means for performing score processing based on the determination result by the similarity determination means (see, for example, Patent Document 5).

JP 09-297594 A JP 2006-162916 A JP 2001-299975 A JP 2000-200088 A JP 2001-224729 A

  An object of one aspect of the present invention is to provide a user who includes a singer in an environment such as karaoke to enjoy choreography and matching hands, and to provide a new evaluation function for choreography and matching hands. .

  In one embodiment, a model data storage unit that stores a plurality of model data that defines models of body movements in accordance with music, together with information on the playback position of the corresponding music, and playback of the music A sample data video reproduction unit that reproduces the plurality of sample data videos, a physical motion acquisition unit that acquires the physical motion of the user in association with the playback position of the music, and the hand The degree of coincidence of the physical motion with respect to the sample data is determined from the degree of coincidence between the data and each of the plurality of physical movements acquired at a predetermined music reproduction time interval including the music reproduction position of the sample data. A body motion that calculates an evaluation value of the body motion of the user based on a predetermined rule using the matching degree identifying unit and the identified degree of matching corresponding to each of the plurality of example data Evaluation value calculator , Body motion evaluation apparatus having a are provided.

  In an environment such as karaoke, it is possible to provide a user who includes a singer with an environment for enjoying choreography and matching hands, and to provide a new evaluation function for choreography and matching hands.

It is a functional block diagram of one Example. It is a figure which shows the flowchart of operation | movement of one Example. It is a figure which shows the example of the utilization form of a karaoke system. It is a figure which shows the example of the user interface displayed on a display apparatus. It is a figure which shows the example of the user interface displayed in another time. It is a figure which shows the several display system of a discrete model image | video. It is a figure which shows the example of the data regarding the some pose contained in one model data. It is a figure which shows the data of the body movement matched with model data. It is a figure which shows the example of operation | movement of a coincidence degree specific | specification part. It is a figure which shows the input / output of a rank provision part. It is a figure which shows the example of the hardware constitutions of an Example.

  Hereinafter, embodiments will be described with reference to the drawings. It should be noted that the following examples are for the purpose of understanding the invention and are not intended to limit the scope of the present invention. Also, the following embodiments are not mutually exclusive. Therefore, it should be noted that the elements of the embodiments are also intended to be combined unless a contradiction arises. Further, in the invention according to the method and the program described in the claims, the order of the processes may be changed as long as there is no contradiction, or a plurality of processes may be performed simultaneously. It goes without saying that these embodiments are also included in the technical scope of the invention described in the claims.

  The karaoke system is connected to each device to be described later in a wired or wireless manner, and can record image data, video data, character data, and audio data. It is possible to detect and analyze information about the singing and choreography of the singer and the viewer's hand, and perform scoring according to the result, and reproduce image data, video data, character data, and audio data.

  The image data, video data, character data, and audio data used in this embodiment may be recorded in the electronic device when the data is reproduced. Like so-called streaming playback, the karaoke system may receive necessary data each time playback is performed via a network line. Data may always be stored in a hard disk or flash memory built in the karaoke system.

  In the present embodiment, a voice input device such as a microphone is used to collect information on a matching hand by voice such as a call. The voice input device only needs to be able to collect voices uttered by the viewer as information. Or the microphone generally used in order to collect a singer's song as information in karaoke may be sufficient. In this embodiment, separate voice input devices can be used for singers and viewers. Alternatively, a single voice input device may be shared and used by the singer and the viewer.

  In this embodiment, an input unit (hereinafter referred to as “body motion acquisition unit”) for collecting information on choreography performed by a singer and matching hands performed by a viewer is used. The information collected by the body motion acquisition unit may include not only information collected by the body motion acquisition unit itself but also voice information collected by the voice input device. The body motion acquisition unit may be any device that can collect information on motions performed by a singer or viewer and detect it as a choreography or a match. A camera that captures and analyzes the motion of the human body to detect the motion content, or a sensor that is attached to the human body and detects the motion content of the human body from the change in position, moving speed, direction, or the like may be used. A button device that detects only which button the viewer has pressed at which timing may be used as the input device. An input apparatus does not ask | require a form. The body motion acquisition unit may share one device between the singer and the viewer. A plurality of body motion acquisition units may be assigned to each of the singer and the viewer. A physical motion acquisition unit may be assigned to each user individually.

  In the present embodiment, an instruction input device is used for the user to give instructions to the karaoke equipment regarding the playback method of video / image / character / sound and the like, the implementation method of various functions, and the like. The instruction input device is typically a touch panel device or an input device with operation buttons, but is not limited thereto.

  The display device used in the present embodiment only needs to have a screen that can display image data, video data, and character data transmitted from the karaoke system. For example, a liquid crystal display or the like is typical, but not limited thereto.

  The sound reproducing device used in the present embodiment may be any device that can reproduce sound data transmitted from the karaoke system as sound. For example, a speaker is typical, but the present invention is not limited to this.

  As the image data used in this embodiment, for example, a data format such as bitmap, JPEG, JPEG2000, GIF, and TIFF can be used. Other unique data formats may be used. Moreover, the data in which the parameter information for producing | generating the data of the image displayed by rendering in a karaoke system each time may be included.

  The audio data used in this embodiment includes, for example, data formats such as MP3, WAV, and AAC. Alternatively, data including parameter information for generating voice data to be reproduced by a sound source device built in the karaoke system, such as MIDI data, may be included.

  In the present embodiment, the music playback position can be used when playing audio data recording karaoke music. That is, the position information of the music piece can be calculated based on how much time has passed since the reproduction of the sound data of a specific sound included in the sound data has started (or the number of beats).

  In the model data, data information on the contents of images, videos, characters, and voices shown as explanations of the contents of choreography and matching hands to viewers, and information on the ideal music playback position for each of these contents were recorded. Contains data.

  The karaoke system reads the model data and reads it in accordance with the performance of the music, so that images, videos, characters, and voices explaining the choreography and matching hands to be performed by the user can be reproduced in accordance with the performance of the karaoke music.

  The body movement data is data including information related to the contents of choreography and matching hands performed by the user on the music and information associated therewith. Specifically, it includes information on what kind of content the choreographed or matched hand is performed on which piece of music by the user's physical motion acquisition unit at which point in the music reproduction time. In addition, information accompanying this (for example, the contents of the evaluation performed by the karaoke device on the contents of the choreography and the hand of the match, the personal information such as the number, sex, age, etc. of the users who performed the choreography and the hand of the match), etc. Information is included.

  Based on the received information, the karaoke system can generate body movement data of the choreography and the matching hand simultaneously with the demonstration of the choreography and matching hand by the user.

  The karaoke system refers to the body movement data, and the image data, video data, character data, and voice data corresponding to the choreography that the singer has performed and the contents of the hand that the viewer has performed are the singer and viewer. Is played at the same music playback position as the choreographer or collaborator, and the contents of the choreographer or collaborator's hand performed by the singer or viewer are simultaneously performed by the singer and viewer. Can be expressed or reproduced after the fact.

  The evaluation criteria set in the karaoke system do not necessarily have to be a single one, and a plurality of evaluation criteria (for example, the degree of strictness with respect to the model and factors that are important in evaluation) according to the skill of the user who performs choreography and matching. It is also possible to select from a different type.

  The comparison / evaluation between the body movement data and the model data by the evaluation function of the present embodiment can be performed not only after the music performance is completed but also at any time during the performance of the music. The contents of the evaluation can be notified to the user via the display device or the audio reproduction device. Thereby, the singer and the viewer can know the evaluation of their own choreography and the hand of the match at the same time as the demonstration of the choreography and the hand of the match, and the enthusiasm for the hand of the choreography and the match can be further increased.

  The user's demonstration of choreography and information on the contents of the hand of the match recorded in the body movement data, and the information on the content of the choreography and the hand of the match recorded in the model data, respectively, Items can be included. The content of such items may be anything that indicates human body movements. Specific examples include choreography at a specific music playback position and a pose of a matching hand. Further, it may be information about what kind of pose is used as the whole human body image. Also included are choreography and hand movement modes (for example, “clockwise rotation”, “jumping”, etc.) within a certain time range, and information on what music playback position these poses and movements are at. . However, the present invention is not limited to these.

  The evaluation function of the present embodiment compares the information of a certain item in the body movement data with the information of the item in the sample data, thereby comparing the model to the user's choreography and matching hands. It becomes possible to evaluate consistency.

  In evaluating the user's choreography and matching hands with the evaluation function of this embodiment, it is set in advance in the karaoke system which item of the body movement data and information on the contents of the choreography and matching hands included in the model data are compared. I can keep it.

  In addition, evaluation can be performed with the same model data regardless of the detection device used.

  In the evaluation function of the present embodiment, not only the evaluation for choreography and matching hands and the evaluation for singing are performed separately, but also as one content of the evaluation method that is set in advance in the karaoke system or determined according to the conditions specified by the user It is also possible to stipulate that the evaluation is performed in a comprehensive manner.

  For example, it can be set as one of the evaluation criteria how much the performance can be performed with the singing of the singer and the choreographer of the singer and the hand of the viewer. This function is a device for enabling a singer to evaluate whether or not the choreography can be demonstrated together with the singing. Also. It is a device for providing an effect that enhances the sense of unity between the singer and the viewer by evaluating how much the singer and the viewer can perform at the same time. In addition, for example, when the singing that the singer is highly appreciated after the viewer performs the matching hand, the matching hand has a positive influence on the singer's psychology and brought about a better singing. High appraisal can be performed even for good hands. In this way, the psychological effect of the hand of the hand that remains originally abstract is presented to the user as a concrete thing, and the viewer is more strongly aware, thereby providing an effect that the viewer is more motivated to do the hand of hand. You can also.

  The central management device can total the results of the evaluations on the choreography and the performance of the matching hand recorded in the physical motion data transmitted from the karaoke system, and can rank and transmit the results to the karaoke system. The karaoke system that has received the ranking information can also show the user the overall ranking of the choreographed choreography and the matching user. The tabulation of such evaluation can be performed on all the body movement data of music pieces that have been choreographed or matched. In addition, the attribute information included in the accompanying information of the body movement data (for example, by the difficulty level of choreography and matching hands, by user gender / age, by area of the karaoke system, etc.) is narrowed down, and the rank in each attribute information Can also be shown. Thereby, the user can know the position of the choreography and matching hands that he has demonstrated from the multifaceted viewpoint.

  The evaluation function of this embodiment changes the reproduction method and reproduction contents of the model data as needed according to the conditions set in advance in the karaoke system or specified by the user in accordance with the evaluation of the choreography and matching hands demonstrated by the user. Can continue. For example, when singing a karaoke song, if the singer evaluates it as being demonstrated as choreographed as an example, he can perform the example according to the example by changing the color of the example image. So that the singer can understand at a glance, or if the viewer ’s match-up timing tends to be faster than the model, the voice playback device will play “Slightly Slower!” It is possible to guide the viewer to a matching hand as shown in the example.

  In this example, with respect to the choreography and matching hand information recorded in the model data, which movement of the choreography and matching hand corresponds to an important part (hereinafter referred to as a “knob”) for understanding and evaluating the movement of the human body. Can be identified.

  As a method for automatically determining information on the location or range of a knot, for example, there may be a location where music is expected to swell as a rule of thumb. As an example, “locations where choreography and movements that are stationary for a certain period of time are set as examples” “locations where the sound pressure of the song's sound is particularly high” “ranges where the same phrase in the song is repeated and song titles And the like "where the same phrase is used. In addition, if the model data includes 3D motion data that expresses the content of the model (information that indicates how to move model data produced by 3D computer graphics on the screen) Examples of the position of the pose, which is a feature of the motion based on the change of the coordinate position of the joint of the model data defined in the data, are given. For example, when all or most of the joints remain stationary for a certain period of time, when the direction of joint movement changes significantly, or when the joint position approximates information registered in advance as an example of a joint pose Thus, the location can be a turning point.

  When the karaoke system reproduces image data, video data, character data, and audio data included in the model data, it is not always necessary to reproduce all the information included in the data. When set in advance in the karaoke system or when instructed by the user, only a part or all of the part corresponding to the movement of the part of the example of choreography or matching is based on the information of the part included in the example data The contents of can also be played. Thereby, the user can perform the performance by focusing only on the movement of the knots, which is a particularly important point in learning the choreography and matching hands, and can efficiently learn the choreography and matching hands.

  In the present embodiment, by utilizing the information of the nodes included in the model data, it is possible to reproduce the contents of the choreography and the matching hand recorded in the model data by various expression methods. For example, when playing a choreography or matching hand model, or expressing a user's choreography or matching hand with video or audio, highlight the image / video / character / sound that shows the choreography or the matching hand. be able to. For example, to increase the image / video / text size and sound pressure, to change the color / display color of the image / video / text, and to enhance the image / video / text of the model Can be added and displayed.

  When evaluating choreography and matching hands, it is possible to determine in advance in the model data how to use the knot information regarding whether or not the choreography or matching hand of the part corresponding to the knot matches the model.

  The sample data used for one piece of music does not necessarily have to be one, and any of a plurality of sample data recorded in the karaoke system is used when the user performs a choreography or a matching hand. Whether to reproduce the action of the model can be determined in advance in the karaoke system or in accordance with conditions designated by the user. Thereby, for example, in the case where a singer or viewer is skilled in choreography and matching hands, it is possible to use sample data in which choreography and matching hands that are difficult to perform are recorded. In addition, if the singer or viewer is a beginner who is a choreographer or collaborator, it is possible to use sample data that records the choreographer and the collaborator that are simpler and easier to perform. It will be possible to enjoy choreography and matching hands at the optimal difficulty level.

  The karaoke system connected to the central management apparatus in which the body movement data is accumulated, when playing the karaoke music, to the central management apparatus to transmit the body movement data recording the choreography and matching hands performed on the music Can be requested. The karaoke system that has received the physical motion data from the central management device reproduces image data, video data, character data, and voice data corresponding to choreography and matching hands based on information recorded in the physical motion data. It is possible to reproduce the choreography and matching hands that the user demonstrated. This makes it possible to reproduce the choreography performed by other singers as the singer performs the choreography, for example, to simulate whether the singer can reproduce the evaluation of the choreography or to the viewer Is to reproduce the hand of the match performed by the singer as the singer sings and choreographed, giving the singer an uplifting feeling as if the viewer was in front of them and sending the hand of the match to their singing Can be realized. Note that the body movement data collected by the central management unit does not necessarily have to be based on information on the choreography or hand of the hand actually demonstrated by the user. -It may be data produced by artificially adjusting the data values so that sound is reproduced.

  Even if there are a plurality of physical motion data candidates and it is left to the user to select which physical motion data is to be used, it is necessary to satisfy the user to select the physical motion data by setting the karaoke system in advance. These conditions can be set. As a result, for example, it is possible to provide choreography or matching hands that can be used only by users who have a history of singing a certain number of times, or choreography or matching hands that can only be used by users who have purchased the right to use separately. It becomes possible to correspond to the business model.

  How the karaoke system expresses the video / image / text / sound that is played back according to the choreography and the contents of the user's demonstration recorded in the body movement data is changed according to the designation from the user can do. Specific examples of changing the playback method include a method for allowing the user to specify data to be played back from candidates of video data, image data, character data, and audio data recorded in the karaoke system, The type of processing you want to perform on image data, text data, or audio data (for example, for video, image, or text, you can enlarge or reduce the display size, change the color tone, or add an animation effect. The method of designating a change of the above is assumed, but it is not limited thereto. Further, by allowing the user to specify a combination of a plurality of these methods at the same time, it is possible to provide more various reproduction methods.

  FIG. 1 shows a functional block diagram of one embodiment.

  FIG. 1 shows an embodiment in which the body motion evaluation device 100 is included in the karaoke system 150. It should be noted that the body motion evaluation apparatus 100 does not necessarily have to exist in the karaoke system 150. Therefore, the body motion evaluation apparatus 100 may be used alone.

  In addition, the present embodiment includes a central management device 170 configured with a server or the like. The karaoke system 150 is connected to the central management device 170 via a network (not shown). For example, the Internet may be used as the network.

  The body motion evaluation apparatus 100 includes a model data video reproduction unit 102, a model data storage unit 106, an effect expression unit 108, a body motion acquisition unit 110, a coincidence degree identification unit 112, and a body motion evaluation value calculation unit. 116 and a simulated playback unit 118.

First, the model data video reproduction unit 102 displays, on a display device (not shown), a model video suitable for choreography model data matched with, for example, karaoke music. Examples of choreography matched to karaoke music include the following. In this specification, “singer” and “viewer” are collectively referred to as “user”.
(1) Physical movements such as dances performed by users (mainly singers) (hereinafter referred to as “choreography”)
(2) User's (mainly viewer) shouts, choruses, applause, etc. (hereinafter referred to as “match hands”).

In this specification, the data defining the choreography and matching hands are referred to as “example data”.
A detailed example of the model video reproduced by the model data video playback unit 102 will be described later with reference to FIGS.

The model data video reproduction unit 102 may include a model data video generation unit 104. The model data video generation unit 104 can display a model video in synchronism with music using the model data 107 stored in the model data storage unit 106. The model image may be both or one of the following two images.
(A) Video that smoothly moves in synchronism with music playback as the user performs the operation.
(B) A video that represents a characteristic motion part or a motion part that becomes a point to be imitated among the actions performed by the user, and is displayed discretely in synchronization with music reproduction.

  The example video of (a) may be generated in advance as video data and combined with music. Examples of the format of video data in which music and model video are combined include MPEG2, H.264, and so on. H.264 / AVC, H.H. 265 / HEVC and the like, but are not limited thereto.

  The model data 107 is data that defines the movement of the body according to the music. The model data can also be used to calculate an evaluation value of the body motion by being compared with the user's body motion in the coincidence degree specifying unit 112 described later. A specific example of the model data will be described later with reference to FIG.

  The model data 107 does not have to be created at a predetermined time interval like a frame in video data. For example, it may be data in which a pause or the like at a turning point (hereinafter referred to as “node”) in choreography is associated with a music playback position. Moreover, the data which defined the timing of applause as a music reproduction position may be sufficient.

  The model data video generation unit 104 reads the model data 107 in order to reproduce the model video of (a) above, interpolates choreography at the music playback position in the middle of each node, and smooth video Data may be generated. The generated video data is played back in synchronization with the music playback.

  The display of the model video in (b) can be realized by displaying the data stored in the model data 107 in synchronization with the music without interpolation.

  As described above, the user can appropriately perform the body movement by watching either one or both of the smooth model image (a) and the discretely displayed model image (b). The example video displayed in this way can be used for gymnastics education as well as application to a karaoke system.

  The body motion acquisition unit 110 acquires data of a user's body model from the user's video imaged by a camera or the like. As model data, a skeletal model composed of body bones and joints may be used. For example, Microsoft's Kinect (registered trademark) can acquire object depth information along with the object image. For example, by using this Kinect, a user's motion capture can be easily performed without wearing a special suit with a marker. By using Kinect, the motion of the skeleton model of the user's body movement can be acquired in real time. Note that this embodiment does not necessarily require the use of Kinect. Other motion capture techniques may be used.

  The coincidence degree specifying unit 112 is acquired, for example, within a predetermined time interval (music reproduction interval for comparison) including one example pose included in the example data and the music reproduction position of the example pose. It is possible to compare the motion capture data of the physical movements of a plurality of users. Details of this comparison operation will be described later with reference to FIGS.

  Therefore, a plurality of motion capture data included in a predetermined music reproduction interval for comparison is compared with one model data. The predetermined music reproduction interval for this comparison may be set in advance by default or may be designated by the user. For example, a comparison result having the highest matching degree among the plurality of comparison results may be adopted as the matching degree. In this way, the user's physical motion is used to absorb the deviation from the correct music playback position by using the plurality of user motion capture data acquired within a predetermined music playback interval for comparison. be able to. That is, by adjusting the predetermined music playback interval for comparison, if the user's action shows a high degree of agreement with the model data in the predetermined music playback cabinet for the comparison, It can be determined that the video and the physical motion of the user match. For this reason, the predetermined music reproduction interval for comparison can adjust the severity of the evaluation of the deviation of the motion timing in the physical motion evaluation value calculated thereafter.

  For example, for users who are not good at dancing, the specified video playback interval for comparison is lengthened, so that even if the sample video and the user's physical movement are misaligned, they match. The probability that it is determined that the That is, by adjusting a predetermined music playback interval for comparison, the handicap can be adjusted for evaluation for each user.

  The body motion evaluation value calculation unit 116 selects, for example, the highest comparison result from among the plurality of comparison results at one node of the model data from the matching degree identification unit 112. The selected degree of coincidence is adopted as the degree of coincidence between the model data at that node and the body movement. Alternatively, an average value of a plurality of comparison results at one node may be calculated and used as the degree of coincidence between the model data at the node and the body movement.

  Then, the body motion evaluation value calculation unit 116 calculates, for example, a weighted average value of the degree of coincidence selected at each node. Details of this weighting calculation will be described later with reference to FIG. By calculating the average value of the degree of coincidence by weighting, the importance level of the action can be defined for each action node. For example, in the body movement (dance), the influence on the evaluation of the determined pose can be increased by increasing the weight at the node of the determined pose.

  Note that the evaluation value includes an evaluation value of physical movement (pose), an evaluation value of a matching hand, and the like. In the evaluation of the mating hand, an evaluation value of the physical action of the mating hand and an evaluation value related to timing such as applause are included. For evaluation values related to timing such as applause, simply calculate the evaluation value of the matching hand by averaging (weighted) the difference between the music playback position of each node in the model data and the music playback position of the applause. Can do.

  In addition, the timing of the matching hand may be transmitted to the body motion evaluation apparatus 100 by a button, an acceleration sensor, or the like of the mobile terminal 370 possessed by the user. Then, the coincidence degree identification unit calculates the evaluation value of the matching hand by averaging (weighting) the difference between the music playback position of each node of the model data and the music playback position at the timing of the transmitted matching hand. Also good. Note that a wireless network such as Bluetooth (registered trademark) or WIFI can be used for connection between the portable terminal 370 and the body motion evaluation apparatus 100. Alternatively, the timing of a matching hand may be transmitted to the physical motion evaluation apparatus 100 using a karaoke music input device.

  The physical motion evaluation value calculation unit 116 can send the calculated evaluation value to the central management device 170 via the network.

  The effect expression unit 108 receives information on body motion from the body motion acquisition unit 110. A predetermined video expression, sound, etc. can be reproduced in accordance with the received body movement. For example, if the physical motion of the user's hand is applause, the effect expression unit 108 generates a sound of applause or other sound from a speaker (not shown) at the timing when both hands touch. Also good.

  Also, the effect expression unit 108 displays a video effect such as playing a predetermined effect video or changing the background video of the screen on the display device when the body motion is in a specific posture. It may be.

  The simulated playback unit 118 uses, for example, the motion capture data of the body motion corresponding to the degree of coincidence specified by the coincidence degree specifying unit and the music playback position, and the video in which the character simulates the user's motion Can be generated. For this reason, it is desirable that the coincidence degree specifying unit 112 stores the motion capture data of the user's physical motion at each node and the music playback position at that time in the storage device as the user's physical motion data. Further, the stored physical motion data of the user may be transferred to another karaoke system 150 via the central management device 170. By transferring physical motion data to another karaoke system 150, it is possible to reproduce the physical motion of another person along with the music playback position on the karaoke system 150.

  The central management device 170 can be realized by a computer such as a server. The central management device 170 includes a collection unit 172, a rank assigning unit 174, and an information providing unit 176.

  The central management device 170 is connected to a plurality of karaoke systems 150 via a network.

  The collection unit 172 collects evaluation values of physical movements, evaluation values of matching hands, and the like. In addition, the collection unit 172 may collect evaluation values related to singing from the karaoke system 150. Note that the evaluation value related to the singing is calculated by combining the degree of coincidence of the pitch, the degree of coincidence with the rhythm, the degree of coincidence between the singing expression based on the strength of the singing and the template, and the like. Since evaluation techniques related to singing are well known to those skilled in the art, the description of this specification is omitted. The sleeve part may collect user's body movement data.

  The rank assigning unit 174 receives, from the collection unit 172, an evaluation value of physical movement, an evaluation value of a good hand, an evaluation value of singing, and the like. For example, the ranking assigning unit 174 calculates an average value by applying a predetermined weight to these evaluation values, and calculates a comprehensive evaluation. A default value may be set for the predetermined weight. Or a weight may be determined by a user's setting.

  Then, the rank assigning unit 174 ranks the user or the group of users based on the comprehensive evaluation value, and provides it to the information main part. Further, the rank assigning unit 174 may rank each of the evaluation value of the body movement, the evaluation value of the match, and the evaluation value of the singing.

  The information providing unit 176 transmits the rank obtained from the rank assigning unit 174 to an appropriate karaoke system 150 among the plurality of karaoke systems 150. In addition, the information providing unit 176 may transmit the collected body movement data of the user to the karaoke system 150 that the collecting unit 172 needs.

  As described above, in the embodiment shown in FIG. 1, the evaluation value of the body movement and the evaluation value of the matching hand are appropriately calculated, and a karaoke system with higher entertainment is provided by using these evaluation values. be able to.

  FIG. 2 shows a flowchart of the operation of one embodiment. FIG. 2A shows an outline of the operation of one embodiment.

  In step S202, the model data video generation unit 104 plays a model video for the user. The user can perform an appropriate operation at an appropriate timing by viewing the example video. Note that, as described above, the model video is (a) a video that smoothly moves in synchronization with the music playback, and (b) a characteristic motion part of the user's motion, or is imitated. It is a video that represents a motion part as a point, and is either or both of video that is discretely displayed in synchronization with music reproduction.

  In step S204, the body motion acquisition unit 110 acquires the user's body motion. The acquired body motion can be acquired as motion capture data of a skeletal model composed of bones and joints, for example. An example of a system that acquires motion capture data is a system using Kinect (registered trademark) as described above. In addition, acquisition of the body movement in a present Example is not limited to this.

  In step S206, the coincidence degree specifying unit 112 compares the model data with the acquired physical action. In this comparison, data of a plurality of physical movements acquired within a music playback interval for comparison before and after the music playback position (node) of one data included in the model data can be used. In this way, by expanding the range to be compared, it is possible to make a comparison in consideration of a shift in the timing of body movement. One piece of data included in the sample data is compared with data of a plurality of physical movements acquired within the music reproduction interval for comparison, and the one with the highest degree of matching is selected as the degree of matching. Then, the body motion corresponding to the selected degree of coincidence and the music playback position of the body motion are stored. The stored body motion and the body motion data related to the music playback position can be used when the character simulates the user's body motion later. Since the stored physical motion data related to the physical motion and the music playback position are discrete data, the amount of data is smaller than that of video data or the like. For this reason, this physical motion data can also reduce the transfer cost also when doting another karaoke system via a network. In addition, the cost for storage can be reduced.

  In step S208, the physical motion evaluation value calculation unit 116 calculates the physical motion evaluation value and / or the evaluation value of the matching hand. In calculating the evaluation value, a more appropriate evaluation value can be calculated by calculating a weighted average value using a predetermined weight at each node. For example, by increasing the weight of the node corresponding to the determined pose, an evaluation value that places importance on the determined pose can be calculated.

  It should be noted that the body motion evaluation value calculation unit 116 can calculate the evaluation value at any time even during the music and can display it.

  In step S210, for example, various evaluation values are collected in the collection unit 172 of the central management apparatus 170. The ranking assigning unit 174 of the central management device 170 performs ranking of users or ranking of user groups using various collected evaluation values.

  In step S212, the karaoke system 150 displays the ranks obtained from the central management device 170 and various evaluation values.

  By the above operation, a user who is a singer, for example, can enjoy the dance operation together with the music. Moreover, the user who is a viewer can enjoy a match. Moreover, the evaluation value and / or ranking information can be obtained, and both the singer and the viewer can enjoy karaoke together.

  FIG. 2B is a flowchart showing the operation performed by the matching degree selection unit 114.

  In step S252, the coincidence degree selection unit 114 initializes the coincidence degree α and the music reproduction position β. For example, 0 may be substituted for α and β for initialization.

  Step S254 forms a loop process with step S260. Of the user actions, the user pose will be taken up and described below. In the loop, loop processing is performed for each of N user poses acquired in the music playback interval for comparison of Δt before and after one node.

  In step S256, the matching level selection unit 114 determines whether there is a user pose having a higher matching level with the example pose than α. If “yes”, the process proceeds to step S258. If “no”, the process proceeds to step S260.

  In step S258, the coincidence degree selection unit 114 substitutes the user pose coincidence degree for α. The music playback position of the user pose is substituted for β.

  By repeating the above operation, the matching degree of the user pose having the highest matching degree between the model pose and the user pose can be acquired as the value of α. Also, the music playback position of the user pose with the highest degree of coincidence can be acquired from the value of β.

  FIG. 3 shows an example of how the karaoke system is used. In the different drawings, elements denoted by the same reference numerals indicate the same elements.

  The karaoke system 150 includes a camera 302, a display device 310, and a body motion evaluation device 100. The camera 302 is connected to the body motion evaluation apparatus 100.

  Further, the microphone 322 and the microphone 324 are connected to the body motion evaluation apparatus 100.

  The display device 310 displays a model image 312 and a matching hand image 314.

  A singer 352 and a viewer 354 are present in front of the microphone 322 and the microphone 324, respectively. The camera 302 images the actions of the singer 352 and the viewer 354.

  The portable terminal 370 is in a state where it can communicate with the body motion evaluation apparatus 100 by wireless communication.

  The physical motion evaluation apparatus 100 is connected to the central management apparatus 170 via the network 360.

  In FIG. 3, the singer 352 can imitate the operation of the model video 312 by viewing the model video 312. The viewer 354 can imitate the action of the match hand by imitating the action of the match hand image 314.

  Since the motion of the singer 352 is captured by the camera 302, the body motion evaluation apparatus 100 can calculate an evaluation value indicating how much the motion of the singer 352 matches the example video 312. it can.

  Since the motion of the viewer 354 is captured by the camera 302, the body motion evaluation apparatus 100 can calculate an evaluation value indicating how much the motion of the viewer 354 matches the matching hand image 314. .

  Further, the effect expression unit 108 can output a sound effect from a speaker (not shown) according to the action of the hand of the viewer 354.

  Further, when another viewer (not shown) imitates the matching hand image 314 and shakes the mobile terminal 370, information from the acceleration sensor built in the mobile terminal 370 is transferred to the body motion evaluation device via wireless communication. 100. The body motion evaluation apparatus 100 recognizes the signal from the acceleration sensor as the motion of the matching hand. Then, based on the signal from the portable terminal 370, evaluation of a match is performed. As described above, when a plurality of viewers are performing the matching hand motion, two evaluation values can be calculated for the matching hand. The evaluation values of these two matching hands may be calculated as an average value, for example, and may be aggregated into the evaluation values of one matching hand.

  As described above, the physical motion evaluation apparatus 100 acquires the evaluation value of the physical movement and the evaluation value of the matching hand, and transmits them to the central management apparatus 170.

  The central management device 170 performs comprehensive evaluation based on at least the received evaluation value of physical movement and the evaluation value of the matching hand. Note that the singing evaluation value may be further used in the calculation of the comprehensive evaluation. The central management device 170 calculates the rank of the singer 352 based on the information acquired from the plurality of karaoke systems 150. Further, the rank of the user group including the singer 352 and the viewer 354 may be calculated. The central management device 170 transmits these rankings and comprehensive evaluation values to the karaoke system 150.

  The karaoke system 150 displays the received ranking and comprehensive evaluation value on the display device 310 (not shown).

  As described above, a karaoke system capable of performing a comprehensive evaluation in consideration of not only a singing evaluation value but also an evaluation value of a singer's physical movement and / or an evaluation value of a matching hand is realized.

  FIG. 4 shows an example of a user interface 400 displayed on the display device 310.

  The display 410 is a discrete model image. For example, the character 412a is displayed darker than the character 416a. A darkly displayed character indicates a past example pose. A lightly displayed character indicates a future model pose. The character 414a has a dark portion and a thin portion. That is, the character 414a shows the current example pose. In this way, the darkness of the character changes with time. By following this change, the user can easily force the current pose and the future pose. By such display, the user can easily imitate the model operation.

  The character 450a is a model image that performs a smooth motion. This model image 450 may be generated in advance. Alternatively, as described above, the model data may be interpolated, and the three-dimensional character may be moved and displayed in real time. The user can also perform his / her body movement by imitating a smooth model image.

  A display 420 is a smooth model image of a matching hand. The character 440a indicates that the hands are put together. The viewer can perform the body motion of the matching hand by imitating the smooth model image of the matching hand.

  FIG. 5 shows an example of a user interface displayed at a time later than FIG.

  In the display 410, characters 412b and 414b are displayed darkly. A part of the character 416b is displayed darkly. Therefore, it can be seen that the current example pose is the character 416b.

  The character 540b poses the character face figure 416b.

  A display 420 is a smooth model image of a matching hand. The character 440b shows the moment when both hands are released. The viewer can perform the body motion of the matching hand by imitating the smooth model image of the matching hand.

  FIG. 6 shows a plurality of display modes of discrete model images.

  In FIG. 6A1, the character 610-A1 is displayed darkly. Therefore, the character 610-A1 shows a pause that has already ended. Since the character 612-A1 is displayed lightly and exists at a position far from the character 610-A1, it can be seen that it is a future pose.

Also, as shown in 620-A1, the portion of the lyrics that has been sung is displayed darkly. From this display, the part of the lyrics to be sung at present can be understood. FIG. 6 (A2) is the same as FIG. 6 (A1), the character position is not changed, and the character density is changed. That is, the video shown in FIGS. 6A2 and 6A1 is a display method that can identify past, current, and future poses using changes in the darkness of the character. The character 610-A1 is similarly displayed darkly. The character 612-A2 is also darkly displayed, and it can be seen that this pause has ended just before the current time.

  Further, as shown in the lyrics 620-A2, the portion of the lyrics that has been sung is displayed darkly. From this display, you can see the lyrics that should be sung.

  FIGS. 6B1 and 6B2 show a display method in which the current character's pose can be known by moving the character row to the left instead of the character density. The leftmost character indicates the pose that ended immediately before. Character 610-B1 shows the pose that was ended immediately before. Since the character 612-B1 is located at a position far away from the character 610-B1, it can be seen that it is a future pose.

  In FIG. 6 (B2), the position of the character moves to the right depending on the passage of time, as compared to FIG. 6 (B1). Character 612-B2 shows the pose that has just ended.

  The video shown in FIG. 6B2 and FIG. 6B1 is a display method that can identify past, present, and future poses by using the leftward movement of the character.

  The lyrics 620-B1 and the lyrics 620-B2 are the same as the display method of the lyrics 620-A1 and the lyrics 620-A2.

  As described above, the pose change can be displayed in an easy-to-understand manner.

  FIG. 7 shows an example of data relating to a plurality of poses included in one example data.

  The music playback position axis 710 indicates the position axis in music playback, and corresponds to the time axis. In the case of music, the music may be played by changing the tempo of the music. For this reason, instead of the time axis, it is advantageous to use an axis based on the music playback position that remains unchanged even if the tempo is changed.

  A numerical value 702 indicates, for example, the beat count from the beginning of the music.

  The example pose 720 indicates one of a plurality of example poses included in the example data. As the model pose, for example, a skeletal model created by a combination of a bone and a joint that joins the bone can be adopted.

  The music playback position 722 indicates the music playback position corresponding to the example pose 720. In this specification, it is synonymous with the position of a joint.

  A weight 724 indicates a weight corresponding to the example pose 720. The weight 724 is used as a weight when calculating the weight average of the degree of coincidence corresponding to a plurality of example poses (720, 730, 740, 749).

  The model data is configured as described above. Further, the model data has information indicating which music data corresponds to which piece of music. In the example of FIG. 7, the data related to the pose has been described in a simplified manner. There are two types of poses: dance poses used mainly by singers and matching hand poses used by viewers. The music playback position of the dance pose and the match pose need not be the same. Further, the sample data may include data related to clapping timing. Therefore, various data can be included in the model data.

  FIG. 8 shows physical movement data 750 corresponding to the model data.

  The body motion data 750 acquired by the body motion acquisition unit 110 is acquired at predetermined intervals. The acquired motion is recognized, for example, in the form of a skeleton model. Then, corresponding to each example pose (720, 730, 740, 749), a plurality of physical movement data acquired during the music reproduction position interval Δt for comparison before and after the position of the knot are compared. Selected as a candidate.

  For example, for the model pose 730, physical motion data (751, 752, 753, 754, 755) acquired during the music reproduction position interval Δt for comparison is selected.

  FIG. 9 shows an example of the operation of the coincidence degree specifying unit 112.

  FIG. 9 shows the example pose 730 shown in FIG. 8 and the data (751, 752, 753, 754, 755) of physical movements acquired at the music playback place interval for comparison of Δt.

  The coincidence degree specifying unit 112 first calculates the degree of coincidence between the model pose 720 and the body movement data 751. A pattern recognition technique well known to those skilled in the art can be used for calculating the degree of coincidence. In the case of the examples in this specification, the degree of coincidence between the skeleton models is calculated. Specifically, the degree of coincidence can be calculated by calculating the distance between the corresponding joints and the difference in the corners of the bone in the joint, for example, by summing the absolute values of the differences. If the reciprocal of this absolute value is defined as the degree of coincidence, the greater the degree of coincidence, the higher the degree of coincidence.

  The coincidence degree specifying unit 112 similarly calculates the degree of coincidence with the model pose 730 for the body movement data 752, 753, 754, and 755.

  Then, the coincidence degree selection unit 114 selects the body movement data indicating the highest degree of coincidence among the coincidence degrees respectively corresponding to the body movement data (751, 752, 753, 754, 755). The highest one is adopted as the selected degree of coincidence. In addition, the selected physical motion data and the music playback position of the selected physical motion data are stored.

  As described above, the degree of coincidence is selected corresponding to each of the plurality of example poses. As described above, a weight is set for each example pose. Using this weight, a weighted average value of a plurality of matching degrees corresponding to a plurality of example poses is obtained. The weighted average value can be calculated even in the middle from the start of music to the current time. Therefore, the weighted monthly average value at the current time may be displayed on the display device as the evaluation value at the current time. When the music ends, the final evaluation value is calculated. The evaluation value may include an evaluation value of physical movement and an evaluation value of a good hand.

  As described above, the body movement evaluation value calculation unit 116 can calculate the evaluation value of the body movement and the evaluation value of the matching hand.

  FIG. 10 shows input / output of the rank assigning unit 174.

  The rank assigning unit 174 receives the physical motion evaluation value 1002, the matching hand evaluation value 1004, and the singing evaluation value 1006. For example, the rank assigning unit 174 may simply average a plurality of input evaluation values. Alternatively, the average value may be calculated with a predetermined weight. Alternatively, a specific input value may be ignored and the evaluation value may be calculated using a predetermined calculation algorithm. The calculated evaluation value is output as a comprehensive evaluation value 1010. Further, the ranking 1012 is calculated and output for each music piece.

  Note that a refinement instruction 1008 may be input from the karaoke system 150. For example, when a prefecture such as Tokyo is specified and it is desired to obtain the ranking, the karaoke system 150 transmits a narrowing instruction 1008 to that effect. The rank assigning unit 174 calculates the rank based on the narrowing down instruction and outputs it.

  FIG. 11 shows an example of the hardware configuration of the embodiment.

  The hardware includes a CPU 1102, a ROM 1104, a RAM 1105, a drive device 1110, an input / output interface 1120, a display unit 1130, an acoustic control unit 1140, and a communication unit 1150. Each component is connected by a bus 1170.

  For example, an input device 1121 such as a keyboard, a microphone 1122, a camera 1123, and a distance sensor 1124 can be connected to the input / output interface 1120.

  A display can be connected to the display unit 1130. A speaker can be connected to the acoustic control unit 1140. A mobile terminal 370, a pad (tablet terminal), and a wired / wireless LAN can be connected to the communication unit 1150.

  A portable storage medium 1112 can be inserted into the drive device 1110.

  It should be noted that all or part of this embodiment can be implemented by a program. This program can be stored in the portable storage medium 961. Illustrative examples of the portable storage medium 961 include a magnetic recording medium, an optical disk, a magneto-optical recording medium, and a nonvolatile memory. Magnetic recording media include HDDs, flexible disks (FD), magnetic tapes (MT) and the like. Examples of the optical disk include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc-Read Only Memory), and a CD-R (Recordable) / RW (ReWriteable). Magneto-optical recording media include MO (Magneto-Optical disk). Non-volatile memory includes USB memory and SD memory. A program stored in the portable storage medium 961 is read and executed by the CPU, whereby all or part of the embodiment of the present invention can be implemented.

  Further, the electronic device provided with the present embodiment does not need to be configured by a single device, and is a set of a plurality of electronic devices having a function of exchanging information with other electronic devices by wire or wireless. Also good.

  Further, the present embodiment is provided to an electronic apparatus having a function of recording and reproducing image / video / character / audio data.

  The evaluation function for choreography and matching hands according to the present embodiment can be installed in the karaoke system together with the existing evaluation function for singing by a singer and can be used simultaneously.

DESCRIPTION OF SYMBOLS 100 Body motion evaluation apparatus 102 Example data image | video reproduction | regeneration part 104 Example data image generation part 106 Example data memory | storage part 107 Example data 108 Effect expression part 110 Body motion acquisition part 112 Matching degree specific | specification part 114 Matching degree selection part 116 Body Motion evaluation value calculation unit 118 Simulated playback unit 150 Karaoke system 170 Central management device 172 Collection unit 174 Order assignment unit 176 Information provision unit

Claims (11)

A sample data storage unit that stores a plurality of sample data defining a model of body movement according to the music, together with information on the reproduction position of the corresponding music;
A sample data video playback unit that plays back the video of the plurality of sample data in synchronization with the playback of the music;
A physical motion acquisition unit that acquires the physical motion of the user in association with the reproduction position of the music;
The degree of coincidence of the physical motion with respect to the sample data from the degree of coincidence of the sample data and each of the plurality of physical movements acquired at a predetermined music reproduction time interval including the music reproduction position of the sample data A degree-of-matching identifying unit,
A body motion evaluation value calculation unit that calculates an evaluation value of the user's body motion based on a predetermined rule using the identified degree of matching corresponding to each of the plurality of model data;
A body movement evaluation apparatus having The coincidence degree specifying unit
A degree of coincidence selecting unit that selects the highest degree of coincidence among the degree of coincidence between the example data and each of the plurality of body movements as a body movement and a degree of coincidence corresponding to the example data; ,
The body movement evaluation apparatus according to claim 1. An effect expression unit that reproduces a predetermined effect image or a predetermined effect sound based on the acquired physical movement of the user;
The body motion evaluation apparatus according to claim 1, further comprising: The predetermined rule is a weighted average of the degree of coincidence corresponding to each of the plurality of example data, and a predetermined weight for the weighted average is determined for each of the plurality of example data. ing,
The body movement evaluation apparatus according to any one of claims 1 to 3. The example data video reproduction unit uses a plurality of interpolation data obtained by interpolating each of the adjacent example data together with the plurality of example data, and the character data moving smoothly corresponding to the music reproduction position. Including a model data video generation unit for generating a model action video,
The body movement evaluation apparatus according to any one of claims 1 to 4. A simulated playback unit that simulates and plays back a user's body motion as a video image of a character, using the body motion corresponding to each of the plurality of example motions and a music playback position of the body motion;
The body motion evaluation apparatus according to claim 2, further comprising: The model data includes at least a model of physical movement and a model of a matching hand,
The physical motion evaluation apparatus according to claim 1, wherein the evaluation value includes an evaluation value of physical motion and an evaluation value of a good hand.   The karaoke system including the body motion evaluation apparatus according to claim 7, wherein the evaluation value of the physical motion and / or the evaluation value of the matching hand influences a singing evaluation value of karaoke. 9. A central management device to which a plurality of karaoke systems according to claim 8 are connected, wherein at least a collection unit that collects evaluation values of the physical movements and / or evaluation values of the matching hand from the plurality of physical movement evaluation devices; ,
A rank assigning unit for assigning a rank to a user based on the collected evaluation value of the physical movement and / or the evaluation value of the matching hand;
An information providing unit that provides the ranking, the evaluation value of the physical movement and / or the evaluation value of the matching hand to a predetermined karaoke system;
A central management device.   The program for functioning a computer as a karaoke system of Claim 8.   A program for causing a computer to function as the central management device according to claim 9.

Images