JP2015097639A - Karaoke device, dance scoring method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that it is impossible to provide a Karaoke device capable of performing dance scoring with a high degree of precision.SOLUTION: A Karaoke device that performs dance scoring with a high degree of precision includes: a photographing part for photographing a singing person and acquiring a singing person's image which is an image taken of the singing person; a skeleton information acquisition part for acquiring skeleton information which is the information indicating the singing person's movement using information contained in the singing person's image; and a score output part for outputting a score which is the result of scoring the singing person's movement using the skeleton information.

Description

  The present invention relates to a karaoke apparatus and the like.

  Conventionally, a karaoke apparatus has been developed (see Non-Patent Document 1).

“JOYSOUND f1 JS-F1”, [online], XING Inc. [searched on September 10, 2013], Internet [URL; http://joysound.biz/product/online/f1/index.html]

  Conventionally, it has not been possible to provide a karaoke apparatus that accurately scores a dance.

  The karaoke apparatus according to the first aspect of the present invention uses a shooting unit that captures a singer and obtains a singer image that is an image of the singer, and information included in the singer image, and moves the singer. It is a karaoke apparatus provided with the skeleton information acquisition part which acquires the skeleton information which is information which shows, and the score output part which outputs the score which is the result of scoring a singer's movement using skeleton information.

  With such a configuration, it is possible to provide a karaoke apparatus that accurately scores a dance.

  Moreover, the karaoke apparatus of this 2nd invention is a singer image is an image containing distance information with respect to 1st invention, and a skeleton information acquisition part uses distance information contained in a singer image. It is a karaoke device that acquires skeleton information.

  With such a configuration, it is possible to provide a karaoke apparatus that accurately scores a dance.

  Moreover, the karaoke apparatus of this 3rd invention is a karaoke apparatus with which the information contained in a singer image is distance information corresponding to a singer's area | region among distance information with respect to 2nd invention.

  With such a configuration, it is possible to provide a karaoke apparatus that accurately scores a dance.

  Moreover, the karaoke apparatus of this 4th invention is a motion determination information storage part in which the motion determination information which is the information for determining a singer's movement is stored with respect to any one of the first to third inventions. And a score calculation unit that determines the movement of the singer indicated by the skeleton information using the movement determination information and the skeleton information, and calculates a score using the determination result, and the score output unit calculates the score. It is a karaoke apparatus that outputs the score calculated by the department.

  With such a configuration, it is possible to provide a karaoke apparatus that accurately scores a dance.

  Moreover, the karaoke apparatus of this 5th invention is 1 or more for judging the movement of the singer for every predetermined timing in the movement determination information storage part with respect to 4th invention. Movement determination information is stored, and the skeleton information acquisition unit acquires one or more skeleton information indicating the movement of the singer at one or more predetermined timings using information included in the singer image, and scores The calculation unit determines the movement of the singer at a predetermined timing indicated by each of the one or more skeleton information acquired by the skeleton information acquisition unit using the movement determination information corresponding to the timing, and uses the determination result. This is a karaoke device for calculating a score.

  With this configuration, it is possible to provide a karaoke apparatus that performs dance scoring with higher accuracy.

  Further, in the karaoke apparatus according to the sixth aspect of the invention, in contrast to the fifth aspect of the invention, the score calculation unit scores the movement of the singer at one or more predetermined timings and scores at one or more timings. 1 is a karaoke device that calculates one or more scores, and calculates a score that is a weighted average of the one or more scores.

  With this configuration, it is possible to provide a karaoke apparatus that performs dance scoring with higher accuracy.

  Moreover, the karaoke apparatus of this 7th invention is 1 or more joint angles which are conditions regarding the angle of one or more each joints of a singer with respect to any one invention of 4th to 6th. The score calculation unit acquires the angles of one or more joints of the singer using the skeleton information acquired by the skeleton information acquisition unit, and uses joint angle conditions corresponding to the one or more joints. It is a karaoke apparatus which determines the angle of one or more each joint, and calculates a score using the determination result.

  With this configuration, it is possible to provide a karaoke apparatus that performs dance scoring with higher accuracy.

  Moreover, the karaoke apparatus of this 8th invention is 1 or more joints whose movement determination information is the conditions regarding the angle between 1 or more each joints of a singer with respect to any one invention of 4th to 7th. The score calculation unit uses the skeleton information acquired by the skeleton information acquisition unit to acquire an angle between one or more joints of the singer, and between the joints corresponding to the one or more joints. It is a karaoke apparatus that determines an angle between one or more joints using an angle condition, and calculates a score using the determination result.

  With this configuration, it is possible to provide a karaoke apparatus that performs dance scoring with higher accuracy.

  Further, the karaoke apparatus of the ninth aspect of the invention is directed to any one of the fourth to eighth aspects of the invention, wherein the motion determination information is one or more joint coordinates which are conditions relating to the coordinates of one or more joints of the singer. The score calculation unit acquires the coordinates of one or more joints of the singer using the skeleton information acquired by the skeleton information acquisition unit, and uses joint coordinate conditions corresponding to the one or more joints. This is a karaoke apparatus that determines the coordinates of one or more joints and calculates a score using the determination result.

  With this configuration, it is possible to provide a karaoke apparatus that performs dance scoring with higher accuracy.

  Further, in the karaoke apparatus of the tenth invention, with respect to any one of the fourth to ninth inventions, the movement determination information is one or more joints which is a condition relating to a moving amount of each of the one or more joints of the singer. It is a movement amount condition, and the score calculation unit acquires the movement amount of each of the one or more joints of the singer using the skeleton information acquired by the skeleton information acquisition unit, and the joint movement amount corresponding to each of the one or more joints It is a karaoke apparatus that determines a moving amount of each of one or more joints using a condition and calculates a score using the determination result.

  With this configuration, it is possible to provide a karaoke apparatus that performs dance scoring with higher accuracy.

  Further, in the eleventh invention, the karaoke apparatus according to any one of the first to tenth inventions, wherein the information included in the singer image deletes pixels satisfying a predetermined condition from the singer image It is a karaoke apparatus which is a singer image after having performed.

  With such a configuration, the real-time performance of the dance scoring can be improved.

  A karaoke apparatus according to the twelfth aspect of the present invention includes a model image storage unit that stores a model image that is an image indicating a dance model, according to any one of the first to eleventh aspects of the invention. The karaoke apparatus further includes an image output unit that outputs a model image.

  With this configuration, a dance example can be displayed.

  Further, the karaoke apparatus of the thirteenth aspect of the invention obtains one or more attribute values of the singer using the information included in the singer image, and at least a part of the singing singing is performed. A model image constructing unit that configures a model image that is an image indicating a model using one or more attribute values of the user, and the image output unit includes a model image and a model image configured by the model image configuring unit. It is a karaoke device that outputs.

  With such a configuration, it is possible to display a model that performs a movement corresponding to the movement of the singer in real time.

  According to the karaoke apparatus and the like according to the present invention, it is possible to provide a karaoke apparatus that accurately scores a dance.

Block diagram of karaoke apparatus 1 in the first embodiment The flowchart explaining the whole operation | movement of the karaoke apparatus 1 Flowchart explaining the acquisition process of the skeleton information Flow chart explaining calculation and output processing of the score A flowchart for explaining the structure and output processing of the model image The figure which shows the example of the same movement determination information The figure which shows the example of the same distance image The figure which shows the example of the same three-dimensional skeleton information The figure which shows the example of the image which imaged the same three-dimensional skeleton information The figure which shows the example of the joint angle of the singer Figure showing an example of the same parameter The figure which shows the example of the same movement determination information The figure which shows the example of the information which shows the result of the same judgment The figure which shows the example of the singer model image Figure showing an example of a copy model image The figure which shows the output example of the same score, model image, and model image Block diagram of the karaoke device 2 Overview of the computer system in the above embodiment Block diagram of a computer system in the above embodiment

  Hereinafter, embodiments of a karaoke apparatus and the like according to the present invention will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again. In addition, the format, content, and the like of each information described in this embodiment are merely examples, and the format, content, and the like are not limited as long as the meaning of each information can be indicated.

(Embodiment 1)
In the present embodiment, a karaoke apparatus 1 that photographs a singer, scores the movement of the singer, and outputs a score that is a result of the grading will be described.

  FIG. 1 is a block diagram of a karaoke apparatus 1 in the present embodiment. The karaoke apparatus 1 includes a movement determination information storage unit 101, a model image storage unit 102, a music data storage unit 103, a reception unit 104, a photographing unit 105, a skeleton information acquisition unit 106, a score calculation unit 107, a score output unit 108, a model. An image construction unit 109, an image output unit 110, and a music playback unit 111 are provided.

  The movement determination information storage unit 101 stores movement determination information that is information for determining the movement of the singer. The “movement determination information” is also information indicating the movement of the model. The “exemplary movement” is a movement that is a model of body movement in dance. The “movement” includes a motion, a posture, a posture, and the like, and is widely understood. The “motion determination information” is usually time-series information. That is, the “motion determination information” is information corresponding to one or more predetermined timings, for example. The “one or more predetermined timings” are, for example, a frame (still image) constituting a moving image, a section of a frame constituting the moving image, a time from the start of the moving image (time stamp), and the like.

Specifically, the “motion determination information” is, for example, one or more of the following.
(A) Joint angle condition (B) Joint coordinate condition (C) Joint movement amount condition (D) Inter-joint angle condition

  (A) Joint angle condition: The “joint angle condition” is a condition relating to an angle (hereinafter, appropriately referred to as a joint angle) of a joint (elbow, knee, wrist, ankle, neck, etc.). The “angle” is usually a so-called inner angle, but may be an outer angle. Further, when the “movement determination information” is “joint angle condition”, the “motion determination information” is usually information composed of one or more joint angle conditions regarding the angle of one or more joints of the singer. . Further, the “joint angle condition” usually includes information for identifying a joint (hereinafter, referred to as joint identification information as appropriate). The “joint identification information” is, for example, a joint name or ID. The “joint angle condition” is, for example, “right elbow = 90 °”, “120 ° ≦ left knee ≦ 130 °”, and the like.

  (B) Joint coordinate condition: “Joint coordinate condition” is a condition related to the coordinates of a joint (hereinafter, appropriately referred to as joint coordinates). A “joint” is a point corresponding to a joint and a part (hand, foot, head, etc.) that moves about the joint. This point is usually the end point of a joint or the above part. Further, this point may be, for example, a point included in the joint or the part such as a joint or the midpoint of the part, or a point adjacent to the joint or the part. That is, this point should just be a point corresponding to a joint or the said site | part. The “joint” may be a “node”, for example. The “coordinates” are usually three-dimensional coordinates. The “coordinates” may be, for example, two-dimensional coordinates. Further, the “coordinates” may be absolute coordinates or relative coordinates. “Absolute coordinates” are coordinates in the singer image. The “relative coordinates” are coordinates indicating positions with respect to other joints. In other words, the “relative coordinates” are coordinates when the coordinates of the reference joint are “(x, y) = (0, 0)”.

  Further, when the “motion determination information” is “joint coordinate condition”, the “motion determination information” is usually information composed of one or more joint coordinate conditions regarding the coordinates of one or more joints of the singer. . The “joint coordinate condition” usually includes information for identifying a joint (hereinafter, appropriately referred to as joint identification information). The “joint coordinate condition” is, for example, “right elbow = (x1, y1, z1)” or “(x2, y2, z2) ≦ left knee ≦ (x3, y3, z3)”.

  (C) Joint movement amount condition: The “joint movement amount condition” is a condition related to the movement amount of the joint (hereinafter referred to as “joint movement amount” as appropriate). The “joint movement amount” is usually the magnitude of movement of the joint and the direction of movement of the joint. Further, the “joint movement amount” may be, for example, only the magnitude of the joint movement. The “joint movement amount” may be considered as a so-called motion vector (vector amount).

  Further, the “joint movement amount” can be any content or format as long as it indicates the joint movement magnitude (length, number of pixels, etc.) and the joint movement direction (coordinates, angle, etc.). Further, the “size of movement” may be, for example, a relative size. The “relative size” is, for example, a relative size with respect to the size (physique) of the singer. Further, the “direction of movement” may be a relative direction, for example. The “relative direction” is, for example, a relative direction with respect to a coordinate axis, a singer's direction, and the like.

  The “movement amount” is, for example, “(x: 10 px, y: 20 px) / sec”, “(25 px, 30 °) / frame”, or the like. The former means that it has moved 10 px in the x-axis direction and 20 px in the y-axis direction per second. The latter means that the frame has moved 25 px in the direction of 30 ° from the horizontal direction per frame.

  When “movement determination information” is “joint movement amount condition”, “movement determination information” is usually composed of one or more joint movement amount conditions regarding the movement amount of one or more joints of the singer. Information. Further, the “joint movement amount condition” usually has joint identification information. The “joint movement amount condition” is, for example, “right elbow = ((x: 10 px, y: 20 px) / sec)” or “((25 px, 30 °) / frame) ≦ left knee ≦ ((30 px, 30 °) / frame) ".

  (D) Inter-joint angle condition: The “inter-joint angle condition” is a condition related to the angle between joints. “An angle between joints” (hereinafter, appropriately referred to as an inter-joint angle) is an angle formed by two line segments connecting two joints of three joints. The “inter-joint angle” is usually a so-called inner angle, but may be an outer angle. That is, “between joints” is usually a combination or permutation of three joints.

  In addition, the “inter-joint angle” may be, for example, an angle of a line segment with respect to a normalized coordinate system (hereinafter, appropriately referred to as a target line segment). The normalized coordinate system is, for example, a line segment (hereinafter appropriately referred to as a reference line segment) or a plane (hereinafter appropriately referred to as a reference plane). The reference line segment is, for example, a coordinate segment, a center line of the singer's body, a line segment passing through the centroid of the singer's body, or the like. The reference plane is, for example, a coordinate plane, a plane that is horizontal to the singer's body, a plane that is perpendicular to the singer's body, a plane that is horizontal to the singer's body, and a direction that is perpendicular to the singer's body. It is a plane. The target line segment is usually a line segment that connects two joints.

  Further, when the “motion determination information” is “inter-joint angle condition”, the “motion determination information” is usually composed of one or more inter-joint angle conditions regarding the angle between one or more joints of the singer. Information. The “joint angle condition” usually includes joint identification information. The “joint angle condition” is, for example, “left shoulder−right shoulder−right wrist = 120 °” or “45 ° ≦ right hand−head−waist ≦ 60 °”. The former joint angle condition means that an angle formed by a line segment connecting the left shoulder and the right shoulder and a line segment connecting the right shoulder and the right wrist is 120 °. The latter “inter-joint angle condition” means that the angle formed by the line segment connecting the right hand and the head and the line segment connecting the head and the waist is 40 ° or more and 60 ° or less.

  Note that the motion determination information may be information including one or more joint angles, for example. In this case, joint identification information is usually associated with the one or more joint angles. The joint angle associated with the joint identification information is, for example, “right elbow: 90 °” or “left knee: 130 °”. The motion determination information is usually included in the joint angle condition.

  In addition, the motion determination information may be information including one or more joint coordinates, for example. In this case, the one or more joint coordinates are usually associated with joint identification information. The joint coordinates with which the joint identification information is associated are, for example, “right elbow: (x1, y1, z1)”, “left knee: (x3, y3, z3)”, and the like. The motion determination information is usually included in the joint coordinate condition.

  Further, the motion determination information may be information composed of one or more joint movement amounts, for example. In this case, the one or more joint movement amounts are usually associated with joint identification information. The joint movement amount associated with the joint identification information is, for example, “right elbow: ((x: 10 px, y: 20 px) / sec)” or “left knee: ((30 px, 30 °) / frame) ". The motion determination information is usually included in the joint movement amount condition.

  Further, the motion determination information may be information including one or more joint angles, for example. In this case, joint identification information is usually associated with the one or more inter-joint angles. Further, the angle between joints corresponding to the joint identification information is, for example, “left shoulder−right shoulder−right wrist: 120 °”, “right hand−head−waist: 60 °”, and the like. The motion determination information is usually included in the joint angle condition.

  The model image storage unit 102 stores a model image which is an image showing a model of dance corresponding to the music. The “musical piece” is usually a musical piece that is output when the musical piece reproducing unit 111 described later reproduces musical piece data. The “example image” is usually a moving image. In addition, the “example image” is an image that shows a situation where, for example, a person or a character is dancing. In other words, the “example image” is an image obtained by photographing a person or character dancing. Further, the “example image” may be an image showing the body movement in the dance. For example, the “model image” may be a distance image described later.

  The “character” is usually a so-called avatar. The “character” may be, for example, a character appearing in a so-called animation, a so-called mascot character, or the like. The “character” may be a singer, for example.

  In the present embodiment, the “image” is a still image or a moving image. The moving image has two or more still images. In addition, two or more still images included in a moving image are usually frames constituting the moving image.

  The music data storage unit 103 stores one or more music data. “Music data” is information for electronically handling music. Also, the “music” is what is called “music”, what is called “music”, and the like. The “music data” may be in any format as long as it is information for electronically handling the music. The “music data” is in a format such as MIDI, WAV, MP3, or the like. The “music data” usually has music data identification information (hereinafter referred to as music data identification information as appropriate). “Music data identification information” is, for example, a singer name and a song name. The identification information may be a so-called file name, for example. The music data storage unit 103 normally stores music data received from the network. The music data is normally received by a receiving unit (not shown).

  The accepting unit 104 accepts an instruction. The instruction is, for example, a music selection instruction that is an instruction to select music data, a power ON instruction, a power OFF instruction, or the like. When the reception unit 104 receives a power-on instruction, a power-off instruction, or the like, the karaoke apparatus 1 normally performs a process according to the received instruction.

  Acceptance refers to acquisition of information input from input devices such as touch panels, remote controllers, and keyboards, acquisition of information stored in recording media such as optical disks, magnetic disks, and semiconductor memories, and wired or wireless communication lines. It is a concept that includes reception of information transmitted through the Internet.

  The input means such as information and instructions in the reception unit 104 may be anything such as a menu screen or a keyboard. The accepting unit 104 can be realized by control software for a menu screen, a device driver for input means such as a keyboard, and the like.

  The photographing unit 105 photographs a singer. Then, the photographing unit 105 acquires a singer image that is an image in which the singer is copied. The “singer” is usually one person. Further, the “singer” may be two or more people, for example. Further, it is sufficient that at least the singer is copied in the “singer image”, and it does not matter whether or not other objects are copied.

  Further, the “singer image” is an image including one or more pieces of information of distance information, luminance information, or RGB information. The “singer image” is, for example, a distance image. A “distance image” is an image having one or more pieces of distance information. The “distance information” is usually information indicating the distance from the karaoke apparatus 1 to the singer. In addition, the “distance to the singer” specifically refers to any part of the singer's head, eyes, nose, mouth, shoulders, chest, waist, knee, or any other surface on the singer's body. The distance to the point. In addition, “distance information” is associated with, for example, one or more pixels constituting an image. In addition, the “distance image” may be composed only of distance information, or may have RGB information, luminance information, and the like. The data structure of the singer image does not matter.

  Further, the “singer image” may be a captured image, for example. A “photographed image” is an image that does not have one or more distance information. The “photographed image” is usually a so-called color image, but may be a so-called grayscale image.

  As described above, the “singer image” may have, for example, one or both of RGB information and luminance information, or may not have both. “RGB information” is information indicating the color of one or more pixels constituting an image. “Luminance information” is information indicating the luminance of one or more pixels constituting an image. In addition, “brightness” is widely understood, including brightness and darkness and shading. The “pixel” is usually specified by coordinates (x, y) in the image.

  Moreover, the imaging unit 105 may acquire only one of the distance image and the captured image, or may acquire both, for example. When acquiring a distance image and a photographed image, the region where the photographer is photographed in the distance image and the photographed image is usually at the same position. That is, when acquiring a distance image and a photographed image, the photographing unit 105 usually obtains a distance image and a photographed image in which the singer is arranged at the same position. In addition, “acquiring a singer image” includes starting shooting of a singer and starting acquiring a singer image.

  Note that the photographing unit 105 usually acquires a distance image from a so-called distance image camera. Further, the photographing unit 105 may acquire a distance image from a so-called stereo camera, for example. The photographing unit 105 acquires a photographed image from an image sensor (solid-state imaging device) such as a CMOS or CCD, or a camera (digital still camera or digital video camera) using the image sensor, for example. In addition, the imaging unit 105 may include, for example, these devices or may be realized by these devices. The processing procedure of the photographing unit 105 is usually realized by software, and the software is recorded on a recording medium such as a ROM.

  The distance image camera includes, for example, a three-dimensional distance image camera ZC series (http://www.optex.co.jp/product/3d.html) and a TOF distance image camera DISSTANZA series “http: // www.brainvision.co.jp/xoops/modules/tinyd4/index.php?id=15 ", KINECT for Windows (http://www.microsoft.com/en-us/kinectforwindows/), etc.

  The skeleton information acquisition unit 106 acquires skeleton information using the singer image. The singer image is a singer image acquired by the photographing unit 105. The “skeleton information” is information indicating the movement of the singer. Further, “acquiring skeleton information” may be constituting skeleton information.

  Specifically, “skeleton information” is a set of one or more coordinates indicating the position of a so-called joint. In other words, “skeleton information” is information composed of one or more coordinates indicating the position of a so-called joint. The “coordinates” are usually three-dimensional coordinates. The “coordinates” may be, for example, two-dimensional coordinates. In addition, the three-dimensional coordinates may be, for example, three-dimensional coordinates acquired using a three-dimensional image, or may be three-dimensional coordinates acquired using a two-dimensional image. Further, the two-dimensional coordinates may be, for example, two-dimensional coordinates acquired using a three-dimensional image, or may be two-dimensional coordinates acquired using a two-dimensional image. Further, the three-dimensional image and the two-dimensional image are images acquired by the imaging unit 105. In addition, the “skeleton information” may be composed of, for example, one or more coordinates indicating the positions of all joints corresponding to the singer's body, or a part of the singer (hand, foot, head) Or the like) may be composed of one or more coordinates indicating the position of the joint corresponding to only.

  In addition, the coordinates of “skeleton information” are usually associated with each connected joint. That is, for example, the elbow and the wrist are connected by a bone. Therefore, in the “skeleton information”, the coordinates of the elbow and the coordinates of the wrist are associated with each other. Further, one or more coordinates included in the skeleton information are usually associated with joint identification information for identifying a joint corresponding to the coordinates. Of the one or more coordinates, a coordinate corresponding to a joint may be associated with an angle of the joint, for example.

Further, the skeleton information acquisition unit 106 acquires skeleton information, for example, in any of the following cases.
(A) When the singer image is a distance image (B) When the singer image is a captured image

Case (A): In this case, the photographing unit 105 acquires a distance image. In this case, the skeleton information acquisition unit 106 normally acquires skeleton information composed of three-dimensional coordinates using a distance image. The procedure for acquiring the skeleton information is, for example, as follows.
(1) Using the distance information that the distance image has, the coordinates (x, y) of the pixels that are so close that the distance indicated by the distance information satisfies a predetermined condition are acquired. Thereby, the area | region (henceforth a singer area | region suitably) of the singer copied in the distance image is detected.
(2) The outline of the singer area detected in (1) is detected.
(3) Pattern recognition is performed on the contour detected in (2), and a singer's joint in the distance image is detected. Thereby, the two-dimensional coordinate (x, y) corresponding to each joint of the singer photographed in the distance image is acquired.
(4) The two-dimensional coordinates (x, y) acquired in (3) are associated with each connected joint.
(5) Distance information corresponding to each pixel indicated by the two-dimensional coordinates (x, y) associated in (4) and a predetermined calculation formula (hereinafter, appropriately referred to as a joint coordinate value calculation formula) Are used to calculate the coordinate value (z) corresponding to the distance information.
(6) The two-dimensional coordinate (x, y) associated in (4) is associated with the coordinate value (z) calculated in (4) to obtain the three-dimensional coordinate (x, y, z). To do. Thereby, a set of three-dimensional coordinates (skeleton information) indicating the position of each joint is acquired.

In the case of (A), the procedure for acquiring skeleton information may be as follows, for example.
(1 ′) Using the distance information included in the distance image, the coordinates (x, y) of the pixels that are close enough that the distance indicated by the distance information satisfies a predetermined condition are acquired. Thereby, a singer area | region is detected.
(2 ′) Using distance information corresponding to each pixel in the singer area detected in (1 ′) and a predetermined calculation formula (hereinafter, referred to as a singer coordinate value calculation formula as appropriate), A coordinate value (z) corresponding to the distance information is calculated.
(3 ′) The two-dimensional coordinates (x, y) acquired in (1 ′) are associated with the coordinate values (z) calculated in (2 ′), and the three-dimensional coordinates (x, y, z) To get. Thereby, the shape of the singer copied in the distance image is detected.
(4 ′) The shape of the singer acquired in (3 ′) is applied to the reference skeleton information held in advance. The “reference skeleton information” is skeleton information indicating the movement of the reference. The “applying” means changing the three-dimensional coordinates of the skeleton information according to the shape of the singer. Also, inverse kinematics is usually used to apply the singer's shape. Thereby, a set of three-dimensional coordinates (skeleton information) indicating the position of each joint is acquired.

  In the above (1) and (1 ′), the skeleton information acquisition unit 106 sings, for example, an area in which one or more pixels that satisfy a predetermined condition for the distance indicated by the distance information are sung. It may be detected as a person area. In this case, the skeleton information acquisition unit 106 usually detects the region as a singer region when the area of the region where the one or more pixels are adjacent to each other is large enough to satisfy a predetermined condition. .

  Further, in the above (1) and (1 ′), the skeleton information acquisition unit 106 may detect the singer area by performing image processing such as binarization and labeling on the distance image, for example.

  Further, the joint coordinate value calculation formula in (5) and the singer coordinate value calculation formula in (2 ′) have, for example, variables for substituting distance information. The joint coordinate value calculation formula and the singer coordinate value calculation formula may be, for example, a so-called function (program). Also, the skeleton information acquisition unit 106 normally holds in advance a joint coordinate value calculation formula and a singer coordinate value calculation formula. Further, the joint coordinate value calculation formula and the singer coordinate value calculation formula are usually different.

In the case of (A) above, the skeleton information acquisition unit 106 may acquire skeleton information composed of two-dimensional coordinates, for example. The procedure for acquiring the skeleton information is, for example, the processes (1) to (4) described above. Also, the procedure for acquiring the skeleton information is, for example, the above-described processing from (1 ′) to (4 ′) and the following processing (5 ′).
(5 ′) The coordinate value (z) is deleted from the one or more three-dimensional coordinates acquired in (4 ′).

Case (B): In this case, the photographing unit 105 acquires a photographed image. In this case, the skeleton information acquisition unit 106 usually acquires skeleton information composed of two-dimensional coordinates using a captured image. The procedure for acquiring the skeleton information is, for example, as follows.
(1) Image processing such as binarization and labeling is performed on the photographed image to detect the contour of the singer photographed in the photographed image.
(2) Pattern recognition is performed on the contour detected in (1), and a singer's joint in the photographed image is detected. Thereby, the two-dimensional coordinates (x, y) corresponding to each joint of the singer photographed in the photographed image are acquired.
(3) The two-dimensional coordinates acquired in (2) are associated with each connected joint. Thereby, a set of two-dimensional coordinates (skeleton information) indicating the position of each joint is acquired.

In the case of (B) above, the procedure for acquiring the skeleton information may be as follows, for example.
(1 ′) Image processing such as binarization and labeling is performed on the photographed image to detect a singer area in the photographed image.
(2 ′) Thinning processing is performed on the singer area detected in (1 ′), and the center line of the singer photographed in the photographed image is acquired.
(3 ′) The two-dimensional coordinates (x, y) such as the intersection of the center lines acquired in (2 ′), the end points, and the midpoints of the intersection points and the end points (points on the center line) are acquired. Thereby, the two-dimensional coordinates (x, y) corresponding to each joint of the singer photographed in the photographed image are acquired.
(4 ′) The coordinates acquired in (3 ′) are associated with each coordinate connected by the center line. Thereby, a set of two-dimensional coordinates indicating the position of each joint is acquired.

  Also, the skeleton information acquisition unit 106 normally associates joint identification information for identifying each joint with the acquired coordinates of each joint. For example, when the coordinates of the right elbow are acquired, the skeleton information acquisition unit 106 associates, for example, joint identification information “right elbow” with the coordinates of the right elbow. The joint identification information is usually stored in a predetermined storage area.

  Hereinafter, the skeleton information composed of the three-dimensional coordinates is appropriately referred to as three-dimensional skeleton information. Hereinafter, the skeleton information composed of the two-dimensional coordinates is appropriately referred to as two-dimensional skeleton information.

  In addition, since the method, procedure, etc. which acquire skeleton information using the image (here singer image) on which the person was copied are well-known, detailed description is abbreviate | omitted. In addition, acquisition of skeleton information using a distance image can be performed by using, for example, commercially available software, SDK (Software Development Kit) attached to a commercially available distance image camera, or the like.

  Moreover, when acquiring skeleton information, the skeleton information acquisition part 106 may delete the pixel which satisfy | fills the conditions (henceforth pixel deletion conditions suitably) from a singer image, for example. In this case, the skeleton information acquisition unit 106 acquires skeleton information using the singer image after the deletion. Thereby, skeleton information can be acquired from a singer image at higher speed. The “deletion of pixels” is so-called thinning out of pixels. The “pixel deletion condition” includes, for example, distance information corresponding to a pixel for which it is determined whether or not to delete, and four (4 neighbors) or 8 (8 neighbors) around the pixel. ) That the difference from the distance information corresponding to each pixel is equal to or less than a predetermined threshold, the coordinate values of the two-dimensional coordinates (x, y) of the pixel are both even, The coordinate values of the two-dimensional coordinates (x, y) are both odd numbers.

  The score calculation unit 107 scores the movement of the singer and calculates a score that is a result of the scoring. In other words, the score calculation unit 107 calculates the dance score of the singer. In the following, “scoring” means “score calculation”.

  Further, the score calculation unit 107 usually scores the movement of the singer indicated by the skeleton information using the skeleton information, and calculates the score. The skeleton information is skeleton information acquired by the skeleton information acquisition unit 106. Specifically, the score calculation unit 107 determines, for example, the movement of the singer indicated by the skeleton information using the movement determination information, and calculates the score using the determination result. In addition, the score calculation unit 107 compares, for example, skeleton information and motion determination information, and calculates a score using the comparison result. The “compare” is to calculate a difference, a degree of coincidence, and the like. That is, the “comparison result” is a difference, a degree of coincidence, and the like.

In addition, the score calculation unit 107 calculates a score according to any of the following cases, for example.
(A) When the motion determination information is a joint angle condition (B) When the motion determination information is a joint coordinate condition (C) When the motion determination information is a joint movement amount condition (D) The motion determination information is an inter-joint angle When the condition is (E) When the motion determination information is a joint angle (F) When the motion determination information is joint coordinates (G) When the motion determination information is a joint movement amount (H) The motion determination information is between joints If it is an angle

In the case of (A): In this case, the score calculation unit 107 normally determines the angle of each of the one or more joints using the joint angle condition corresponding to the one or more of the joints, and uses the result of the determination. Calculate the score. A specific procedure for calculating the score in the case of (A) is, for example, as follows.
(1) One or more joint angles included in the skeleton information are acquired from the skeleton information acquired by the skeleton information acquisition unit 106. At this time, the joint angle is acquired together with the joint identification information associated with the joint angle.
(2) The joint angle condition having the same joint identification information as the joint identification information associated with one or more joint angles acquired in (1) is acquired from the motion determination information.
(3) It is determined whether one or more joint angles acquired in (1) satisfy a joint angle condition (joint angle condition acquired in (2)) corresponding to the joint angle. Then, a parameter indicating the result of the determination is calculated.
(4) A score is calculated using the parameters calculated in (3) and a predetermined calculation formula (hereinafter referred to as a score calculation formula as appropriate).

  In the above (1), when the skeleton information does not have one or more joint angles, the score calculation unit 107 normally calculates the joint angle using one or more coordinates included in the skeleton information. The method and procedure for calculating the angle of the angle formed by the three coordinates are well known and will not be described in detail. Further, the score calculation unit 107 associates joint identification information associated with the coordinates corresponding to the joint among the three coordinates used for calculating the joint angle with the joint angle calculated as the joint identification information.

  In (3) above, the “parameter” refers to, for example, the number of joint angles that satisfy the joint angle condition, the number of joint angles that do not satisfy the joint angle condition, and the joint that satisfies the joint angle condition with respect to the number of joint angles The ratio of the number of angles, the ratio of the number of joint angles that do not satisfy the joint angle condition with respect to the number of joint angles, and the like.

  In (4) above, the “score calculation formula” usually has variables for substituting the parameters. And the score calculation part 107 calculates a score by substituting the calculated parameter to the variable which the said score calculation formula has, and calculating the said calculation formula. The “score calculation formula” may be a so-called function (program). The score calculation unit 107 normally holds a score calculation formula in advance. Also, the “score calculation formula” has the same meaning as described above.

In the case of (B): In this case, the score calculation unit 107 normally determines the coordinates of one or more joints using joint coordinate conditions corresponding to the one or more joints, and uses the determination result. Calculate the score. A specific procedure for calculating the score in the case of (B) is, for example, as follows.
(1) One or more joint coordinates included in the skeleton information are acquired from the skeleton information acquired by the skeleton information acquisition unit 106. At this time, joint coordinates are acquired together with joint identification information associated with the joint coordinates.
(2) A joint coordinate condition having the same joint identification information as the joint identification information associated with one or more joint coordinates acquired in (1) is acquired from the motion determination information.
(3) It is determined whether one or more joint coordinates acquired in (1) satisfy a joint coordinate condition (joint coordinate condition acquired in (2)) corresponding to the joint coordinate. Then, a parameter indicating the result of the determination is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (3), the “parameter” means, for example, the number of joint coordinates that satisfy the joint coordinate condition, the number of joint coordinates that do not satisfy the joint coordinate condition, and the joint that satisfies the joint coordinate condition for the number of joint coordinates. For example, the ratio of the number of coordinates and the ratio of the number of joint coordinates that do not satisfy the joint coordinate condition with respect to the number of joint coordinates.

In the case of (C): In this case, the score calculation unit 107 normally determines the movement amount of each of the one or more joints using the joint movement amount condition corresponding to the one or more of the joints, and determines the result of the determination. To calculate the score. A specific procedure for calculating the score in the case (C) is, for example, as follows.
(1) One or more joint movement amounts are calculated using joint coordinates included in the skeleton information acquired by the skeleton information acquisition unit 106. At this time, the joint identification information associated with the joint coordinates used for the calculation is associated with the calculated joint movement amount.
(2) A joint movement amount condition having the same joint identification information as the joint identification information associated with each of the one or more joint movement amounts calculated in (1) is acquired from the motion determination information.
(3) It is determined whether one or more joint movement amounts acquired in (1) satisfy a joint movement amount condition (joint movement amount condition acquired in (2)) corresponding to the joint movement amount. Then, a parameter indicating the result of the determination is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (1), for example, a method and a procedure for calculating the joint movement amount using two or more still images constituting a moving image are known. Therefore, a detailed description of the method and procedure for calculating the joint movement amount is omitted.

  In the above (3), the “parameter” means, for example, the number of joint movement amounts that satisfy the joint movement amount condition, the number of joint movement amounts that do not satisfy the joint movement amount condition, and the joint movement amount relative to the number of joint movement amounts. The ratio of the number of joint movements satisfying the movement amount, the ratio of the number of joint movements not satisfying the joint movement amount condition to the number of joint movements, and the like.

In the case of (D): In this case, the score calculation unit 107 normally determines an angle between one or more joints using an inter-joint angle condition corresponding to the one or more joints, and the result of the determination The score is calculated using A specific procedure for calculating the score in the case of (D) is, for example, as follows.
(1) One or more inter-joint angles are calculated using joint coordinates included in the skeleton information acquired by the skeleton information acquisition unit 106. At this time, the joint identification information associated with the joint coordinates used for the calculation is associated with the calculated inter-joint angle.
(2) An inter-joint angle condition having inter-joint identification information identical to the inter-joint identification information associated with one or more inter-joint angles acquired in (1) is acquired from the motion determination information.
(3) It is determined whether one or more inter-joint angles acquired in (1) satisfy an inter-joint angle condition (inter-joint angle condition acquired in (2)) corresponding to the inter-joint angle. Then, a parameter indicating the result of the determination is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (1), the joint between which the angle is calculated may be determined in advance or may not be, for example. Otherwise, the score calculation unit 107 acquires, for example, all combinations or permutations of three joint coordinates among one or more joint coordinates included in the skeleton information, and obtains inter-joint angle information for each combination or permutation. It may be calculated.

  In (3) above, the “parameter” refers to, for example, the number of joint angles that satisfy the joint angle condition, the number of joint angles that do not satisfy the joint angle condition, and the joint to the number of joint angles. The ratio of the number of inter-joint angles satisfying the inter-angle angle, the ratio of the number of inter-joint angles not satisfying the inter-joint angle condition with respect to the number of inter-joint angles, and the like.

In the case of (E): In this case, the score calculation unit 107 normally compares the angle of each of the one or more joints of the singer with the joint angle corresponding to each joint and the motion determination information. Then, a score is calculated using the result of the comparison. A specific procedure for calculating the score in the case (E) is, for example, as follows.
(1) One or more joint angles included in the skeleton information are acquired from the skeleton information acquired by the skeleton information acquisition unit 106. At this time, the joint angle is acquired together with the joint identification information associated with the joint angle. Hereinafter, the acquired joint angle is appropriately referred to as a singer joint angle.
(2) The joint angle associated with the same joint identification information as the joint identification information associated with one or more joint angles obtained in (1) is obtained from the motion determination information. Hereinafter, the acquired joint angle is appropriately referred to as a model joint angle.
(3) One or more singer joint angles acquired in (1) are compared with the model joint angles (model joint angles acquired in (2)) corresponding to the singer joint angles. Then, a parameter indicating the result of the comparison is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (1), when the skeleton information does not have one or more joint angles, the score calculation unit 107 normally calculates the joint angle using one or more coordinates included in the skeleton information. The method and procedure for calculating the angle of the angle formed by the three coordinates are well known and will not be described in detail. Further, the score calculation unit 107 associates joint identification information associated with the coordinates corresponding to the joint among the three coordinates used for calculating the joint angle with the joint angle calculated as the joint identification information.

  In (3) above, “parameter” refers to, for example, the absolute value of the difference between the singer joint angle and the model joint angle, the ratio of the absolute value to the model joint angle, and the singing to the model joint angle. The ratio of the joint angle of the person.

In the case of (F): In this case, the score calculation unit 107 normally compares the coordinates of each of the one or more joints of the singer with the joint coordinates that are the joint coordinates corresponding to each of the joints and are motion determination information. Then, a score is calculated using the result of the comparison. A specific procedure for calculating the score in the case of (F) is, for example, as follows.
(1) One or more joint coordinates included in the skeleton information are acquired from the skeleton information acquired by the skeleton information acquisition unit 106. At this time, joint coordinates are acquired together with joint identification information associated with the joint coordinates. The acquired joint coordinates are hereinafter referred to as singer joint coordinates as appropriate.
(2) The joint coordinates associated with the same joint identification information as the joint identification information associated with one or more joint coordinates obtained in (1) are obtained from the motion determination information. Hereinafter, the acquired joint coordinates are appropriately referred to as model joint coordinates.
(3) The one or more singer joint coordinates acquired in (1) are compared with the model joint coordinates (example joint coordinates acquired in (2)) corresponding to the singer joint coordinates. Then, a parameter indicating the result of the comparison is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (3), the “parameter” means, for example, the absolute value of the difference between the singer joint coordinates and the model joint coordinates, the ratio of the absolute value to the model joint coordinates, and the singing with respect to the model joint coordinates. This is the ratio of the joint joint coordinates.

In the case of (G): In this case, the score calculation unit 107 is generally a movement amount of each of the one or more joints of the singer, a joint movement amount corresponding to each joint, and a joint movement amount that is motion determination information. And a score is calculated using the result of the comparison. A specific procedure for calculating the score in the case of (G) is, for example, as follows.
(1) One or more joint movement amounts are calculated using joint coordinates included in the skeleton information acquired by the skeleton information acquisition unit 106. At this time, the joint identification information associated with the joint coordinates used for the calculation is associated with the calculated joint movement amount. Moreover, let the calculated joint movement amount be a singer joint movement amount.
(2) A joint movement amount associated with the same joint identification information as the joint identification information associated with one or more joint movement amounts calculated in (1) is acquired from the motion determination information. Hereinafter, the acquired joint movement amount is appropriately referred to as a model joint movement amount.
(3) Compare one or more singer joint movement amounts acquired in (1) with the sample joint movement amount corresponding to the singers joint movement amount (example joint movement amount acquired in (2)). To do. Then, a parameter indicating the result of the comparison is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (3), the “parameter” means, for example, the absolute value of the difference between the singer joint movement amount and the sample joint movement amount, the ratio of the absolute value to the sample joint movement amount, the sample joint For example, the ratio of the singer joint movement amount to the movement amount.

In the case of (H): In this case, the score calculation unit 107 is usually an angle between one or more joints of the singer and an inter-joint angle corresponding to each joint, and is the motion determination information. The angle is compared, and a score is calculated using the result of the comparison. A specific procedure for calculating the score in the case of (H) is, for example, as follows.
(1) One or more inter-joint angles are calculated using joint coordinates included in the skeleton information acquired by the skeleton information acquisition unit 106. At this time, the joint identification information associated with the joint coordinates used for the calculation is associated with the calculated inter-joint angle. Moreover, let the calculated inter-joint angle be an angle between singer joints.
(2) An inter-joint angle corresponding to the same inter-joint identification information corresponding to the one or more inter-joint angles calculated in (1) is acquired from the motion determination information. Hereinafter, the acquired inter-joint angle is appropriately referred to as a sample inter-joint angle.
(3) Compare one or more inter-singer joint angles acquired in (1) with an inter-sample joint angle corresponding to the inter-singer joint angle (inter-joint joint angle acquired in (2)). To do. Then, a parameter indicating the result of the comparison is calculated.
(4) The score is calculated using the parameter calculated in (3) and the score calculation formula.

  In the above (3), “parameter” means, for example, the absolute value of the difference between the angle between the singer joints and the angle between the model joints, the ratio of the absolute value to the angle between the model joints, the model joints, etc. For example, the ratio of the angle between the singer joints with respect to the interval angle.

  Further, in the cases (A) to (H) above, the “score calculation formula” indicates, for example, that the better the dance (the closer the singer's movement matches the movement of the model), the higher the score. It is preferable that the calculation formula is such that the higher the value is, the lower the dance is (the more the singer's movement does not match the movement of the model), the lower the score.

  Further, for example, when scoring is performed by a so-called scoring method, the “score calculation formula” may be a calculation formula for calculating a score. In addition, for example, when scoring by a so-called deduction method, the “score calculation formula” may be a calculation formula for calculating a deduction amount.

  In the case of (B), the score calculation unit 107 may perform so-called normalization on, for example, the coordinates of the joint coordinate condition and the coordinates of the skeleton information. Further, the score calculation unit 107 may perform the normalization on, for example, one of the coordinates included in the joint coordinate condition or the coordinates included in the skeleton information. The normalization here is a process for absorbing the difference between the coordinates of the joint coordinate condition and the coordinates of the skeleton information. In other words, the normalization here is indicated by skeleton information, a process for absorbing the difference between the size of the singer indicated by the skeleton information and the size of the model indicated by the coordinates of the joint coordinate condition. Processing for absorbing the difference between the position of the singer and the position of the model indicated by the coordinates of the joint coordinate condition, the direction of the singer indicated by the skeleton information, and the hand indicated by the coordinates of the joint coordinate condition It is one or more of the three processes for absorbing the difference from the book orientation.

  In the case of (F), the score calculation unit 107 may perform so-called normalization on, for example, joint coordinates and coordinates included in the skeleton information. Further, the score calculation unit 107 may perform the normalization on, for example, one of joint coordinates or coordinates included in the skeleton information. The normalization here is a process for absorbing the difference between the joint coordinates and the coordinates of the skeleton information. In other words, normalization here means processing to absorb the difference between the size of the singer indicated by the skeleton information and the size of the model indicated by the joint coordinates, and the position of the singer indicated by the skeleton information. To absorb the difference between the position of the model indicated by the joint coordinates and the direction of the singer indicated by the skeleton information and the direction of the model indicated by the joint coordinates Are one or more of the three processes.

  Further, the score calculation unit 107 may calculate the score using the skeleton information without comparing the skeleton information and the motion determination information, for example. In this case, the score calculation unit 107 calculates a score corresponding to the movement indicated by the skeleton information using, for example, information included in the skeleton information or information calculated using the skeleton information and a score calculation formula. At this time, the score calculation unit 107 uses information included in the skeleton information as a parameter to be substituted into the score calculation formula. The “information calculated using the skeleton information” is, for example, a joint angle, a joint movement amount, an angle between joints, and the like.

  Further, the parameter used for calculating the score may be considered to be a coefficient (weight) for the score of each joint or each joint, for example.

  In addition, as described above, the score calculation unit 107 normally only needs to be able to calculate a score, which is a singer's movement scoring result, using at least the skeleton information, and the method and procedure are not limited.

  The score output unit 108 outputs the score calculated by the score calculation unit 107. Output refers to display on a display, projection using a projector, printing on a printer, sound output, transmission to an external device, storage on a recording medium, processing results to other processing devices or other programs, etc. It is a concept that includes delivery. In addition, regarding transmission, accumulation, and delivery of processing results, an output target is finally presented to the user.

  Moreover, the output mode of the score by the score output unit 108 does not matter. For example, the score output unit 108 may output the score every time the score calculation unit 107 calculates the score, for example. The score output unit 108 may statistically process the score calculated by the score calculation unit 107 and output the result of the statistical processing, for example. The statistical processing includes, for example, summing and calculating an average (simple average or weighted average).

  The score output unit 108 may or may not include an output device such as a display or a speaker. The score output unit 108 can be realized by driver software of an output device or driver software and an output device of an output device.

  The model image construction unit 109 constructs a model image. Specifically, the model image construction unit 109 acquires one or more attribute values of the singer using information included in the singer image, and at least a part thereof uses one or more attribute values of the singer. A model image that is an image showing a model is constructed. “Information included in the singer image” means, for example, distance information, luminance information, or RGB information corresponding to a singer's area among one or more pieces of information of distance information, luminance information, or RGB information. One or more pieces of information. The “part” is a part of time, space, or part of time and space. “Part of time” refers to a part of the time zone when the model image is a moving image. In addition, “spatial part” refers to a part of one or more parts of a singer. The “singer's attribute value” is information relating to the singer, and may be referred to as a singer's feature value. The “one or more attribute values” are, for example, one or more pieces of information of a singer's movement, a singer's shape, or a singer's color. Further, “one or more attribute values” are usually one or more types of attribute values.

  The “model image” is, for example, an image that at least partially moves according to the movement of the singer, or an image that reflects the shape or color of the singer. In addition, the “movement according to the movement of the singer” reflects, for example, the movement linked to the movement of the singer, the same movement (including almost the same) as the movement of the singer, and the movement of the singer. Movement, movement corresponding to the movement of the singer, etc., widely understood.

  The “model image” is, for example, an image imitating a singer, an image in which a singer's face and a character's body are combined, an image showing a character, or the like. The “configuration of a model image imitating a singer” means that a model image imitating the color of a singer, the shape of a singer, or the like is configured. More specifically, “the configuration of the model image imitating a singer” means, for example, obtaining the singer's attribute values such as the color of the singer and the shape of the singer, and using the attribute value, To construct a model image. The “singer color” includes, for example, the hair color of the singer, the skin color of the singer, the color of the singer's clothes, and the like. In addition, the “color” usually includes luminance. The “singer's shape” is, for example, the contour of the singer or the undulation of the surface of the singer's body.

  The “model image” is usually an image indicating a three-dimensional model (hereinafter, appropriately referred to as a three-dimensional model image). A “three-dimensional model image” is an image in which a model is expressed three-dimensionally (three-dimensionally). Further, the “model image” may be, for example, an image indicating a two-dimensional model (hereinafter, appropriately referred to as a two-dimensional model image). A “two-dimensional model image” is an image in which a model is expressed two-dimensionally (planar). Further, the “model image” is merely an image, not a hologram.

Further, the model image construction unit 109 constructs a model image by, for example, any one of the following methods.
(A) Method using distance image (part 1)
(B) Method using distance image (part 2)
(C) Method using distance image (part 3)
(D) Method using photographed image (E) Method using skeleton information

Method (A): This method is a method of constructing a three-dimensional model image showing a three-dimensional model imitating the color and shape of a singer using a distance image and a photographed image. That is, this method is a method of constructing the three-dimensional model image when the photographing unit 105 acquires the distance image and the photographed image. In addition, this method is a method for performing so-called surface modeling. The specific procedure of the method is, for example, as follows.
(1) Using the distance information that the distance image has, the coordinates (x, y) of the pixels that are so close that the distance indicated by the distance information satisfies a predetermined condition are acquired. Thereby, a singer area | region is detected.
(2) The coordinate value (z) corresponding to the distance information is calculated using the distance information associated with the pixel whose coordinates have been acquired in (1) and the singer coordinate value calculation formula.
(3) The coordinate value (z) calculated in (2) is associated with the coordinate (x, y) of the pixel used to calculate the coordinate value, and a three-dimensional coordinate (x, y, z) is acquired. To do. Thereby, one or more three-dimensional coordinates are acquired. In other words, the shape of the singer copied in the distance image is detected. Further, this process may be considered to be polygon modeling, which is a process for creating 3DCG, for example.
(4) An image corresponding to the singer area detected in (1) (hereinafter appropriately referred to as a singer area image) is cut out from the captured image.
(5) A three-dimensional model that imitates the color and shape of a singer by applying the singer area image cut out in (4) to one or more three-dimensional coordinates (x, y, z) acquired in (3). Construct a model image. “Applying an image to three-dimensional coordinates” means, for example, superimposing the image on the shape of a singer indicated by the one or more three-dimensional coordinates (x, y, z). Further, this process may be considered to be texture mapping, which is a process for creating 3DCG, for example.

  By the above processing, for example, when the photographed image is a color image, a three-dimensional model image imitating the color and shape of the singer is constructed. For example, when the captured image is a grayscale image, a three-dimensional model image imitating the brightness and shape of the singer is constructed.

Method (B): This method is a method of constructing a two-dimensional model image showing a two-dimensional model imitating the color and shape of a singer using a distance image and a photographed image. That is, this method is a method of constructing the two-dimensional model image when the photographing unit 105 acquires the distance image and the photographed image. The specific procedure of the method is, for example, as follows.
(1) Using the distance information that the distance image has, the coordinates (x, y) of the pixels that are so close that the distance indicated by the distance information satisfies a predetermined condition are acquired. Thereby, a singer area | region is detected.
(2) A singer area image corresponding to the singer area detected in (1) is cut out from the photographed image. The singer area image acquired by the cutout is a two-dimensional model image.

  By the above processing, for example, when the photographed image is a color image, a two-dimensional model image imitating the color and shape of the singer is constructed. For example, when the captured image is a grayscale image, a two-dimensional model image simulating the brightness and shape of the singer is constructed.

Method (C): This method is a method of constructing a three-dimensional model image or a two-dimensional model image showing a three-dimensional or two-dimensional model imitating the shape of a singer using a distance image. That is, this method is a method of configuring the three-dimensional model or the two-dimensional model when the photographing unit 105 acquires a distance image. A specific procedure for constructing the three-dimensional model image is, for example, as follows.
(1) Using the distance information that the distance image has, the coordinates (x, y) of the pixels that are so close that the distance indicated by the distance information satisfies a predetermined condition are acquired. Thereby, a singer area | region is detected.
(2) The coordinate value (z) corresponding to the distance information is calculated using the distance information associated with the pixel whose coordinates have been acquired in (1) and the singer coordinate value calculation formula.
(3) The coordinate value (z) calculated in (2) is associated with the coordinate (x, y) of the pixel used to calculate the coordinate value, and a three-dimensional coordinate (x, y, z) is acquired. To do. Thereby, one or more three-dimensional coordinates are acquired. In other words, the shape of the singer copied in the distance image is detected. Further, this processing may be considered to be polygon modeling, for example.
(4) A three-dimensional model image imitating the shape of a singer is applied to one or more three-dimensional coordinates (x, y, z) acquired in (3) by applying an image indicating a pattern held in advance. Configure. The “pattern” is, for example, clay, water, foam or the like. Further, the “image showing a pattern” may be considered as a texture used for texture mapping, for example. Further, this processing may be considered as texture mapping, for example.

In the method (C), the specific procedure for constructing the two-dimensional model image is, for example, as follows.
(1 ′) Using the distance information included in the distance image, the coordinates (x, y) of the pixels that are close enough that the distance indicated by the distance information satisfies a predetermined condition are acquired. Thereby, a singer area | region is detected.
(2 ′) An image indicating a pattern held in advance is applied to the one or more two-dimensional coordinates acquired in (1 ′) to form a two-dimensional model image imitating the shape of a singer. “Applying an image to two-dimensional coordinates” means, for example, superimposing the image on the shape of a singer indicated by the one or more two-dimensional coordinates (x, y). Further, this processing may be considered as texture mapping, for example.

  In the methods (A) to (C) above, the model image constructing unit 109 may detect the singer area by performing image processing such as binarization and labeling on the distance image, for example. .

Method (D): This method is a method of constructing a two-dimensional model image showing a two-dimensional model imitating the color and shape of a singer using a photographed image. That is, this method is a method of constructing the two-dimensional model when the photographing unit 105 acquires a photographed image. The specific procedure of the method is, for example, as follows.
(1) Image processing such as binarization and labeling is performed on the photographed image, and a singer area is detected.
(2) A singer area image corresponding to the singer area detected in (1) is cut out from the photographed image. The singer area image acquired by the cutout is a two-dimensional model image.

  Method (E): This method is a method of constructing a model image showing a character using character information and skeleton information. In other words, the method is a method of constructing a model image showing a character by so-called skeletal animation.

  “Character information” is information indicating a character. “Character information” is usually a set of information indicating one or more coordinates indicating the shape of the character and the color of each part of the character. The coordinates are usually three-dimensional (x, y, z). Further, the coordinates may be two-dimensional (x, y), for example. Further, the “character information” may be an image showing a character, for example. The “image showing a character” is usually an image in which a character is expressed three-dimensionally. Further, the “image showing a character” may be, for example, an image in which a character is expressed two-dimensionally. The “image showing a character” is usually a still image. Character information is usually stored in a predetermined storage area.

  As described above, the character information only needs to be information that can indicate the shape, color, etc. of the character, and the data structure thereof does not matter. Hereinafter, the character information indicating the three-dimensional character is appropriately referred to as three-dimensional character information. Further, the character information indicating the two-dimensional character is hereinafter appropriately referred to as two-dimensional character information.

The method (E) can be further classified into the following two methods, for example.
(E1) Method using three-dimensional skeleton information (E2) Method using two-dimensional skeleton information

Method (E1): This method is a method of constructing a three-dimensional model image of a character indicated by character information using three-dimensional skeleton information. The specific procedure of the method is, for example, as follows.
(1) Obtain three-dimensional character information from a predetermined storage area.
(2) The three-dimensional skeleton information is applied to the three-dimensional character information acquired in (1). “Applying the 3D skeleton information” is to associate the joint of the character indicated by the 3D character information with the coordinates of the 3D skeleton information and move the character to the movement indicated by the 3D skeleton information. .
(3) Using the 3D character information to which the 3D skeleton information is applied in (2), processing such as so-called rendering and texture mapping is performed to form a 3D model image.

In the case of (E2): This method is a method of constructing a two-dimensional model image of the character indicated by the character information using the two-dimensional skeleton information. The specific procedure of the method is, for example, as follows.
(1) Two-dimensional character information is acquired from a predetermined storage area.
(2) The two-dimensional skeleton information is applied to the two-dimensional character information acquired in (1). “Applying the two-dimensional skeleton information” is to associate the joint of the character indicated by the two-dimensional character information with the coordinates of the two-dimensional skeleton information, and move the character to the movement indicated by the two-dimensional skeleton information. .
(3) In step (2), two-dimensional character information to which the two-dimensional skeleton information is applied is imaged to form a two-dimensional model image.

  The model image construction unit 109 also has, for example, one or more two-dimensional coordinates (x, y) indicating a singer area, or one or more three-dimensional information indicating the shape of a singer photographed in a distance image. A character image may be applied to the coordinates (x, y, z) to form a two-dimensional model image or a three-dimensional model image showing a character imitating the shape of a singer.

It should be noted that a method and procedure for constructing a three-dimensional model image using one or more three-dimensional coordinates, a method and procedure for constructing a two-dimensional model image using one or more two-dimensional coordinates, skeleton information, and character information Since a method and procedure for constructing a three-dimensional model image or a two-dimensional model image using the method are well-known, detailed description thereof will be omitted. For example, the following software can be used as software that configures an image (here, a model image) showing a three-dimensional model using one or more three-dimensional coordinates.
・ Maya (http://www.autodesk.co.jp/products/autodesk-maya/overview)
・ 3dsMAX (http://www.autodesk.co.jp/products/autodesk-3ds-max/overview)
・ Shade (http://shade.e-frontier.co.jp/)
・ OpenGL (http://www.opengl.org/)
・ DirectX (http://www.microsoft.com/en-us/directx/default.aspx)

  Moreover, the model image which shows what imitates a singer among the model images created by said method etc. is hereafter called a singer model image suitably. Moreover, the model image which shows a character among the model images created by said method etc. is hereafter called a character model image suitably.

  In creating a three-dimensional model image using a distance image, the model image construction unit 109 may delete, for example, pixels that satisfy the pixel deletion condition from the distance image. In this case, the model image construction unit 109 constructs a three-dimensional model image using the deleted distance image. Further, when constructing a three-dimensional model image using a distance image, the model image construction unit 109 may perform so-called culling, for example. By these processes, a three-dimensional model image can be constructed using the distance image at a higher speed.

  The image output unit 110 outputs a model image. For example, when the model image construction unit 109 constructs a model image, the image output unit 110 normally outputs the model image. In this case, the image output unit 110 may output a model image and a model image, for example. Further, the image output unit 110 may output a composite image that is an image obtained by combining two or more images, for example. The “composite image” is, for example, an image in which a model image is arranged on a background image. The “background image” is a background image. The “background image” is, for example, an image showing a so-called stage, landscape, geometric pattern, or the like. That is, the “background image” may be an image that can be a so-called background, and the content thereof is not limited. The “composite image” may be an image in which a model image and a model image are arranged on a background image, for example. In this case, the model image and the model image are usually arranged at positions that do not overlap on the background image. Further, the composition of the composite image by combining the two or more images is normally performed by a composite image construction unit (not shown).

  Also, for example, when outputting a model image, the image output unit 110 normally indicates the tempo (rhythm) of the music output when the music playback unit 111 described later plays back the music data, and the model image indicates A model image is output so that the tempo (rhythm) of the model dance is synchronized (matched). In addition, “output of a model image” includes starting output of a composite image.

  Further, the image output unit 110 may or may not include an output device such as a display. The score output unit 108 can be realized by driver software of an output device or driver software and an output device of an output device.

  The music playback unit 111 plays back the music data stored in the music data storage unit 103. “Play music data” means, for example, an apparatus capable of transmitting music data to a device capable of playing music data, outputting music data as a sound, and outputting music data as a sound. It is widely understood, including transmitting music data. Further, “reproduction of music data” is hereinafter referred to as “reproduction of music” as appropriate. In addition, “reproduction of music data” includes starting reproduction of music data.

  The music data reproduced by the music reproducing unit 111 is normally music data selected by a music selection instruction received by the receiving unit 104. In this case, for example, when the identification information of the music data is presented to the user, the user selects music data that the user likes from one or more music data.

  Further, the music reproducing unit 111 may be considered to include an output device such as a speaker, or may not be included. The music reproducing unit 111 can be realized by output device driver software, or output device driver software and an output device.

  The motion determination information storage unit 101, the model image storage unit 102, and the music data storage unit 103 are preferably non-volatile recording media, but can also be realized by volatile recording media. Further, the process in which the predetermined information is stored in the motion determination information storage unit 101 or the like does not matter. For example, the predetermined information may be stored in the motion determination information storage unit 101 or the like via a recording medium, a communication line, an input device, or the like.

  In addition, the skeleton information acquisition unit 106, the score calculation unit 107, and the model image configuration unit 109 can be usually realized by an MPU, a memory, or the like. The processing procedure of the skeleton information acquisition unit 106 and the like is usually realized by software, and the software is recorded on a recording medium such as a ROM. Note that the skeleton information acquisition unit 106 and the like may be realized by hardware (a dedicated circuit).

  Next, the overall operation of the karaoke apparatus 1 will be described using a flowchart. Note that the i-th information in the predetermined information is described as “information [i]”. FIG. 2 is a flowchart showing the overall operation of the karaoke apparatus 1.

  (Step S201) The karaoke apparatus 1 determines whether or not the receiving unit 104 has received a power-on instruction. If accepted, the process proceeds to step S202; otherwise, the process returns to step S201.

  (Step S202) The karaoke apparatus 1 performs a power ON process.

  (Step S203) A receiving unit (not shown) determines whether or not music data has been received. If received, the process proceeds to step S204, and if not, the process proceeds to step S205.

  (Step S204) The storage unit (not shown) stores the music data received in step S203 in the music data storage unit 103.

  (Step S205) The music reproducing unit 111 determines whether the receiving unit 104 has received a music selection instruction. If accepted, the process proceeds to step S206; otherwise, the process proceeds to step S207.

  (Step S206) The music reproducing unit 111 registers the music data identification information included in the music data selected by the music selection instruction received in step S205 in a reproduction list that is a list storing the music data identification information.

  (Step S207) The music playback unit 111 determines whether it is the playback timing of the music data. The timing is, for example, after the reproduction of music data is finished or after the reception unit 104 receives a music selection instruction. If it is time to reproduce the music data, the process proceeds to step S208; otherwise, the process proceeds to step S219.

  (Step S208) The music reproducing unit 111 determines whether or not music data identification information is registered in the reproduction list. If registered, the process proceeds to step S209, and if not, the process proceeds to step S219.

  (Step S209) The music reproducing unit 111 acquires music data identification information of music data to be reproduced next from the reproduction list.

  (Step S210) The music reproducing unit 111 acquires music data identified by the music data identification information acquired in step S209 from the music data storage unit 103.

  (Step S211) The music playback unit 111 starts playback of the music data acquired in step S210.

  (Step S212) The image output unit 110 determines whether to output a model image. The determination is made using, for example, information indicating whether or not to output a model image is stored in a predetermined storage area. If a model image is to be output, the process proceeds to step S213, and if not, the process proceeds to step S214.

  (Step S213) The image output unit 110 acquires a model image from the model image storage unit 102. Then, the image output unit 110 starts outputting the acquired model image.

  (Step S214) The photographing unit 105 starts photographing the singer. In other words, the imaging unit 105 starts acquiring the distance image.

  (Step S215) The skeleton information acquisition unit 106 determines whether the imaging unit 105 has acquired a distance image. If acquired, the process proceeds to step S216, and if not, the process proceeds to step S219.

  (Step S216) The skeleton information acquisition unit 106 acquires skeleton information using the distance image acquired in step S215. Details of this processing will be described with reference to the flowchart of FIG.

  (Step S217) The score calculation unit 107 calculates a score using the skeleton information acquired in step S216, and the score output unit 108 outputs the calculated score. Details of this processing will be described with reference to the flowchart of FIG.

  (Step S218) The model image construction unit 109 constructs a model image, and the image output unit 110 outputs the constructed model image. Details of this processing will be described with reference to the flowchart of FIG.

  (Step S219) The karaoke apparatus 1 determines whether or not the reception unit 104 has received a power-off instruction. If accepted, the process proceeds to step S220, and if not, the process returns to step S203.

  (Step S220) The karaoke apparatus 1 performs a power-off process. Then, the process returns to step S201.

  In the flowchart of FIG. 2, the process may be terminated by powering off or a process termination interrupt.

  In the flowchart of FIG. 2, the photographing by the photographing unit 105 ends when the reproduction of the music data by the music reproducing unit 111 is completed, for example. In addition, the reproduction of the music data ends, for example, when the reception unit 104 receives an instruction to end the reproduction of the music data or when the reproduction of the music data is completed.

  FIG. 3 is a flowchart showing the skeleton information acquisition process in step S216 of the flowchart of FIG.

  (Step S301) The skeleton information acquisition unit 106 detects a singer area using the distance information included in the distance image.

  (Step S302) The skeleton information acquisition unit 106 performs pattern recognition on the outline of the singer area detected in step S301, and detects a singer's joint in the distance image.

  (Step S303) The skeleton information acquisition unit 106 calculates the coordinate value corresponding to the distance information using the distance information associated with the pixel indicating the joint detected in step S302 and the joint coordinate value calculation formula. .

  (Step S304) The skeleton information acquisition unit 106 configures a three-dimensional coordinate by associating the two-dimensional coordinates of the pixel indicating the joint detected in Step S302 with the coordinate value calculated in Step S303. And the skeleton information acquisition part 106 matches the said three-dimensional coordinate for every connected joint, and comprises skeleton information. Then, the process returns to the upper process.

  FIG. 4 is a flowchart showing score calculation and output processing in step S217 of the flowchart of FIG.

  (Step S401) The score calculation unit 107 acquires determination target information using the skeleton information acquired by the skeleton information acquisition unit 106. The determination target information includes, for example, a singer joint angle, singer joint coordinates, singer joint movement amount, and the like. Here, it is assumed that the score calculation unit 107 has acquired m pieces of determination target information.

  (Step S402) The score calculation unit 107 determines whether the type of motion determination information stored in the motion determination information storage unit 101 is a motion determination condition or a motion determination value. The motion determination condition is, for example, a joint angle condition, a joint coordinate condition, a joint movement amount condition, or the like. The motion determination value is, for example, a model joint angle, a model joint coordinate, a model joint movement amount, or the like. If it is a motion determination condition, the process proceeds to step S403; otherwise, the process proceeds to step S414.

  (Step S403) The score calculation unit 107 sets 0 to the variable true.

  (Step S404) The score calculation unit 107 sets 0 to the variable false.

  (Step S405) The score calculation unit 107 sets 1 to the counter i.

  (Step S406) The score calculation unit 107 acquires a motion determination condition associated with the same identification information as the identification information associated with the determination target information [i] from the motion determination information storage unit 101.

  (Step S407) The score calculation unit 107 determines whether or not the determination target information [i] satisfies the motion determination condition acquired in step S406. If satisfied, the process proceeds to step S408, and if not, the process proceeds to step S409.

  (Step S408) The score calculation unit 107 increments true by 1.

  (Step S409) The score calculation unit 107 increments false by 1.

  (Step S410) The score calculation unit 107 determines whether i is m. When it is m, it progresses to step S412, and when that is not right, it progresses to step S411.

  (Step S411) The score calculation unit 107 increments i by 1. Then, the process returns to step S406.

  (Step S412) The score calculation unit 107 calculates a predetermined type of parameter using true and false.

  (Step S413) The score calculation unit 107 calculates a score using the parameter calculated in step S412 and a score calculation formula stored in advance.

  (Step S414) The score calculation unit 107 sets 1 to the counter i.

  (Step S415) The score calculation unit 107 acquires, from the motion determination information storage unit 101, a motion determination value associated with the same identification information as the identification information associated with the determination target information [i].

  (Step S416) The score calculation unit 107 compares the determination target information [i] with the motion determination value acquired in Step S415, and calculates a parameter [i] that is a parameter indicating the result of the comparison.

  (Step S417) The score calculation unit 107 determines whether i is m. When it is m, it progresses to step S419, and when that is not right, it progresses to step S418.

  (Step S418) The score calculation unit 107 increments i by 1. Then, the process returns to step S415.

  (Step S419) The score calculation unit 107 calculates a score using the m parameters calculated in step S416 and a score calculation formula stored in advance.

  (Step S420) The score output unit 108 determines whether it is time to output a score. The timing is, for example, periodically while the music reproducing unit 111 is reproducing the music data, or after the music reproducing unit 111 finishes reproducing the music data. If it is time to output the score, the process proceeds to step S421, and if not, the process returns to the upper process.

  (Step S421) The score output unit 108 outputs the score calculated in step S413 or step S419. Then, the process returns to the upper process.

  In the flowchart of FIG. 4, the type of determination target information and the type of motion determination information are the same. That is, when the determination target information is a singer joint angle, the motion determination information is a joint angle condition or a model joint angle. Moreover, when determination object information is a singer joint coordinate, movement determination information is a joint coordinate condition or a model joint coordinate. Further, when the determination target information is a singer joint movement amount, the movement determination information is a joint movement amount condition or a model joint movement amount.

  FIG. 5 is a flowchart showing the configuration and output processing of the model image in step S218 of the flowchart of FIG.

  (Step S501) The image output unit 110 determines whether to output a model image. The determination is made using, for example, information indicating whether or not to output a model image is stored in a predetermined storage area. If the model image is to be output, the process proceeds to step S502. If not, the process returns to the upper process.

  (Step S502) The model image construction unit 109 determines whether the type of the model image to be constructed is a singer model image or a character model image. The determination is performed using, for example, information indicating the type of model image to be configured is stored in a predetermined storage area. Then, when configuring the singer model image, the process proceeds to step S503, and when configuring the character model image, the process proceeds to step S506.

  (Step S503) The model image construction unit 109 detects a singer area that is a singer area copied in the distance image, using the distance information included in the distance image.

  (Step S504) The model image construction unit 109 uses the distance information associated with the pixel in the singer area detected in Step S503 and the singer coordinate value calculation formula, and the coordinate value corresponding to the distance information. Is calculated.

  (Step S505) The model image construction unit 109 associates the two-dimensional coordinates of the pixels in the singer area with the coordinate values calculated in Step S504, and constructs a three-dimensional coordinate. And the model image structure part 109 comprises a singer model image using the said three-dimensional coordinate.

  (Step S506) The model image construction unit 109 acquires the skeleton information acquired by the skeleton information acquisition unit 106.

  (Step S507) The model image construction unit 109 acquires character information stored in a predetermined storage area.

  (Step S508) The model image construction unit 109 constructs a character model image using the skeleton information acquired in step S506 and the character information acquired in step S507.

  (Step S509) The image output unit 110 outputs the singer model image acquired in step S505 or the character model image acquired in step S508. Then, the process returns to the upper process.

  In addition, the whole operation | movement of the karaoke apparatus 1 demonstrated above is an example to the last. That is, the overall operation of the karaoke apparatus 1 is not limited to the above description.

(Concrete example)
Next, a specific example of the operation of the karaoke apparatus 1 will be described. In this specific example, it is assumed that the music data storage unit 103 stores one or more music data in advance. The photographing unit 105 acquires a distance image.

(Example 1)
In this example, an example of scoring a singer's movement and calculating a score by comparing the skeleton information and the movement determination information will be described. In this example, it is assumed that the motion determination information storage unit 101 stores the motion determination information shown in FIG. The movement determination information (item name: model joint angle) is a model joint angle. Further, the motion determination information (example joint angle) is associated with an ID for uniquely identifying a record and joint identification information (item name: joint name).

  First, it is assumed that the user operates the remote controller of the karaoke apparatus 1 and performs an operation for selecting a song that the user wants to sing. Then, the reception unit 104 receives a music selection instruction.

  Next, the music reproducing unit 111 registers the music data identification information of the music data selected by the music selection instruction in the reproduction list. Here, it is assumed that only the registered music data identification information is registered in the reproduction list. Then, the music reproducing unit 111 acquires the music data identified by the music data identification information from the music data storage unit 103. Then, the music playback unit 111 starts playback of the acquired music data.

  Next, the photographing unit 105 starts photographing the singer. That is, the imaging unit 105 starts acquiring a distance image. Here, it is assumed that the distance image acquired by the photographing unit 105 at a certain timing is the still image shown in FIG. The distance image is an image expressing the distance from the karaoke apparatus 1 (strictly, a distance image camera constituting the photographing unit 105) to the singer by shading.

  Next, the skeleton information acquisition unit 106 detects a singer area from the distance image acquired by the imaging unit 105. And the 1 or more joint of the singer copied in the distance image is detected using the outline of the said singer area | region. Thereby, the skeleton information acquisition unit 106 acquires the coordinates (x, y) of the pixel indicating the joint. The skeleton information acquisition unit 106 associates the acquired coordinates (x, y) with each connected joint.

  Next, the skeleton information acquisition unit 106 uses the distance information associated with the acquired pixel indicated by one or more coordinates (x, y) and the joint coordinate value calculation formula to correspond to each distance information. The coordinate value (z) to be calculated is calculated. The skeleton information acquisition unit 106 associates the coordinates (x, y) indicating each pixel of the joint with the calculated coordinate value (z) for each corresponding pixel, and one or more three-dimensional coordinates. (X, y, z) is constructed. Thereby, the skeleton information acquisition unit 106 acquires three-dimensional skeleton information composed of one or more three-dimensional coordinates (x, y, z). The three-dimensional skeleton information is, for example, FIG. In FIG. 8, three-dimensional skeleton information includes an ID for uniquely identifying a record, joint identification information (item name: joint name 1, joint name 2), and three-dimensional coordinates (item name: coordinate 1, coordinate). 2). Moreover, the image which imaged the said skeleton information is FIG. 9, for example.

  Next, the score calculation part 107 calculates a singer joint angle using the skeleton information of FIG. At this time, the score calculation unit 107 acquires joint identification information, which is joint identification information for identifying a joint, from the skeleton information of FIG. And the score calculation part 107 acquires two records which have the said joint identification information about the acquired one joint identification information. Then, the angle of the joint identified by the one joint identification information is calculated using three coordinates that are unique in the two acquired records.

  For example, when calculating the joint angle corresponding to the joint identification information “right elbow”, the score calculation unit 107 first records “ID = 011” and “ID = 012” having the joint identification information “right elbow”. To get. Then, the score calculation unit 107 calculates three coordinates “(30, 30, 30)”, “(25, 40, 30)”, “(20, 50, 35)” that are unique in the two acquired records. Use to calculate the angle of the “right elbow”.

  The singer joint angle calculated as described above is, for example, FIG. In FIG. 10, an ID for uniquely identifying a record and joint identification information (item name: joint name) are associated with the singer joint angle.

  Next, the score calculation unit 107 compares the singer joint angle corresponding to the same joint identification information and the sample joint angle with respect to the singer joint angle in FIG. 10 and the sample joint angle in FIG. 6. Then, a parameter indicating the result of the comparison is calculated. The model joint angle in FIG. 6 is a model joint angle corresponding to the timing at which the distance image in FIG. 7 is acquired.

  For example, the score calculation unit 107 corresponds to the singer joint angle “25 °” of “ID = 012” in FIG. 10 associated with the same joint identification information “left shoulder” and the hand of “ID = 012” in FIG. The main joint angle “30 °” is compared, and the parameters are calculated. The calculation formula for calculating the parameter is “abs (singer joint angle−example joint angle) / example joint angle”. The calculation formula means that the absolute value of the difference between the singer joint angle and the model joint angle is divided by the model joint angle.

  The parameters calculated as described above are, for example, FIG. In FIG. 11, parameters are associated with IDs for uniquely identifying records and joint identification information (item name: joint name).

  Next, the score calculation unit 107 calculates a score using the parameters and the score calculation formula of FIG. The score calculation formula is “100− (average parameter × 100)”. The “average parameter” in the score calculation formula is the average parameter in FIG. Then, the score calculation unit 107 calculates a score “85”, for example.

  Next, the score output unit 108 outputs the score “85”.

(Example 2)
In this example, an example will be described in which a singer's movement is scored and a score is calculated based on determination of skeleton information based on movement determination information. In this example, it is assumed that the motion determination information storage unit 101 stores the motion determination information shown in FIG. The motion determination information (item name: joint angle condition) is a joint angle condition. Further, the motion determination information (joint angle condition) is associated with an ID for uniquely identifying a record.

  First, as in Example 1, it is assumed that the skeleton information acquisition unit 106 has acquired the skeleton information of FIG. 8 as a result of the music playback unit 111 starting playback of music data in accordance with a user instruction.

  Next, the score calculation unit 107 calculates the singer joint angle of FIG.

  Next, the score calculation unit 107 performs the singing for the singer joint angle and the joint angle condition associated with the same joint identification information in the singer joint angle of FIG. 10 and the joint angle condition of FIG. It is determined whether the joint angle satisfies the joint angle condition. Then, the score calculation unit 107 acquires information indicating the result of the determination.

  For example, the score calculation unit 107 corresponds to the singer joint angle “25 °” of “ID = 012” in FIG. 10 associated with the same joint identification information “left shoulder” and the joint of “ID = 012” in FIG. For the angle condition “25 ° ≦ left shoulder ≦ 35 °”, it is determined that the singer joint angle “25 °” satisfies the joint angle condition “25 ° ≦ left shoulder ≦ 35 °”. Then, the score calculation unit 107 acquires information “1” indicating the result of the determination.

  In addition, for example, the score calculation unit 107 matches the singer joint angle “100 °” of “ID = 013” in FIG. 10 associated with the same joint identification information “right elbow” and “ID = 013” in FIG. The joint angle condition “115 ° ≦ right elbow ≦ 125 °” is determined to be that the singer joint angle “100 °” does not satisfy the joint angle condition “115 ° ≦ right elbow ≦ 125 °”. Then, the score calculation unit 107 acquires information “0” indicating the result of the determination.

  Information indicating the result of the determination acquired as described above is, for example, FIG. In FIG. 13, information (item name: determination result) indicating a determination result is associated with an ID for uniquely identifying a record and joint identification information (item name: joint name).

  Next, the score calculation unit 107 calculates a parameter using information indicating the determination result of FIG. Here, it is assumed that the parameter is the number of “1” in the information indicating the determination result. Then, the score calculation unit 107 counts the number of “1” in the information indicating the determination result in FIG. 13 and calculates a parameter. As a result, it is assumed that the score calculation unit 107 calculates the parameter “10”.

  Next, the score calculation unit 107 calculates a score using the calculated parameter “8” and the score calculation formula. The score calculation formula is “100 × (parameter / 14)”. Then, the score calculation unit 107 calculates a score “71”, for example.

  Next, the score output unit 108 outputs the score “71”.

(Example 3)
Next, an example of outputting a model image and a model image together with the score will be described.

  First, similarly to Example 1, it is assumed that the score calculation unit 107 calculates the score “85”.

  Next, the model image construction unit 109 detects a singer area from the distance image of FIG. And the coordinate value (z) corresponding to each said distance information is calculated using the distance information corresponding to each pixel in the said singer area | region, and a singer coordinate value calculation formula. Then, the model image construction unit 109 associates the coordinates (x, y) indicating each pixel in the singer area with the calculated coordinate value (z) for each corresponding pixel, and 1 or more 3 Construct dimension coordinates (x, y, z).

  Next, the model image construction unit 109 performs processing for creating 3DCG, such as modeling, rendering, and texture mapping, using the constructed one or more three-dimensional coordinates (x, y, z). And the model image structure part 109 comprises a singer model image. The singer model image is, for example, an image shown in FIG.

  Next, the image output unit 110 acquires a model image from the model image storage unit 102. The model image is assumed to be an image shown in FIG.

  Next, the score output unit 108 outputs the score “85”, and the image output unit 110 outputs the model image of FIG. 14 and the model image of FIG. At this time, the score output unit 108 outputs the score so as to be arranged at a position that does not overlap with the model image. In addition, the image output unit 110 outputs the model image and the model image on different screens. The state at this time is, for example, FIG. FIG. 16 is a diagram illustrating an example in which a model image is output on the left side of the screen, a model image is output on the right side of the screen, and a score is output on the upper part of the model image.

  As described above, according to the karaoke apparatus 1 according to the present embodiment, it is possible to provide a karaoke apparatus that accurately scores a dance. Thereby, the user (singer) can enjoy not only singing a song but also dancing, and not only singing but also enjoying karaoke while dancing. Further, by outputting a model image, even a user who does not know the dance corresponding to the song to be sung can enjoy the dance while singing. That is, according to the karaoke apparatus 1 according to the present embodiment, karaoke entertainment can be further improved, and karaoke that the user can enjoy more can be provided to the user.

  In addition, as described above, the conventional karaoke apparatus is a karaoke apparatus that provides an individual scoring type karaoke for scoring only the song of the singer. It can be said that it is a karaoke device that provides a comprehensive scoring karaoke that can be scored. In other words, it can be said that the karaoke apparatus 1 according to the present embodiment is a karaoke apparatus that provides the next generation of karaoke.

  In the present embodiment, the model image storage unit 102 normally stores model images in advance. The model image may be a model image configured according to music data reproduced by the music reproduction unit 111, for example. In this case, the karaoke apparatus 1 includes, for example, a character information storage unit that stores character information that is information indicating a character, and a model information storage unit that stores model information that is information indicating movement of a model. And a model image configuration unit that configures a model image that is an image showing a character that moves according to the movement of the model indicated by the model information using the character information and the model information. . Then, the model image configured by the model image configuration unit is accumulated in the model image storage unit 102.

  Further, the data structure of the “model information” is the same as the data structure of the skeleton information, and thus the description thereof is omitted. Further, the method and procedure for the model image construction unit to construct the model image are the same as the method and procedure for the model image construction unit 109 to construct the model image using the character information and the skeleton information. The description is omitted.

  In the present embodiment, the motion determination information storage unit 101 normally stores one or more pieces of motion determination information. The one or more pieces of movement determination information are usually movement determination information corresponding to one or more predetermined timings. The one or more pieces of movement determination information are movement determination information for determining the movement of the singer at each of one or more predetermined timings. The timing is, for example, a frame number in the music data. The timing may be, for example, an elapsed time from the start of reproduction of music data, a so-called time stamp, or the like. That is, the one or more pieces of motion determination information are normally associated with music data and information indicating timing in the music data.

  In the present embodiment, one or more singer images (still images) acquired by the photographing unit 105 are usually associated with information indicating timing from the start of photographing. The said timing is a frame number in a moving image, for example. The timing may be, for example, an elapsed time from the start of shooting, a so-called time stamp, or the like. That is, the photographing unit 105 normally associates information indicating the timing from the start of photographing with each acquired one or more singer images.

  Moreover, in this Embodiment, the imaging | photography part 105 usually starts acquisition of a singer image (photographing of a singer) simultaneously with the reproduction | regeneration of music data being started. The “simultaneous” also includes “almost simultaneous”. That is, when the music reproducing unit 111 starts reproducing the music data, the photographing unit 105 starts acquiring the singer image.

  As described above, in the present embodiment, the score calculation unit 107 normally obtains the motion determination information corresponding to the music data being played back by the music playback unit 111 and the timing of the music data being played back. Obtained from the information storage unit 101. Then, the score calculation unit 107 scores the movement of the singer indicated by the skeleton information using the skeleton information acquired using the singer image corresponding to the frame number and the acquired movement determination information. And calculate the score. Note that, for example, when no motion determination information is associated with the music data played by the music playback unit 111, the score calculation unit 107 does not normally calculate a score.

  Specifically, for example, it is assumed that the music playback unit 111 is playing back music data identified by the music data identification information “music01”. Further, it is assumed that the photographing unit 105 has acquired a singer image of the 10th frame. In such a case, the score calculation unit 107 first acquires motion determination information corresponding to the music data identification information “music01” and the frame number “10th frame” from the motion determination information storage unit 101. Then, the score calculation unit 107 uses the skeleton information acquired by using the singer image of the 10th frame and the acquired motion determination information, and the singing indicated by the skeleton information corresponding to the 10th frame. A person's movement (singer's movement copied to the singer image of the 10th frame) is scored.

  Further, for example, it is assumed that the music reproducing unit 111 is reproducing music data identified by the music data identification information “music01”. Assume that the photographing unit 105 acquires a singer image 10 seconds after the start of photographing. In such a case, the score calculation unit 107 first acquires, from the motion determination information storage unit 101, motion determination information corresponding to the music data identification information “music01” and the elapsed time “10 seconds” from the start of reproduction. Then, the score calculation unit 107 uses the skeleton information acquired using the singer image 10 seconds after the start of shooting and the acquired motion determination information, and the corresponding skeleton 10 seconds after the start of shooting. The movement of the singer indicated by the information (the movement of the singer shown in the singer image 10 seconds after the start of the shooting) is scored.

  In the present embodiment, the skeleton information acquisition unit 106 detects, for example, a microphone area (hereinafter, appropriately referred to as a microphone area) from a singer image, and an area adjacent to the microphone area or the You may detect the area | region containing a microphone area | region as a singer area | region. The microphone region is detected by image processing such as contour extraction and pattern recognition. Thereby, skeleton information acquisition part 106 is actually singing a song, when two or more persons who can become a singer are copied in the singer image which photography part 105 acquired by photography, for example. Only a user (singer) area can be detected from the singer image as a singer area.

  In the present embodiment, for example, when the singer image is a moving image having one or more still images (when the singing image is a moving image composed of one or more frames), the skeleton information acquisition unit 106 may include the one or more The skeleton information corresponding to each of the one or more still images is acquired using each of the still images. In this case, the skeleton information acquisition unit 106 may acquire skeleton information corresponding to each of the one or more still images using, for example, one or more still images that satisfy a predetermined condition. In addition, for example, when the singer image is a still image (one or more still images included in the moving image or one or more frames constituting the moving image), the skeleton information acquisition unit 106 Skeleton information corresponding to the singer image is acquired using the still image. In this case, for example, the skeleton information acquisition unit 106 may acquire skeleton information corresponding to each of the one or more singer images by using one or more still images that satisfy a predetermined condition. . Note that “predetermined conditions” are, for example, the timing of still images in a video (frame number, elapsed time, etc.), conditions regarding whether a singer photographed in a still image is moving or stationary, etc. It is.

  Further, for example, each time the skeleton information acquisition unit 106 acquires skeleton information, the score calculation unit 107 calculates a score corresponding to the still image (frame) from which the skeleton information is acquired, using the skeleton information. May be. The score calculation unit 107 corresponds to the still image from which the skeleton information is acquired using, for example, skeleton information that satisfies a predetermined condition among the one or more skeleton information acquired by the skeleton information acquisition unit 106. A score may be calculated. Note that “predetermined conditions” are, for example, the timing of still images in a video (frame number, elapsed time, etc.), conditions regarding whether a singer photographed in a still image is moving or stationary, etc. It is.

  In addition, the score calculation unit 107 may statistically process two or more calculated scores and calculate a new score, for example. The statistical processing is, for example, calculating a total or calculating an average. The average may be a simple average or a weighted average. When calculating a weighted average, weights for two or more scores (two or more timings corresponding to two or more scores) are usually determined in advance. Further, the weight is normally held in advance by the score calculation unit 107. Further, the score calculation unit 107 may further perform statistical processing on the calculated two or more new scores to calculate a new score, for example.

  For example, when calculating a weighted average of five scores corresponding to each frame from the fifth frame to the ninth frame and calculating a new score, the score calculation unit 107 calculates each score from the fifth frame to the ninth frame. Five weights corresponding to the frame are acquired. Then, the score calculation unit 107 calculates a weighted average using the five scores and the five weights, and calculates a new score.

  Moreover, in this Embodiment, the score calculation part 107 may consider a singer's movement delay with respect to the movement of a model, for example, and may score a singer's movement. Specifically, in this case, the score calculation unit 107 uses the motion determination information stored in the motion determination information storage unit 101 and the skeleton information acquired by the skeleton information acquisition unit 106, for example, to determine the motion determination information. The degree of delay indicating the degree of delay of the singer's movement indicated by the skeleton information with respect to the movement of the model indicated by is calculated. And the score calculation part 107 scores a singer's movement according to the said delay degree, and calculates the score which is the result of the said scoring. The “delay degree” is, for example, the number of frames that are delayed, the number of seconds that are delayed, and the like. “According to the degree of delay” means, for example, that there is no delay, or that the degree of delay is reflected in the score.

  The above-mentioned “considering that there is no delay” means, for example, scoring the movement of a singer copied in the singer image using movement determination information corresponding to a frame number different from the frame number of the singer image. It is to be. In other words, “considering that there is no delay” means, for example, using motion determination information indicating the movement of the model most similar to the movement of the singer indicated by the skeleton information acquired using one singer image. Then, scoring the movement of the singer shown in the one singer image. “Using the motion determination information indicating the movement of the model most similar to the movement of the singer” means, for example, that the highest score among the scores obtained as a result of scoring using two or more motion determination information is acquired. That is. The above-mentioned “reflecting the degree of delay in the score” means, for example, multiplying the calculated score by a coefficient corresponding to the degree of delay.

  For example, the movement of the singer can be scored using the motion determination information corresponding to the frame number that satisfies a predetermined condition with the frame number of the singer image (still image) acquired by the photographing unit 105. Suppose that it is decided beforehand. The “predetermined condition” is assumed to be +5 frames from the target frame number. In this case, it is assumed that the photographing unit 105 acquires a singer image of the 10th frame. In such a case, the score calculation unit 107 uses, for example, motion determination information corresponding to each frame from the 5th frame to the 10th frame, and the skeleton information acquired using the singer image of the 10th frame. Score the singers' movements shown. And the score calculation part 107 acquires the highest score among the calculated six scores as a score corresponding to the singer image of the 10th frame.

  In addition, for example, the score calculation unit 107 acquires the score calculated using the motion determination information of the eighth frame among the calculated six scores as a score corresponding to the singer image of the tenth frame. And In such a case, for example, the score calculation unit 107 calculates a delay degree “10−8 = 2”. And the score calculation part 107 calculates the coefficient according to the said delay degree. The calculation method of the coefficient does not matter. Then, the score calculation unit 107 calculates a new score by multiplying the score corresponding to the acquired 10th-frame singer image by the calculated coefficient.

  Needless to say, the frame number in the above description may be an elapsed time.

  As described above, in the present embodiment, the score calculation unit 107 uses, for example, the degree of coincidence of the singer's movement with respect to the movement of the model and the degree of delay of the singer's movement with respect to the movement of the model. Is scored, and a score as a result of the scoring is calculated. In this case, the score calculation unit 107 calculates the score using, for example, a score calculation formula. Specifically, the score calculation unit 107 substitutes the degree of coincidence and the degree of delay as parameters in the score calculation formula.

  Moreover, in this Embodiment, the karaoke apparatus 1 does not need to be provided with the score calculation part 107. FIG. In this case, the karaoke apparatus 1 receives, for example, the score calculated by the above-described score calculation apparatus (hereinafter, as appropriate, the score calculation apparatus) from the score calculation apparatus. Then, the score output unit 108 outputs the received score. Note that the processes and operations performed by the score calculation apparatus are the same as the processes and operations performed by the score calculation unit 107, and thus description thereof will be omitted.

  In the present embodiment, the karaoke apparatus 1 may not include the model image configuration unit 109. In this case, the image output unit 110 outputs only the model image.

  In the present embodiment, the model image output from the image output unit 110 is usually associated with the music data played back by the music playback unit 111. That is, when outputting a model image, the image output unit 110 normally acquires a model image associated with the music data reproduced by the music reproduction unit 111 from the model image storage unit 102. Then, the image output unit 110 outputs the acquired model image. Further, for example, when no model image is associated with the music data reproduced by the music reproduction unit 111, the image output unit 110 normally does not output the model image.

  In the present embodiment, the image output unit 110 normally also outputs a lyrics image that is an image showing lyrics. The lyrics image is normally stored in a predetermined storage area. The predetermined storage area may be the music data storage unit 103, for example. The lyrics image to be output usually corresponds to the music data reproduced by the music reproducing unit 111. That is, the image output unit 110 normally acquires a lyric image associated with music data being reproduced by the music reproduction unit 111 from a predetermined storage area. Then, the image output unit 110 outputs the acquired lyrics image. Moreover, when outputting a lyric image, a synthetic | combination image structure part may comprise a synthetic | combination image also using a lyric image, for example.

  Further, in the present embodiment, when a model image is output, normally, the reproduction of the music data is started and the output of the model image is started at the same time. The “simultaneous” also includes “almost simultaneous”. That is, when the music reproducing unit 111 starts reproducing the music data, the image output unit 110 starts outputting the model image.

  Further, in the present embodiment, when a model image is output, normally, the reproduction of the music data is started and the output of the model image is started at the same time. The “simultaneous” also includes “almost simultaneous”. That is, when the music reproducing unit 111 starts reproducing the music data, the photographing unit 105 starts photographing the singer. In response to the start of shooting by the shooting unit 105, the model image construction unit 109 constructs a model image, and the image output unit 110 starts outputting the model image.

  Further, when outputting the model image and the model image, the image output unit 110 may output the model image and the model image, for example, in an overlapping manner. In this case, for example, the image output unit 110 outputs the model image and the model image so that the model image overlaps the model image. In this case, the image output unit 110 may output the model image and the model image so that the model image overlaps the model image, for example. In this case, for example, the image output unit 110 may perform a so-called translucent process on an image that overlaps another image. This also makes it easier for the user to see whether the dance is performed according to the model.

  Moreover, in this Embodiment, the karaoke apparatus 1 may be provided with the song scoring part 212 which scores the song which a singer sings, for example. The singing scoring unit 212 usually scores a song sung by a singer, and calculates a score (hereinafter referred to as a song score as appropriate) as a result of the scoring. In addition, since the method, the procedure, etc. of scoring the song which a singer sings (a method, a procedure, etc. which calculate a song score) are well-known, detailed description is abbreviate | omitted.

  In addition, when the singing scoring unit 212 is provided, the karaoke apparatus 1 normally replaces the score calculation unit 107 with, for example, a score obtained by scoring the movement of the singer (hereinafter referred to as a dance score as appropriate), A score calculation unit 207 that calculates a score (hereinafter, referred to as an overall score as appropriate) that is a result of scoring the song of the singer and the movement of the singer using the song score may be provided. In this case, the score output unit 108 normally outputs the total score calculated by the score calculation unit 207. The score output unit 108 may output one or more of the total score or dance score calculated by the score calculation unit 207 and the song score acquired by the singing scoring unit 212, for example.

  The procedure for calculating the total score is, for example, as follows. That is, for example, the score calculation unit 207 holds in advance a calculation formula for calculating the total score (hereinafter, appropriately referred to as a total score calculation formula). The “total score calculation formula” usually has variables for substituting a dance score and a song score. Then, for example, the score calculation unit 207 substitutes the calculated dance score and the song score calculated by the singing scoring unit into the total score calculation formula, and calculates the total score calculation formula after the substitution, Calculate the total score. The “total score calculation formula” may be a so-called function (program). The “total score calculation formula” is preferably an increase function for the dance score and song score.

  Further, it can be said that the “total score” is a performance score that is a score of performance (song and dance) by a singer.

  Moreover, the block diagram of the karaoke apparatus 2 which is a karaoke apparatus provided with the singing scoring part 212 and the score calculation part 207 is FIG. In FIG. 17, the karaoke apparatus 2 includes a motion determination information storage unit 101, a model image storage unit 102, a music data storage unit 103, a reception unit 104, a photographing unit 105, a skeleton information acquisition unit 106, a score calculation unit 207, and a score output. Unit 108, model image construction unit 109, image output unit 110, music playback unit 111, and singing scoring unit 212.

  In each of the above embodiments, each process or each function may be realized by centralized processing by a single device or a single system, or distributed by a plurality of devices or a plurality of systems. It may be realized by being processed.

  In each of the above embodiments, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  Moreover, the software which implement | achieves the karaoke apparatus in each said embodiment is the following programs, for example. That is, this program uses a computer to shoot a singer, obtain a singer image that is an image of the singer, and information included in the singer image. Using the skeleton information acquisition unit that acquires skeleton information that is information indicating movement, and the skeleton information acquired by the skeleton information acquisition unit, the movement of the singer indicated by the skeleton information is scored, and the result of the scoring It is a program for functioning as a score calculation unit for calculating a score and a score output unit for outputting the score calculated by the score calculation unit.

  In the program, the functions realized by the program do not include functions that can be realized only by hardware.

  The program may be executed by being downloaded from a server or the like, or a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, or the like) is read out. May be executed. Further, this program may be used as a program constituting a program product.

  Moreover, the computer which performs the said program may be single, and plural may be sufficient as it. That is, centralized processing may be performed, or distributed processing may be performed.

  FIG. 18 is a schematic diagram of a computer system 9 that executes the above-described program to realize the karaoke apparatus of the above-described embodiment. The above-described embodiments can be realized by computer hardware and a computer program executed thereon.

  In FIG. 18, the computer system 9 includes a computer 901 including a CD-ROM drive 9011 and an FD drive 9012, a keyboard 902, a mouse 903, and a monitor 904.

  FIG. 19 is a block diagram of the computer system 9. In FIG. 19, in addition to a CD-ROM drive 9011 and an FD drive 9012, a computer 901 is connected to an MPU 9013, a ROM 9014 for storing a program such as a boot-up program, and an MPU 9013, and temporarily commands the application program. And a hard disk 9016 for storing application programs, system programs, and data, a CD-ROM drive 9011, an FD drive 9012, an MPU 9013, etc. 9017. Although not shown here, the computer 901 may further include a network card that provides connection to a LAN.

  A program that causes the computer system 9 to execute the functions of the karaoke apparatus or the like of the above-described embodiment is stored in the CD-ROM 9101 or FD 9102, inserted into the CD-ROM drive 9011 or FD drive 9012, and further stored in the hard disk 9016. May be forwarded. Alternatively, the program may be transmitted to the computer 901 via a network (not shown) and stored in the hard disk 9016. The program is loaded into the RAM 9015 when executed. The program may be loaded directly from the CD-ROM 9101, the FD 9102, or the network.

  The program does not necessarily include an operating system (OS), a third party program, or the like that causes the computer 901 to execute the functions of the karaoke apparatus of the above-described embodiment. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 9 operates is well known and will not be described in detail.

  The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the karaoke apparatus according to the present invention has an effect that it can provide a karaoke apparatus that accurately scores a dance, and is useful as a so-called communication karaoke apparatus.

DESCRIPTION OF SYMBOLS 1 Karaoke apparatus 101 Motion determination information storage part 102 Model image storage part 103 Music data storage part 104 Reception part 105 Shooting part 106 Skeleton information acquisition part 107 Score calculation part 108 Score output part 109 Model image structure part 110 Image output part 111 Music Playback section

Claims (15)

A shooting section for shooting a singer and obtaining a singer image that is an image of the singer;
Using information included in the singer image, a skeleton information acquisition unit that acquires skeleton information that is information indicating the movement of the singer,
A karaoke apparatus provided with the score output part which outputs the score which is the result of scoring the movement of the said singer using the said skeleton information. The singer image is an image including distance information,
The skeleton information acquisition unit
The karaoke apparatus of Claim 1 which acquires the said skeleton information using the distance information contained in the said singer image. The karaoke apparatus according to claim 2, wherein the information included in the singer image is distance information corresponding to the area of the singer in the distance information. A motion determination information storage unit storing motion determination information which is information for determining the movement of the singer;
Using the movement determination information and the skeleton information, further determining a movement of a singer indicated by the skeleton information, and further including a score calculation unit that calculates the score using a result of the determination;
The score output unit
The karaoke apparatus as described in any one of Claims 1-3 which outputs the score which the said score calculation part calculated. In the movement determination information storage unit,
One or more movement determination information for determining the movement of the singer at every predetermined timing is stored,
The skeleton information acquisition unit
Using the information included in the singer image, obtain one or more skeleton information indicating the movement of the singer at one or more predetermined timings,
The score calculation unit
The movement of the singer indicated by each of the one or more skeleton information acquired by the skeleton information acquisition unit is determined using the movement determination information corresponding to each of the one or more timings, and the score is used using the determination result. The karaoke apparatus according to claim 4 which calculates The score calculation unit
A score that is a weighted average of the one or more scores is obtained by scoring the movement of the singer at each of the predetermined timings, calculating one or more scores as a result of scoring at the one or more timings. 6. The karaoke apparatus according to claim 5, wherein the karaoke device is calculated. The movement determination information is one or more joint angle conditions that are conditions relating to the angle of one or more joints of the singer,
The score calculation unit
The skeleton information acquired by the skeleton information acquisition unit is used to acquire angles of one or more joints of the singer, and the joint angle conditions corresponding to the one or more joints are used. The karaoke apparatus according to any one of claims 4 to 6, wherein an angle of a joint is determined and the score is calculated using a result of the determination. The movement determination information is an angle condition between one or more joints that is a condition related to an angle between one or more joints of the singer,
The score calculation unit
Using the skeleton information acquired by the skeleton information acquisition unit, an angle between one or more joints of the singer is acquired, and the inter-joint angle condition corresponding to the one or more joints is used to determine the 1 The karaoke apparatus according to any one of claims 4 to 7, wherein an angle between the joints is determined, and the score is calculated using a result of the determination. The movement determination information is one or more joint coordinate conditions that are conditions related to the coordinates of one or more joints of the singer,
The score calculation unit
Using the skeleton information acquired by the skeleton information acquisition unit, acquire the coordinates of one or more joints of the singer, and use the joint coordinate conditions corresponding to the one or more joints, The karaoke apparatus according to any one of claims 4 to 8, wherein coordinates of a joint are determined, and the score is calculated using a result of the determination. The movement determination information is one or more joint movement amount conditions that are conditions relating to the movement amount of each of the one or more joints of the singer.
The score calculation unit
Using the skeleton information acquired by the skeleton information acquisition unit, the movement amount of one or more joints of the singer is acquired, and the one or more joint movement amount conditions corresponding to the one or more joints are used. The karaoke apparatus according to any one of claims 4 to 9, wherein a movement amount of each joint is determined, and the score is calculated using a result of the determination. The karaoke apparatus according to any one of claims 1 to 10, wherein the information included in the singer image is a singer image after deleting a pixel that satisfies a predetermined condition from the singer image. A model image storage unit in which a model image which is an image showing a model of dance is stored;
The karaoke apparatus according to any one of claims 1 to 11, further comprising an image output unit that outputs the model image. Using the information included in the singer image, one or more attribute values of the singer are obtained, and at least a part of the model image is a model image that indicates a model using the one or more attribute values of the singer. A model image construction unit
The image output unit includes:
The karaoke apparatus according to claim 12, wherein the model image and the model image configured by the model image configuration unit are output. A dance scoring method performed using an imaging unit, a skeleton information acquisition unit, and a score output unit,
The photographing unit is
A shooting step of shooting a singer and obtaining a singer image that is an image of the singer;
The skeleton information acquisition unit
Using the information included in the singer image, a skeleton information acquisition step of acquiring skeleton information that is information indicating the movement of the singer,
The score output unit
A dance scoring method comprising: a score output step of outputting a score that is a result of scoring the movement of the singer using the skeleton information. Computer
A shooting section for shooting a singer and obtaining a singer image that is an image of the singer;
Using information included in the singer image, a skeleton information acquisition unit that acquires skeleton information that is information indicating the movement of the singer,
The program for functioning as a score output part which outputs the score which is the result of scoring the movement of the singer using the skeleton information.