JP2016183989A - Information processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to heighten the sense of presence as if it were singing before an audience.SOLUTION: An information processing device comprises: acquisition means for acquiring musical data indicating a piece of music; output control means for making the piece of music that the musical data acquired by the acquisition means indicates outputted by output means; detection means for detecting a motion of a user while the piece of music is being outputted by the output means; determination means for determining motion information indicating a repetitive motion on the basis of the motion detected by the detection means; and display control means for making a video indicating an object that does the repetitive motion that the motion information determined by the determination means indicates displayed by display means while the piece of music is being outputted by the output means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technical field of an information processing apparatus that displays an image showing a spectator during output of karaoke music.

  Patent Document 1 discloses a karaoke device that enables a similar simulated experience when a user sings in front of an audience. Specifically, Japanese Patent Laid-Open No. 2004-26883 discloses a singer that is reproduced on a display device by playing back a video that records the reaction of the audience when the singer sang on the stage, and a sound that is reproduced by a speaker in the background. It is disclosed that the user sings while watching the reaction of the audience.

JP 2004-287362 A

  However, the audience reaction displayed on the display device is recorded in advance, and the audience reaction does not change according to the operation when the user actually sings. For this reason, the sense that the audience is responding to the user is diminished, and there is a lack of realism as if the user is singing in front of the audience.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an information processing apparatus and the like that can enhance a sense of reality as if singing in front of a spectator.

  In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that an acquisition means for acquiring music data indicating a music and an output control for causing the output means to output a music indicated by the music data acquired by the acquisition means. Means for detecting a user action during the output of the music by the output means, a determination means for determining action information indicating a repetitive action based on the action detected by the detection means, And display control means for displaying on the display means a video indicating an object that performs the repetitive motion indicated by the motion information determined by the determination means while the music is being output by the output means.

  According to a second aspect of the present invention, in the information processing apparatus according to the first aspect, the determination unit is configured to detect a user's motion less than or equal to a predetermined amount of exercise by the detection unit after the detection unit starts detecting the operation of the user. The motion information is determined based on the motion of the user detected until the motion is detected.

  According to a third aspect of the present invention, in the information processing apparatus according to the second aspect, the predetermined momentum is detected by the detection unit after a first predetermined time has elapsed since the detection unit started detecting the user's movement. The motion information is determined based on the user motion detected until the following user motion is detected.

  According to a fourth aspect of the present invention, in the information processing apparatus according to any one of the first to third aspects, the identification that identifies the operation repeatedly performed by the user from among the operations detected by the detecting means. Means for determining the action information indicating the repetitive action performed by the object based on the action specified by the specifying means.

  According to a fifth aspect of the present invention, in the information processing apparatus according to any one of the first to fourth aspects, the information processing apparatus further includes predetermined operation information storage means for storing a plurality of predetermined operation information indicating a predetermined repetitive operation. The determining unit determines any predetermined operation information among a plurality of predetermined operation information stored in the predetermined operation information storage unit based on the operation detected by the detection unit; The display means displays an image showing the object performing the action indicated by the predetermined action information determined by the determining means.

  According to a sixth aspect of the present invention, in the information processing apparatus according to the fifth aspect, the determining unit is detected by the detecting unit among the plurality of predetermined operation information stored in the predetermined operation information storage unit. Predetermined operation information indicating an operation similar to the detected operation is determined, and when the predetermined operation information indicating an operation similar to the operation detected by the detection unit is not stored in the predetermined operation information storage unit, the detection unit The motion information indicating the detected motion is determined as the motion information indicating the repeated motion performed by the object.

  The invention according to claim 7 is the information processing apparatus according to any one of claims 1 to 6, wherein the acquisition unit acquires the music data indicating a music composed of a plurality of performance parts, The apparatus further comprises assigning means for assigning the motion information determined by the determining means to any one of the plurality of performance parts, and the display control means includes a performance part to which the motion information is assigned by the assigning means. When output by the output means, the display means displays a video showing the object performing the repetitive motion indicated by the motion information.

  The invention according to claim 8 is the information processing apparatus according to claim 7, wherein the acquisition means indicates the music data indicating music composed of a plurality of performance parts including two or more performance parts of the same type. And the assigning means outputs the operation information determined by the determining means and the output means when the detecting means detects the user's action for which the action information is determined by the determining means. The type information indicating the type of the performance part that has been output is associated with and stored in the type information storage means, and the display control means includes the type of performance part indicated by the type information stored in the type information storage means. The object that performs a repetitive operation indicated by the operation information stored in the type information storage unit in association with the type information when being output by the output unit. An image indicating the transfected and wherein the to be displayed on the display means.

  According to a ninth aspect of the present invention, in the information processing apparatus according to any one of the first to eighth aspects, the request information indicating a request to execute an operation is output during the output of the music by the output unit. A second display control unit for displaying the operation information on the basis of an operation detected by the detection unit after the request information is displayed on the display unit. Features.

  According to a tenth aspect of the present invention, in the information processing apparatus according to any one of the first to ninth aspects, the determination unit is configured to perform a second predetermined operation after the detection unit starts detecting the user's action. If the repetitive motion is not determined before the time elapses, the user motion detected by the detection means is detected before the second predetermined time elapses after the detection of the user motion is started. The operation information is determined based on the information.

  According to an eleventh aspect of the present invention, in the information processing apparatus according to any one of the first to tenth aspects, the speed at which the repetitive motion indicated by the motion information determined by the determining means is performed is the repetitive speed. The apparatus further includes an adjusting unit that adjusts based on a cycle of the operation and a tempo of the music indicated by the music data acquired by the acquiring unit, and the display control unit indicates the operation information determined by the determining unit An image showing the object for performing a repetitive operation at a speed adjusted by the adjusting means is displayed on the display means.

  The invention according to claim 12 is an acquisition step of acquiring music data indicating a music, an output control step of outputting a music indicated by the music data acquired by the acquisition step by an output means, and the output means by the output means. A detection step for detecting a user's motion during the output of the music, a determination step for determining motion information indicating a repetitive motion based on the motion detected by the detection step, and the output of the music by the output means Further, the display control step of causing the display means to display a video indicating the object that performs the repetitive motion indicated by the motion information determined in the determination step is performed by the computer.

  According to the first or twelfth aspect of the present invention, an image of an object that repeatedly performs an operation is displayed in response to the user performing an operation. Therefore, when the user views the object as a spectator, it is possible to enhance a sense of presence as if singing in front of the spectator.

  According to the second aspect of the present invention, it is possible to appropriately detect the end of the user's operation.

  According to the third aspect of the present invention, it is possible to determine the repetitive motion based on the user motion during the first predetermined time at least.

  According to the fourth aspect of the present invention, when the user repeatedly performs a motion, it is possible to display an image of an object that performs the repetitive motion.

  According to the fifth aspect of the present invention, even if the user performs an operation that does not meet some criteria, an object that performs a repetitive operation that meets the criteria can be displayed.

  According to the sixth aspect of the present invention, even when the user performs a dissimilar operation to a predetermined repetitive operation, an object that performs the repetitive operation can be displayed.

  According to the seventh aspect of the present invention, it is possible to display an object that performs repetitive actions in units of performance parts.

  According to the eighth aspect of the present invention, when a performance part of the same type as the performance part performed by the user who has performed the operation in which the repetitive motion is detected is output, the object that performs the repetitive motion can be displayed. .

  According to the ninth aspect of the invention, the user can start the operation by viewing the displayed request information. Therefore, it is possible to appropriately detect the user's operation for determining the repetitive operation.

  According to the tenth aspect of the present invention, it is possible to prevent a situation in which an object that performs a repetitive motion is not displayed indefinitely despite the user performing a motion.

  According to the eleventh aspect of the present invention, it is possible to display an object that repeatedly performs an operation in accordance with the tempo of the music to be output.

It is a figure showing an example of outline composition of karaoke system S of one embodiment. (A) is a figure which shows the structural example of a music list. (B) is a figure showing an example of composition of music composition information. (C) is a figure which shows the structural example of the parameter information for operation | movement detection. (D) is a figure which shows the example of the content of default repetition operation | movement. (E) is an example of the relative momentum stored in the RAM. (A) is a figure which shows the example of a display of input operation execution request information. (B) is a figure which shows typically an example of a mode that a singer performs the operation | movement which shakes a hand. It is a figure which shows an example of the data flow of operation data. It is a flowchart which shows an example of the performance drawing main process in the karaoke apparatus. (A) is a flowchart which shows the 1st example of the audience operation | movement setting process in the karaoke apparatus 2. FIG. (B) is a flowchart showing a second example of audience operation setting processing in the karaoke apparatus 2. (A) is a flowchart which shows an example of the designated operation | movement production | generation process in the karaoke apparatus 2. FIG. (B) is a flowchart illustrating an example of an input repetition operation generation process in the karaoke apparatus 2. (C) is a figure which shows an example of the default repetition operation | movement acquisition process in the karaoke apparatus 2. FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, embodiment described below is embodiment at the time of applying this invention with respect to the karaoke system utilized in a karaoke store, for example.

[1. Outline of configuration and operation of karaoke system]
First, the outline of the configuration of the karaoke system of the present embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating a schematic configuration example of the karaoke system S of the present embodiment. As shown in FIG. 1, the karaoke system S includes a music selection device 1, a karaoke device 2, and a distance image camera 3. The karaoke apparatus 2 is an example of an information processing apparatus of the present invention. The music selection device 1, the karaoke device 2, and the distance image camera 3 are installed, for example, in a karaoke room (guest room) of a karaoke store. The music selection device 1 can communicate with the karaoke device 2 in the same room, for example, by infrared rays. The music selection device 1 and the karaoke device 2 can communicate with each other via, for example, a LAN (Local Area Network) and an access point AP. The karaoke device 2 and the distance image camera 3 can be connected. For example, it may be connected using a USB (Universal Serial Bus) cable or the like, or may be connected via a LAN.

  The music selection device 1 is used by a user who uses a guest room of a karaoke store to select karaoke music.

  The range image camera 3 is a camera that can capture a range image. The distance image is an image representing the distance from the distance image camera 3 to the subject for each pixel. The distance image camera 3 is used for detecting an operation performed by a singer in accordance with the music. Specifically, the distance image camera 3 detects distances from a position where the distance image camera 3 is installed to a plurality of body parts of the singer. The body part from which the distance is detected may be, for example, a joint such as a neck, shoulder, elbow, wrist, waist, knee, or ankle. For example, the distance image camera 3 generates distance image data indicating a pixel value indicating the distance to the subject for each pixel, and transmits the distance image data to the karaoke apparatus 2. The distance image camera 3 captures a distance image at a predetermined frame rate, and transmits distance image data every time a distance image is captured. The frame rate at this time may be the same as the frame rate when the display 5 displays an image, for example. The distance image camera 3 may include, for example, an LED (light-emitting diode) that emits infrared rays and a CCD image sensor that receives infrared rays. In this case, for example, the distance image camera 3 measures the time until the infrared light emitted from the LED is reflected by the subject and received by the CCD image sensor. Then, the distance image camera 3 determines the distance to the subject for each pixel based on the measured time. The distance image camera 3 may be a stereo camera including a plurality of CCD image sensors, for example. In this case, the distance image camera 3 may calculate the distance to the subject for each pixel based on a plurality of images taken by a plurality of CCD image sensors, and generate distance image data.

  The karaoke apparatus 2 is a performance apparatus that reproduces karaoke music selected by the user. The karaoke apparatus 2 is connected to a speaker 4 and a display 5. The speaker 4 is an example of the output means of the present invention. The display 5 is an example of the display means of the present invention. The karaoke apparatus 2 outputs karaoke music by the speaker 4. During the reproduction of karaoke music, the karaoke device 2 detects the operation of the singer based on the distance image transmitted from the distance image camera 3 at a predetermined frame rate. The karaoke apparatus 2 determines a repetitive operation based on the detected operation. The repetitive operation is an operation performed repeatedly a plurality of times. During the reproduction of the karaoke music, the karaoke apparatus 2 displays a video showing the spectator character performing the determined repetitive motion on the display 5. The audience character is an object in the form of, for example, a human being, an animal, a virtual creature, or a robot. The karaoke apparatus 2 may display one spectator character or may display a plurality of spectator characters.

[2. Configuration of Karaoke Device 2]
Next, with reference to FIG.1 and FIG.2, the structure of the karaoke apparatus 2 is demonstrated. As shown in FIG. 1, the karaoke apparatus 2 includes a control unit 21, a storage unit 22, a video RAM 23, a video reproduction unit 24, a video control unit 25, a microphone control unit 26, a synthesizer 27, a mixing amplifier 28, an operation unit 29a, and an operation. A processing unit 29b, an infrared receiving unit 29c, a LAN communication unit 29d, and the like are provided. The control unit 21 includes a CPU, a ROM, a RAM, and the like. The control unit 21 has a clock function and a clock function. The storage unit 22 is configured by, for example, a hard disk. The storage unit 22 is an example of each of predetermined operation information storage means and type information storage means. The predetermined operation information storage unit and the type information storage unit may be the same storage device as the predetermined operation information storage unit and the type information storage unit, or may be different storage devices. The storage unit 22 stores programs such as an OS, a karaoke device processing program, a music sequencer, and a 3D engine. The control unit 21 performs overall control of the entire karaoke apparatus 2 by executing the OS and the karaoke apparatus processing program. The karaoke device processing program may be downloaded from a predetermined server device, or may be recorded on a recording medium and read via a drive device.

  The storage unit 22 stores music data, music list, music composition information, motion detection parameter information, default repeated motion data, and normal motion data. The music data indicates karaoke music. The music data is data for outputting the karaoke music by the speaker 4. Examples of music data include MIDI (Musical Instrument Digital Interface) data, data obtained by encoding live performance sounds using PCM (Pulse Code Modulation), and the like. In the present embodiment, a case where MIDI data is used as music data will be described as an example. The music data includes the original tempo of the karaoke music. For each karaoke song, song data and song ID are stored in the storage unit 22 in association with each other. The music ID is identification information that can identify the karaoke music. For example, the music ID may be a music number.

  FIG. 2A is a diagram illustrating a configuration example of a music list. The music list is a list in which information on karaoke music is registered. Specifically, in the music list, for each karaoke music, the music ID and the audience action designation part type are registered. The audience action designation part type is a type of performance part on which a video of a spectator character performing a repetitive action determined by the karaoke apparatus 2 is displayed. That is, when the performance part indicated by the audience action designation part type is output from the speaker 4, an image of the audience character performing the repeated action determined by the karaoke apparatus 2 is displayed. Karaoke music is composed of a plurality of performance parts. Examples of types of performance parts include intro, A melody, B melody, interlude, rust, and outro. A plurality of performance parts having the same type may be included in one piece of music. One audience motion designation part type may be set for one music piece, or a plurality of audience motion designation part types may be set. Further, for example, the administrator of the karaoke system S may set the audience operation designation part type for each karaoke piece.

  FIG. 2B is a diagram illustrating a configuration example of music configuration information. The music composition information is information relating to a plurality of performance parts constituting the karaoke music. For each karaoke song, the song composition information and the song ID are stored in the storage unit 22 in association with each other. Specifically, the music composition information includes a performance number and a part type for each performance part constituting the karaoke music. The performance number is a number indicating the order in which the performance parts of the type indicated by the part type are played among the plurality of performance parts constituting the karaoke music piece.

  FIG. 2C is a diagram illustrating a configuration example of the motion detection parameter information. The motion detection parameter information includes predetermined parameters for detecting the motion of the singer. Specifically, the motion detection parameter information includes a detection determination time, a detection determination exercise amount, a maximum acquisition time, and a minimum acquisition time. The detection determination time and the detection determination momentum are parameters for determining whether or not the detection of the singer's motion is to be terminated. Specifically, during the detection of the singer's motion, the control unit 21 detects the singer's motion when the momentum of the singer's body part during the latest predetermined time indicated by the detection determination time is equal to or less than the detection determination momentum. End. The maximum acquisition time is the maximum value of the time for detecting the singer's movement. The minimum acquisition time is the minimum value of the time for detecting the singer's movement.

  The predetermined repetitive motion data is motion data that defines a predetermined repetitive motion that can be performed by the spectator character. The action data is data indicating actions to be performed by the audience character or actions performed by the singer. The motion data includes the coordinates of each body part of the audience character or singer in the three-dimensional space according to the progress of the motion along the time series. For example, every time when the reciprocal time of the frame rate of the video displayed on the display 5 elapses from the start of the operation, the elapsed time may be associated with the coordinates of each body part. The coordinates of the body part are, for example, three-dimensional coordinates. However, the coordinates of the body part may be two-dimensional coordinates. The predetermined repetitive operation is called a default repetitive operation. The predetermined repeated operation is, for example, a natural operation. For each predetermined repetition operation, the predetermined repetition operation data and the predetermined operation ID are stored in the storage unit 22 in association with each other. The default operation ID is identification information for identifying a default repetitive operation. FIG. 2D is a diagram illustrating an example of the contents of the default repetition operation. Examples of the predetermined repetitive operation include an operation of raising a fist, a clapping hand, an operation of shaking a hand, jump, rotation, and the like. For example, each predetermined repetition operation data may be generated so that the tempos of all the predetermined repetition operations coincide. The tempo of the default repeat operation may be, for example, the number of times that the default repeat operation is performed per second.

  The normal motion data is motion data indicating a motion performed by the spectator character in a state where the repetitive motion is not determined based on the singer's motion.

  The control unit 21 executes the 3D engine and sequentially generates still images of the spectator characters based on the motion data. The control unit 21 sequentially outputs the generated still image video signal to the video RAM 23, whereby the video of the spectator character performing the operation is displayed on the display 5.

  The control unit 21 generates various events according to the set tempo by executing the music sequencer. Examples of the event include a MIDI clock, a MIDI event, a performance part switching event, and the like. The MIDI clock is an event output at a time interval corresponding to the reproduction speed. The MIDI event is an event for controlling the reproduction of music. The control unit 21 outputs a MIDI event to the synthesizer 27. Thereby, the control unit 21 causes the speaker 4 to output karaoke music. The performance part switching event is an event for notifying that the performance part to be reproduced has been switched.

  The synthesizer 27 outputs an audio signal of karaoke music to the mixing amplifier 28 based on the MIDI event. The microphone control unit 26 outputs the user's voice signal collected by the microphone to the mixing amplifier 28. Further, the microphone control unit 26 samples the audio signal from the microphone at a predetermined sampling frequency, and outputs the sample data to the control unit 21. The mixing amplifier 28 mixes the audio signal from the synthesizer 27 and the audio signal from the microphone control unit 26 and outputs them to the speaker 4. The video signal and the like are reproduced by the video RAM 23 and the video reproduction unit 24. Then, the video control unit 25 outputs the reproduced video signal to the display 5. Moreover, the control part 21 receives the instruction | indication from the operation part 29a by a user via the operation process part 29b. The control unit 21 can receive various commands from the music selection device 1 via the infrared receiving unit 29c. The LAN communication unit 29d is for communicating by connecting to a wired LAN.

[3. Display of spectator characters performing repetitive actions]
In order to enhance the presence as if the singer is singing in front of the audience, the control unit 21 determines a repetitive action based on the action of the singer during the reproduction of the karaoke music, and performs the determined repetitive action. A spectator character is displayed on the display 5. The repeated action that the spectator character performs is called a designated action.

  First, the control unit 21 detects the operation of the singer based on the distance image transmitted from the distance image camera 3 at a predetermined frame rate during the reproduction of the karaoke music. The detected operation is called an input operation. For example, the control part 21 detects the coordinate of each body part of a singer from a distance image for every distance image. The control unit 21 generates motion data including each body part detected for each distance image in time series. This operation data is referred to as input operation data. The input operation data is data indicating the input operation of the singer. Thereby, the control part 21 detects operation | movement of a singer.

  A method for acquiring coordinates from the distance image will be described. For example, there is a method using machine learning. For example, the administrator of the karaoke system S prepares a number of distance images that show how humans perform various operations. In addition, the administrator prepares coordinates of each body part corresponding to the distance image. For example, the administrator may create the coordinates manually. A set of distance images and coordinates is used as teacher data. The distance image corresponds to an example, and the coordinates correspond to an answer. The administrator inputs teacher data to predetermined information processing for machine learning. The information processing apparatus performs predetermined machine learning using the teacher data, and generates information on a speculation model for estimating the coordinates of each body part from the distance image. The karaoke apparatus 2 stores information on the generated speculation model in advance. The karaoke apparatus 2 determines the coordinates of each body part from the distance image data received from the distance image camera 3 based on the information of the guess model. The method for obtaining coordinates from range image data is disclosed in “Real-Time Human Pose Recognition in Parts from Single Depth Images” published by Jamie Shotton et al. In IEEE CVPR 2011 in 2011. Detailed description is omitted. The distance image camera 3 may determine the coordinates of each body part, for example, based on information on the distance image and the estimation model. Then, the karaoke apparatus 2 may acquire coordinates from the distance image camera 3.

The example of the timing which starts detection of a singer's operation | movement is demonstrated. For example, the control unit 21 inputs input operation execution request information for requesting the singer to execute an operation when reproduction of a performance part of the audience operation designation part type among a plurality of performance parts constituting the karaoke piece is started. May be displayed on the display 5. And the control part 21 may start the detection of a singer's operation | movement from when input operation execution request information is displayed. FIG. 3A is a diagram illustrating a display example of input operation execution request information. For example, as illustrated in FIG. 3A, the control unit 21 may display the input action execution request information 130 on the display 5 together with the singer character 110 and the audience character 120. The singer character 110 is an object indicating a singer. The input operation execution request information 130 may be, for example, a message such as “Please perform an operation for the audience!”. The singer can start the operation by looking at the displayed input operation execution request information.

  Further, for example, the control unit 21 may start detecting the operation of the singer when the momentum of the body part of the singer in the latest detection determination time becomes equal to or greater than the detection determination momentum. For example, during playback of karaoke music, the control unit 21 detects the coordinates of each body part for each distance image transmitted from the distance image camera 3. The control unit 21 calculates the total value of the difference between the body part coordinates detected this time and the body part coordinates detected last time as a relative momentum. And the control part 21 memorize | stores the calculated relative momentum in RAM for every time interval according to a frame rate. FIG. 2E shows an example of relative momentum stored in the RAM. The control part 21 calculates the sum of the relative momentum during the latest detection determination time among the relative momentum memorize | stored in RAM as a momentum of a singer's body part.

  The example of the timing which complete | finishes the detection of a singer's motion is demonstrated. For example, the control unit 21 may end the detection of the singer's motion when the momentum of the singer's body part in the most recent detection determination time becomes equal to or less than the detection determination momentum during the detection of the singer's motion. . That is, the control part 21 produces | generates the input operation data of the operation | movement detected after the detection of the operation | movement of a singer is started until the momentum below a detection determination exercise amount is detected. Thereby, the end of operation | movement of a singer can be detected appropriately.

  Further, for example, when the minimum acquisition time has elapsed since the detection of the singer's motion has started, the control unit 21 has the momentum of the singer's body part in the most recent detection determination time less than or equal to the detection determination momentum. The detection of the singer's movement may be terminated. That is, even if the momentum of the singer's body part in the latest detection determination time becomes equal to or less than the detection determination momentum until the minimum acquisition time elapses after the detection of the movement of the singer is started. , Continue detection. Thereby, based on the operation | movement of a singer during the minimum acquisition time at the shortest, it is possible to determine the repetitive operation.

  Also, for example, if the repeated action to be performed by the audience character is not determined before the maximum acquisition time elapses after the detection of the singer's action, the control unit 21 determines the singer's action at this time. The detection may be terminated. That is, the control part 21 produces | generates the input motion data of the motion detected until the maximum acquisition time passes after the detection of a singer's motion is started. Accordingly, it is possible to prevent a situation in which a spectator character that performs a repetitive motion is not displayed indefinitely even though the singer performs a motion. In the present embodiment, the maximum acquisition time is common for all performance parts constituting the karaoke music piece. However, the maximum acquisition time may be, for example, the time from the start of detection of the singer's motion until the end of the performance part output by the speaker 4 when the detection is started.

  When detecting the operation of the singer, the control unit 21 determines the designated operation data based on the input operation data indicating the detected operation. The designated action data is action data indicating a designated action that is a repeated action to be performed by the spectator character. For example, the control unit 21 may determine the input operation data itself as designated operation data.

  In addition, for example, the control unit 21 may specify an operation repeatedly performed by the singer from the detected operations based on the input operation data. The operation performed repeatedly by the singer is called input repetition operation. And the control part 21 may determine designation | designated operation | movement data based on an input repetition operation | movement. When the singer repeats the same action, the same posture appears periodically. FIG. 3B is a diagram schematically illustrating an example of a state in which a singer shakes his hand. As shown in FIG. 3B, at a certain time T1, the singer starts moving his hand with his left hand, for example. The singer moves the left hand once in the left direction and moves it in the right direction. Then, at time T2, the position of the left hand of the singer returns to the time when the movement of waving is started. When the singer repeats such an operation, the operation of waving is repeated. Control part 21 may specify time T2 when a singer took the same posture after movement of a body part more than a predetermined momentum from time T1 when a singer took a certain posture, for example. For example, when the difference between the coordinates of each body part at time T1 and the coordinates of each body part at time T2 is less than a predetermined value, the control unit 21 determines the posture of the singer at time T1 and the singer's attitude at time T2. It may be determined that the posture is the same. Next, the control unit 21 compares the operation between times T1 and T2 with the operation between times T2 and T3. For example, the length from time T2 to time T3 may be the same as the length from time T1 to time T2. When the operation between the times T1 and T2 and the operation between the times T2 and T3 are the same, the control unit 21 may determine that the singer has repeatedly performed the operation. For example, in the input motion data, the control unit 21 calculates the time series of the difference between the coordinates of each body part during the operation between times T1 and T2 and the coordinates of each body part during the operation between times T2 and T3. You may calculate along. And the control part 21 may determine with it being the same operation | movement, when the average value of the calculated difference is below a predetermined value. When it is determined that the singer has performed a repetitive operation, the control unit 21 may extract, for example, an operation between times T1 and T2 as an input repetitive operation. For example, the control unit 21 extracts the coordinates of the body part between times T1 and T2 among the coordinates of the body part included in the input action data as action data indicating the input repeated action. This operation data is referred to as input repetition operation data. When the same operation cannot be specified from the input operations, the control unit 21 does not have to determine the designated operation data. Moreover, the control part 21 may extract an input repetition operation | movement from input operation data, for example with an autocorrelation function.

  Further, for example, the control unit 21 may determine the designated operation data from a plurality of predetermined operation data based on the input operation or the input repetition operation. Thereby, even if a singer performs an undesired operation | movement, the audience character which performs a desirable repeating operation | movement can be displayed. For example, the control unit 21 may determine default repeated motion data similar to the input motion data or the input repeated motion data as the designated motion data. For example, the control unit 21 calculates the similarity between the input motion data or the input repeated motion data and each predetermined repeated motion data stored in the storage unit 22. For example, the control unit 21 calculates the difference between the coordinates of the body part included in the input motion data or the input repeated motion data and the coordinates of the body part included in the default repeated motion data along a time series. And the control part 21 may calculate a similarity based on the average value of the calculated difference. The control unit 21 may determine, as the operation data of the designated operation, the predetermined repetition operation data for which the maximum similarity is calculated among the predetermined repetition operation data whose calculated similarity is equal to or greater than a predetermined value. When there is no predetermined repetitive motion data whose similarity is equal to or greater than a predetermined value, the control unit 21 may determine, for example, input motion data or input repetitive motion data as designated motion data.

  The control unit 21 may adjust the speed at which the designated operation indicated by the determined designated operation data is performed based on the cycle of the designated operation and the tempo of the karaoke song being output from the speaker 4. For example, the control unit 21 may adjust the cycle of the repetitive operation so that the repetitive operation is performed at a cycle that is a natural number times the reciprocal of the tempo of the music. For example, the control unit 21 may expand and contract the designated operation data so that the cycle of the repetitive operation matches a cycle that is a natural number times the reciprocal of the tempo of the music. Thereby, it is possible to display a spectator character that performs a repeated operation in accordance with the tempo of the music output from the speaker 4. When shortening the length of the designated motion data, for example, the control unit 21 determines the body at a time interval corresponding to the ratio between the original cycle of the designated motion and the cycle that is a natural number times the reciprocal of the tempo of the music. The coordinates of the part may be deleted. In this case, the time for one designated operation is shortened, and the speed at which the designated operation is performed is increased. When increasing the length of the designated motion data, for example, the control unit 21 adds the motion data to the body at a time interval corresponding to the ratio between the original cycle of the designated motion and a cycle that is a natural number times the inverse of the tempo of the music. Add the coordinates of the part. The coordinates of the body part to be added may be, for example, an average value of coordinates before and after the position where the coordinates are added. In this case, the time for one designated operation becomes longer, and the speed at which the designated operation is performed becomes slower.

  When the designated motion data is determined, the control unit 21 causes the display 5 to display an image indicating the audience character that repeatedly performs the designated motion based on the determined designated motion data.

  For example, the control unit 21 may assign the determined designated operation data to any one of a plurality of performance parts constituting the music being output from the speaker 4. And the control part 21 may display the image | video which shows the audience character which repeats designation | designated operation | movement, when the assigned performance part is output from the speaker 4. FIG. Accordingly, it is possible to display a spectator character that repeatedly performs the designated operation in units of performance parts. For example, the control unit 21 may store the determined designated action data in the RAM in association with the part type of the performance part that was output when the input action for which the designated action data was determined was detected. Then, when the performance part of the stored part type is output, the control unit 21 displays a video showing a spectator character that repeatedly performs the specified action indicated by the specified action data stored in the RAM in association with the part type. It may be displayed. Thereby, when the performance part of the same kind as the performance part which the singer performed the detected operation | movement is output, the audience character which performs a repetition operation | movement can be displayed.

  Further, for example, the control unit 21 may display a spectator character that repeatedly performs the designated operation that is determined regardless of the performance part. For example, the control unit 21 may start displaying a spectator character performing a new designated action at each timing when new designated action data can be determined.

[4. Operation of Karaoke System S]
Next, the operation of the karaoke system S will be described. FIG. 4 is a diagram illustrating an example of a data flow of operation data. As shown in FIG. 4, the control part 21 produces | generates the input operation | movement data which show the operation | movement of the detected singer (step S1). Subsequently, the control part 21 produces | generates the input repetition operation | movement data which show the operation | movement repeatedly performed in operation | movement of a singer based on input operation data (step S2). Next, the control unit 21 searches the storage unit 22 for predetermined repetitive motion data similar to the input repetitive motion data (step S3). Then, the control unit 21 sets the retrieved default repeated motion data as unmodified repeated motion data (step S4). On the other hand, when there is no default repeated motion data similar to the input repeated motion data, the control unit 21 sets the input repeated motion data as unmodified repeated motion data. The control unit 21 generates corrected repeated motion data by adjusting the expansion / contraction of the uncorrected repeated motion data based on the tempo of the karaoke piece to be played back (step S5). The corrected repetitive motion data is motion data indicating a repetitive motion adjusted for expansion and contraction. Next, the control unit 21 sets the corrected repeated motion data as designated motion data (step S6). The designated operation data is operation data indicating the determined repetitive operation. And the control part 21 sets designation | designated operation | movement data to audience operation | movement data (step S7). Audience action data is action data indicating actions to be performed by the spectator character. When the designated motion data is not determined, normal motion data is set as the spectator motion data.

  FIG. 5 is a flowchart showing an example of the performance drawing main process in the karaoke apparatus 2. The performance drawing main process is started when the karaoke apparatus 2 is activated. As shown in FIG. 5, the control part 21 acquires the information of the karaoke music reproduced | regenerated (step S11). Specifically, the control unit 21 acquires the song ID of the karaoke song selected using the song selection device 1 from the song selection device 1. Next, the control unit 21 acquires music data and music configuration information corresponding to the acquired music ID from the storage unit 22. Next, the control unit 21 acquires the audience motion designation part type corresponding to the acquired music ID from the storage unit 22 (step S12). Next, the control unit 21 sets the performance number N to 1. Next, the control unit 21 sets the normal operation data stored in the storage unit 22 in the audience operation data (step S13).

  Subsequently, the control part 21 starts reproduction | regeneration of karaoke music based on music data (step S14). At this time, the control unit 21 reproduces the karaoke music according to the tempo of the karaoke music included in the music data. Thereby, the karaoke music is output from the speaker 4. Further, the control unit 21 starts to capture a distance image by the distance image camera 3 and receives the distance image from the distance image camera 3. And the control part 21 starts the memory | storage to RAM of the relative momentum based on the coordinate of each body part of a singer based on a distance image, and the coordinate of a body part.

  Next, the control unit 21 starts the display of the video of the spectator character performing the motion indicated by the spectator motion data based on the current spectator motion data by executing the 3D engine (step S15). The control unit 21 sequentially generates still images including the video of the spectator character while acquiring the coordinates of each body part from the spectator motion data according to the frame rate, and sequentially outputs the generated still images to the video RAM 23. Subsequently, the control part 21 performs audience operation setting processing (step S16). The audience motion setting process is a process for setting audience motion data. Details of the audience operation setting will be described later. Next, the control unit 21 determines whether or not the reproduction of the karaoke music is finished (step S17). At this time, if the control unit 21 determines that the reproduction of the karaoke song is not finished (step S17: NO), the control unit 21 proceeds to step S16. On the other hand, when the control unit 21 determines that the reproduction of the karaoke music piece is finished (step S17: YES), the control unit 21 proceeds to step S18.

  In step S18, the control unit 21 stops the display of the video of the spectator character performing the motion indicated by the spectator motion data by stopping the execution of the 3D engine. Subsequently, the control part 21 determines whether the karaoke apparatus 2 stops (step S19). At this time, when the control unit 21 determines that the karaoke apparatus 2 does not stop (step S19: NO), the control unit 21 proceeds to step S11. On the other hand, when it determines with the karaoke apparatus 2 stopping (step S19: YES), the control part 21 complete | finishes a performance drawing main process.

  FIG. 6A is a flowchart showing a first example of audience operation setting processing in the karaoke apparatus 2. In the processing example shown in FIG. 6A, the input operation execution request information is displayed during the reproduction of the performance part of the audience action designation part type, and the operation of the singer during the reproduction of the performance part of the audience action designation part type is detected. This is an example of processing when In addition, the processing example shown in FIG. 6A displays the video performed by the spectator character when playing the performance part of the spectator motion designation part type with the repetitive motion determined based on the detected motion. This is an example of processing when

  As shown in FIG. 6A, the control unit 21 determines whether or not a performance part change has been detected by outputting a performance part change event from the music sequencer (step S21). At this time, when it is determined that the performance part switching is not detected (step S21: NO), the control unit 21 ends the audience motion setting process. On the other hand, if the control unit 21 determines that the switching of the performance part has been detected (step S21: YES), the control unit 21 proceeds to step S22.

  In step S22, the control unit 21 adds 1 to the performance number N. Next, the control unit 21 acquires the part type corresponding to the performance number N from the music composition information. Next, the control unit 21 determines whether or not the designated operation data corresponding to the acquired part type is already stored in the storage unit 22. At this time, if the control unit 21 determines that the designated operation data is stored in the storage unit 22 (step S22: YES), the control unit 21 proceeds to step S23. On the other hand, when it is determined that the designated operation data is not stored in the storage unit 22 (step S22: NO), the control unit 21 proceeds to step S24.

  In step S23, the control unit 21 sets designated motion data corresponding to the acquired part type in the audience motion data. Next, the control unit 21 ends the audience operation setting process. The control unit 21 that is executing the 3D engine starts displaying the video of the spectator character that performs the repetitive motion based on the newly set spectator motion data.

  In step S24, the control unit 21 sets normal operation data in the audience operation data. Next, the control unit 21 determines whether or not the part type corresponding to the performance number N matches any of the audience motion designation part types (step S25). At this time, if the control unit 21 determines that the part type corresponding to the performance number N does not match any of the audience motion designation part types (step S25: NO), the control unit 21 ends the audience motion setting process. In this case, the control unit 21 that is executing the 3D engine starts displaying the video of the spectator character performing the motion indicated by the normal motion data set as the spectator motion data. On the other hand, if the control unit 21 determines that the part type corresponding to the performance number N matches any one of the audience motion designation part types (step S25: YES), the control unit 21 proceeds to step S26.

  In step S <b> 26, the control unit 21 displays input operation execution request information on the display 5. Next, the control unit 21 executes a designated action generation process (step S27). The specified action generation process is a process for generating specified action data. Details of the specified action generation process will be described later. Next, the control unit 21 determines whether or not the generation of the designated motion data has succeeded in the designated motion generation process (step S28). At this time, if the control unit 21 determines that the generation of the designated operation data has failed (step S28: NO), the control unit 21 proceeds to step S30. In this case, the control unit 21 that is executing the 3D engine starts displaying the video of the spectator character performing the motion indicated by the normal motion data set as the spectator motion data. On the other hand, if the control unit 21 determines that the generation of the designated operation data has been successful (step S28: YES), the control unit 21 proceeds to step S29.

  In step S29, the control unit 21 sets the designated motion data generated by the designated motion generation process and stored in the RAM in the audience motion data, and the process proceeds to step S30. The control unit 21 that is executing the 3D engine starts displaying the video of the spectator character that performs the repetitive motion based on the newly set spectator motion data.

  In step S30, the control unit 21 displays information indicating the success or failure of the generation of the designated motion data by the designated motion generation process on the display 5, and ends the audience motion setting process.

  FIG. 6B is a flowchart showing a second example of the audience operation setting process in the karaoke apparatus 2. The example shown in FIG. 6B is an example in the case where the detection of the singer's motion is started when the most recent momentum is equal to or greater than the detection determination momentum.

  As shown in FIG. 6B, the control unit 21 calculates the sum of the relative momentum stored in the RAM during the latest detection determination time as the latest momentum. And the control part 21 determines whether the calculated exercise amount is more than a detection determination exercise amount (step S31). At this time, when it is determined that the calculated exercise amount is not equal to or greater than the detection determination exercise amount (step S31: NO), the control unit 21 ends the audience operation setting process. On the other hand, when it is determined that the calculated exercise amount is equal to or greater than the detection determination exercise amount (step S31: YES), the control unit 21 proceeds to step S32.

  In step S32, the control unit 21 executes a designated action generation process. Next, the control unit 21 determines whether or not the generation of designated motion data has succeeded in the designated motion generation process (step S33). At this time, when it is determined that the generation of the designated motion data has failed (step S33: NO), the control unit 21 ends the audience motion setting process. On the other hand, if the control unit 21 determines that the generation of the designated operation data has been successful (step S33: YES), the control unit 21 proceeds to step S34.

  In step S34, the control unit 21 sets the designated motion data generated by the designated motion generation process and stored in the RAM in the audience motion data, and ends the audience motion setting process. The control unit 21 that is executing the 3D engine starts displaying the video of the spectator character that performs the repetitive motion based on the newly set spectator motion data.

  FIG. 7A is a flowchart illustrating an example of the designated action generation process in the karaoke apparatus 2. As shown in FIG. 7A, the control unit 21 executes an input repetition operation generation process (step S41). The input repetition operation generation process is a process for generating input repetition operation data. Details of the input repetition operation generation processing will be described later. Next, the control unit 21 determines whether or not the generation of the input repetition operation data in the input repetition operation generation process has succeeded (step S42). At this time, if the control unit 21 determines that the generation of the input repetition operation data has failed (step S42: NO), the control unit 21 proceeds to step S50. In step S50, the control unit 21 determines that the generation of the designated operation data has failed, and ends the designated operation generation process. On the other hand, when the control unit 21 determines that the generation of the input repetitive motion data has been successful (step S42: YES), the control unit 21 proceeds to step S43.

  In step S43, the control unit 21 executes a default repeated operation acquisition process. The default repeated motion acquisition process is a process for acquiring default repeated motion data similar to the generated input repeated motion data. Details of the default repetition operation acquisition process will be described later. Next, the control unit 21 determines whether or not there is default repeated motion data similar to the input repeated motion data as a result of the default repeated motion acquisition process (step S44). At this time, if the control unit 21 determines that similar default repetitive motion data exists (step S44: YES), the control unit 21 proceeds to step S45. In step S45, the control unit 21 sets the obtained default repeated motion data in the unmodified repeated motion data, and proceeds to step S47. On the other hand, when it is determined that there is no similar default repetitive motion data (step S44: NO), the control unit 21 proceeds to step S46. In step S46, the control unit 21 sets the generated input repeated motion data in the unmodified repeated motion data, and proceeds to step S47.

  In step S47, the control unit 21 acquires the tempo from the music data of the karaoke music being reproduced. Next, the control unit 21 generates corrected repeated motion data by expanding and contracting the uncorrected repeated motion data so that the repeated motion is performed at a cycle that is a natural number times the reciprocal of the acquired tempo. Next, the control unit 21 sets the generated corrected repeated motion data in the designated motion data (step S48). Next, the control unit 21 stores the designated operation data in the storage unit 22 in association with the part type corresponding to the performance number N (step S49), and ends the designated operation generation process.

  FIG. 7B is a flowchart illustrating an example of an input repetition operation generation process in the karaoke apparatus 2. As shown in FIG. 7B, the control unit 21 starts detecting an input operation (step S51). In detecting the input operation, the control unit 21 detects the coordinates of each body part of the singer and stores the coordinates of each body part in the RAM every time a distance image is received from the distance image camera 3.

  Next, the control unit 21 determines whether or not the maximum acquisition time has elapsed since the detection of the input operation was started (step S52). At this time, when it is determined that the maximum acquisition time has elapsed (step S52: YES), the control unit 21 proceeds to step S55. On the other hand, when it is determined that the maximum acquisition time has not elapsed (step S52: NO), the control unit 21 proceeds to step S53.

  In step S53, the control unit 21 determines whether or not the minimum acquisition time has elapsed since the detection of the input operation was started. At this time, if the control unit 21 determines that the minimum acquisition time has elapsed (step S53: YES), the control unit 21 proceeds to step S54. On the other hand, when it is determined that the minimum acquisition time has not elapsed (step S53: NO), the control unit 21 proceeds to step S52.

  In step S54, the control unit 21 calculates the sum of the relative momentum stored in the RAM during the latest detection determination time as the latest momentum. And the control part 21 determines whether the calculated exercise amount is below a detection determination exercise amount. At this time, when it is determined that the calculated exercise amount is not equal to or less than the detection determination exercise amount (step S54: NO), the control unit 21 proceeds to step S52. On the other hand, when it is determined that the calculated exercise amount is equal to or less than the detection determination exercise amount (step S54: YES), the control unit 21 proceeds to step S55.

  In step S55, the control unit 21 ends the detection of the input operation. And the control part 21 produces | generates the input motion data containing the coordinate of the body part memorize | stored in RAM after the detection of input motion was started. Next, the control unit 21 removes the coordinates of the body part during the period in which the motion is not performed from the input motion data (step S56). For example, the control unit 21 removes the coordinates of the body part whose relative momentum is continuously below a predetermined value from the beginning of the input motion data, and the relative momentum is continuously below the predetermined value from the end of the input motion data. The body part coordinates may be removed. Next, the control unit 21 extracts the coordinates of the repeated motion part from the input motion data (step S57). When it is possible to extract the coordinates of the repetitively performed motion part, the control unit 21 generates input repetitive motion data including the extracted coordinates. When it is not possible to extract the coordinates of the repeated motion part, the control unit 21 determines that the generation of the repeated input motion data has failed. When step S57 is completed, the control unit 21 ends the input repetition operation generation process.

  FIG. 7C is a diagram illustrating an example of a default repeated motion acquisition process in the karaoke apparatus 2. As shown in FIG. 7C, the control unit 21 acquires the tempo of the default repetition operation (step S61). Next, the control unit 21 changes the length of the input repetition operation data in accordance with the tempo of the default repetition operation (step S62). For example, the control unit 21 adjusts the input repeat operation data so that the length of time during which the input repeat operation is performed matches the reciprocal of the tempo of the default repeat operation. Next, the control unit 21 retrieves and acquires predetermined repetitive motion data similar to the input repetitive motion data whose length has been changed from the storage unit 22 (step S63). Then, the control unit 21 ends the default repeated action acquisition process.

  As described above, according to the present embodiment, the karaoke device 2 detects the operation of the singer during the output of music by the speaker 4. Further, the karaoke apparatus 2 determines operation data indicating a repetitive operation based on the detected operation. Further, the karaoke device 2 displays an image showing the audience character performing the repetitive motion indicated by the determined motion data during the output of the music by the speaker 4 on the display 5. Therefore, the video of the spectator character performing the repetitive motion is displayed in response to the singer performing the motion. Therefore, when the singer sees the audience character as the audience, it is possible to enhance the sense of presence as if singing in front of the audience.

DESCRIPTION OF SYMBOLS 1 Music selection apparatus 2 Karaoke apparatus 3 Distance image camera 4 Speaker 5 Display S Karaoke system

Claims (12)

Obtaining means for obtaining music data indicating the music;
Output control means for causing the output means to output the music indicated by the music data acquired by the acquisition means;
Detecting means for detecting a user's action during output of the music by the output means;
A determination unit that determines operation information indicating a repetitive operation based on the operation detected by the detection unit;
Display control means for displaying on the display means a video indicating an object that performs the repetitive motion indicated by the motion information determined by the determination means during the output of the music by the output means;
An information processing apparatus comprising: The information processing apparatus according to claim 1,
The determining means is based on the user's action detected between the start of detection of the user's action by the detecting means and the detection of the user's action below a predetermined amount of exercise by the detecting means. An information processing apparatus that determines the operation information. The information processing apparatus according to claim 2,
Based on the user's action detected by the detecting means until the user's action equal to or less than the predetermined exercise amount is detected after a first predetermined time has elapsed since the detection means started detecting the user's action. An information processing apparatus that determines the operation information. The information processing apparatus according to any one of claims 1 to 3,
Further comprising a specifying means for specifying an action repeatedly performed by the user from the actions detected by the detecting means;
The information processing apparatus, wherein the determining unit determines the motion information indicating a repetitive motion performed by the object based on the motion specified by the specifying unit. The information processing apparatus according to any one of claims 1 to 4,
Further comprising predetermined operation information storage means for storing a plurality of predetermined operation information indicating predetermined repetitive operations;
The determining means determines any predetermined operation information among a plurality of predetermined operation information stored in the predetermined operation information storage means based on the operation detected by the detection means,
The information processing apparatus, wherein the display control unit causes the display unit to display a video indicating the object performing the operation indicated by the predetermined motion information determined by the determination unit. The information processing apparatus according to claim 5,
The determining means determines predetermined action information indicating an action similar to the action detected by the detecting means from the plurality of predetermined action information stored in the predetermined action information storage means, and is detected by the detecting means. If the predetermined motion information indicating the motion similar to the motion performed is not stored in the predetermined motion information storage means, the motion indicating the motion detected by the detection means is the motion indicating the repeated motion performed by the object An information processing apparatus characterized by determining information. The information processing apparatus according to any one of claims 1 to 6,
The acquisition means acquires the music data indicating a music composed of a plurality of performance parts,
An allocation unit that allocates the motion information determined by the determination unit to any one of the plurality of performance parts;
The display control means displays, on the display means, a video showing the object performing the repetitive action indicated by the action information when the performance part assigned with the action information by the assigning means is output by the output means. An information processing apparatus characterized in that The information processing apparatus according to claim 7,
The acquisition means acquires the music data indicating a music composed of a plurality of performance parts including two or more performance parts of the same type.
The allocating means was output by the output means when the operation information determined by the determining means and the user action for which the operation information was determined by the determining means were detected by the detecting means The type information indicating the type of the performance part is associated with the type information and stored in the type information storage means.
When the performance part of the type indicated by the type information stored in the type information storage unit is output by the output unit, the display control unit is stored in the type information storage unit in association with the type information. An information processing apparatus that causes the display unit to display a video indicating the object that performs the repetitive motion indicated by the motion information. The information processing apparatus according to any one of claims 1 to 8,
A second display control means for causing the display means to display request information indicating an operation execution request while the music is being output by the output means;
The information processing apparatus according to claim 1, wherein the determination unit determines the operation information based on an operation detected by the detection unit after the request information is displayed on the display unit. The information processing apparatus according to any one of claims 1 to 9,
The determination unit starts detecting the user's operation when the repetitive operation is not determined before the second predetermined time elapses after the detection unit starts detecting the user's operation. The operation information is determined based on the operation of the user detected by the detection means until the second predetermined time elapses. The information processing apparatus according to any one of claims 1 to 10,
Adjustment for adjusting the speed at which the repetitive motion indicated by the motion information determined by the determining means is performed based on the cycle of the repetitive motion and the tempo of the music indicated by the music data acquired by the acquiring means Further comprising means,
The display control means causes the display means to display an image showing the object for performing the repetitive motion indicated by the motion information determined by the determination means at a speed adjusted by the adjustment means. apparatus. An acquisition step of acquiring music data indicating the music;
An output control step of causing the output means to output the music indicated by the music data acquired by the acquisition step;
A detection step of detecting a user's action during the output of the music by the output means;
A determination step for determining operation information indicating a repetitive operation based on the operation detected by the detection step;
A display control step of causing the display means to display a video indicating an object that performs the repetitive motion indicated by the motion information determined by the determination step during the output of the music by the output means;
A program that causes a computer to execute.

Images