JP2015185020A - Event method, concert throwing method, event venue and content trading method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an event method, concert throwing method, event venue and content trading method that can give to fans an impression as if artists sign songs or play music although the artists do not appear before the fans.SOLUTION: An event method comprises: a body motion information acquisition step S10 of acquiring body motion information about a body motion; a voice information acquisition step S11 of acquiring voice information about a voice; a motion data generation step S12 of generating motion data on the basis of the body motion information; a voice data generation step S13 of generating voice data on the basis of the voice information; a virtual video image data generation step S14 of generating virtual video image data associated with the motion data and the voice data; and a virtual video image display control step S15 of projecting a virtual video image on a display part on the basis of the virtual video image data.

Description

  The present invention relates to an event method by an artist or the like, a concert holding method, an event venue, and a content buying and selling method.

  Traditionally, concerts by various artists have been held at concert venues throughout the country.

  At the concert, various services are provided to the fans as artists sing and play live. For this reason, the concert venue has a fun time as if it were another world, and fans can have a good time.

JP 2001-8186 A JP 2011-155942 A

  By the way, conventional concerts are based on the premise that artists will sing and perform on the spot. Some artists are hiding their faces for various reasons. For such an artist, when a concert is held, the fans may be faced, leading to problems such as a loss of charisma and a decrease in future box office revenue.

  Therefore, in view of the above problems, the present invention provides an event method that can give an impression to a fan as if the artist is singing or playing on the spot without the artist appearing in front of the fan. The purpose is to provide a concert holding method, an event venue, and a content buying and selling method.

  The first invention is an event method using a virtual image, wherein a body motion information acquisition step for acquiring body motion information related to body motion by the body motion information acquisition means, and voice information related to voice by the voice information acquisition means. An audio information acquisition step for acquiring, a motion data generation step for generating motion data based on the body motion information by a motion data generation means, and an audio data generation for generating audio data based on the audio information by an audio data generation means A virtual video data generating step for generating virtual video data in which the motion data and the audio data are associated with each other by a virtual video data generating unit; and a virtual video data based on the virtual video data by a virtual video display control unit. It characterized by having a a virtual image display control step for projecting on a display unit.

  In this case, it has a voice output step of outputting a voice based on the voice data by a voice output means.

  In this case, the physical motion information is information regarding a motion when singing a song, and the motion data is data regarding a motion when singing a song.

  In this case, the voice information is information relating to a singing voice, and the voice data is data relating to a singing voice.

  In this case, the virtual video data is data relating to a silhouette of a body that moves in accordance with voice.

  In this case, in the virtual image display control step, the virtual image is displayed in the vicinity of the artist.

  In this case, the virtual image is a two-dimensional image.

  The second invention is a concert holding method in which a concert is performed by the event method using the virtual image of the present invention.

  The third invention is an event venue that implements the event method using the virtual video of the present invention.

  In this case, the event venue has a spectator seat.

  According to a fourth aspect of the present invention, at least a part of virtual video or virtual video data acquired by the event method using virtual video according to the present invention is sold as content or an article to which the content is attached. It is a buying and selling method.

  The “content” of the present invention is defined to mean the contents of music or other information. The content includes a design, a photograph, a picture, etc. in addition to a moving image / still image.

  The “article” of the present invention includes daily necessities, plants, animals, food, luxury goods, vehicles and the like.

  According to the present invention, an event can be progressed by displaying on the display unit a virtual image that closely resembles the physical movement and sound of an actual artist. This gives the audience a fresh impression as if the artist is on the spot, and can excite the event.

  For artists, even if they are not at the event venue, by generating their own virtual video data in advance, it is possible to hold an event in which their alternations behave.

  In addition, since there is an audio output step of outputting audio based on audio data by the audio output means, optimal audio is output at a timing that matches the movement of the virtual video, thus eliminating the unnaturalness peculiar to virtual. Can improve the completeness of the event.

  The body motion information is information related to the motion when singing a song, and the motion data is data related to the motion when singing a song, so that the motion of the artist can be accurately expressed by a virtual image.

  Further, since the audio information is information related to the singing voice and the audio data is data related to the singing voice, the artist's singing voice can be accurately expressed by the virtual video.

  In addition, since the virtual video data is data relating to the silhouette of the body that moves according to the sound, it can look like an effect by the artist himself, making the audience more excited and making the event successful.

  Also, in the virtual video display control step, a virtual video can be displayed in the vicinity of the artist, thereby realizing a performance in which the artist himself and his alternation are advanced side by side. As a result, the audience can be stimulated and the event can be excited.

  In addition, since the virtual video is a two-dimensional video, the physical characteristics of the artist main body can be captured in a plane. Thereby, it can prevent that it becomes difficult to see a virtual image | video by the position where a spectator exists.

  In addition, by performing a concert by an event method using virtual images, it is possible to excite the artist in the same way as a concert in which the artist himself appears, even if the artist is not at the concert venue.

  In addition, it is possible to provide an event venue that can display a virtual video that closely resembles the movement and sound of the artist.

  In addition, a large event can be held by accommodating many spectators by providing spectator seats at the event venue.

  Moreover, novel content can be sold to fans by selling at least part of virtual video or virtual video data acquired by an event method using virtual video as content or design. This will delight fans and revitalize the music industry.

It is a block diagram of the event system of 1st Embodiment of this invention. It is a block diagram of the memory | storage part of the event system of 1st Embodiment of this invention. It is a figure which shows each step of the event method of 1st Embodiment of this invention. It is the figure which showed an example of the virtual image | video by the event method of 1st Embodiment of this invention. It is the figure which showed the state where the artist stood in the vicinity of the virtual image by the event method of 1st Embodiment of this invention, and sings a song. It is a top view of the concert hall of 1st Embodiment of this invention. It is a block diagram which shows the outline of a structure of the motion production apparatus in 1st Example of 1st Embodiment of this invention. It is explanatory drawing which shows an example of a human body model in 1st Example of 1st Embodiment of this invention. It is explanatory drawing which shows an example of a motion graph in 1st Example of 1st Embodiment of this invention. It is explanatory drawing which shows an example of a mode that the key frame component MCkeyframe is set in 1st Example of 1st Embodiment of this invention. It is explanatory drawing which shows an example of a mode that the motion excitement component MCintensity is set in 1st Example of 1st Embodiment of this invention. It is a flowchart which shows an example of the main routine performed by the motion production apparatus in 1st Example of 1st Embodiment of this invention. It is a flowchart which shows an example of a music data analysis process in 1st Example of 1st Embodiment of this invention. It is explanatory drawing which shows a mode that the spectrum power increment d (t, f) is calculated in 1st Example of 1st Embodiment of this invention. It is explanatory drawing which shows an example of the histogram Pist (t, f) of a frequency component in 1st Example of 1st Embodiment of this invention. It is a flowchart which shows an example of a motion creation process in 1st Example of 1st Embodiment of this invention. It is a block diagram which shows the structure of 2nd Example of 1st Embodiment of this invention. It is explanatory drawing of the state which attached the gyro sensor to the to-be-measured person in 2nd Example of 1st Embodiment of this invention. It is a computer image | video which shows the mutual position of a gyro sensor and frame | skeleton in 2nd Example of 1st Embodiment of this invention. It is the figure which showed an example of the business model which buys and sells the content of the event of 1st Embodiment of this invention.

  An event method, concert holding method, and event venue using a virtual image according to the first embodiment of the present invention will be described with reference to the drawings.

  An event system for realizing an event method using virtual video will be described.

  As shown in FIGS. 1 and 2, the event system 10 mainly includes a main control unit 10A, a storage unit 10B, a body motion information input unit 10C, a voice information input unit 10D, and a motion data generation unit 10E. The audio data generation unit 10F, the virtual video data generation unit 10G, the virtual video display control unit 10H, and the audio output unit 10I are included.

  The main control unit 10A comprehensively controls each component of the event system 10.

  The main control unit 10A can use a CPU, for example.

  The storage unit 10B stores various data and programs.

  The storage unit 10B can use, for example, a ROM, a RAM, or the like.

  The body motion information input unit 10C records the body motion and acquires this as the body motion information DB1. For example, the body motion information DB 1 is acquired based on a record of body motion such as the posture of the artist, acting (comte), and choreography when singing a song.

  The body motion information DB1 is stored in the storage unit 10B.

  As the body motion information input unit 10C, for example, various sensors for measuring the motion of the body can be used.

  The voice information input unit 10D records the voice of the target person and acquires it as the voice information DB2. For example, the voice information DB 2 based on the recording of voices such as artist singing voices and spoken words is acquired.

  The voice information DB2 is stored in the storage unit 10B.

  For example, a microphone can be used as the audio information input unit 10D.

  The motion data generation unit 10E generates the motion data DB3 based on the body motion information DB1. The motion data DB 3 is data that can reproduce physical movements such as the posture and performance of an artist, for example.

  The motion data DB3 is stored in the storage unit 10B.

  As the motion data generation unit 10E, for example, a CPU that generates the motion data DB3 based on a dedicated motion data generation program (software) PG1 can be used.

  The motion data generation program PG1 is stored in the storage unit 10B.

  The voice data generation unit 10F generates the voice data DB 4 based on the voice information. The audio data DB 4 is data that can reproduce, for example, an artist's singing voice and spoken language.

  Note that the audio data DB4 is stored in the storage unit 10B.

  As the audio data generation unit 10F, for example, a CPU that generates the audio data DB4 based on a dedicated audio data generation program (software) PG2 can be used.

  The audio data generation program PG2 is stored in the storage unit 10B.

  The virtual video data generation unit 10G generates a virtual video data DB5 in which the motion data DB3 and the audio data DB4 are associated with each other. The virtual video data DB 5 may be data relating to a character such as a predetermined avatar or data relating to an artist's body silhouette, for example.

  An example of the virtual video data DB 5 is video data in which an artist sings or dances. At this time, the virtual video data DB 5 is synthesized data formed by harmonizing motion data and audio data or synchronizing their timings, and can simultaneously express the physical movement and audio of the artist with a natural feeling. .

  As an example of the virtual video data DB 5, for example, 2D video data or 3D video data with motion and sound is employed. The virtual video data DB 5 is data that allows, for example, a character or an artist's silhouette to move in the same manner as an actual artist's motion, and to sing a song with the same singing voice as the actual artist's singing voice. .

  A well-known technique can be used as a method (editing method) for harmonizing the motion data DB 3 and the audio data DB 4 or synchronizing their timings. Details will be described in Examples.

  The virtual video data DB5 is stored in the storage unit 10B.

  As the virtual video data generation unit 10G, for example, a CPU that generates the virtual video data DB 5 based on a dedicated virtual video data generation program (software) PG3 can be used.

  The virtual video data generation program PG3 is stored in the storage unit 10B.

  Further, as the virtual video data generation unit 10G, a method may be used in which dedicated virtual video data generation software is used to manually synchronize the motion data DB3 and the audio data DB4 with a predetermined timing.

  For example, the virtual video display control unit 10H projects the virtual video 14 (see FIG. 4) based on the virtual video data DB 5 on the display unit 12 (see FIG. 4). The virtual video display control unit 10H has a function of controlling the display content of the virtual video.

  The virtual video 14 is, for example, a video with movements of the artist's movement and sound based on the virtual video data DB 5, and gives the audience the impression that the artist is singing a song on the display unit 12. Can be given.

  The virtual video 14 may be, for example, a video related to a predetermined character or a video related to an artist's silhouette.

  For the virtual video 14, for example, a two-dimensional image projected in two dimensions is adopted, but a three-dimensional image projected in three dimensions may be created.

  For the virtual video display control unit 10H, for example, an electric / electronic device capable of reproducing video, such as a projector or a projector, can be used.

  The display unit 12 may be a screen for displaying the virtual video 14, a liquid crystal display, or the like.

  The display unit 12 is preferably a display or a screen capable of projecting a two-dimensional video or a three-dimensional video, for example.

  The audio output unit 10I outputs audio based on the audio data. The audio output from the audio output unit 10I is associated with the motion data by the virtual video data, and is output corresponding to the operation of the virtual video. For this reason, there is no deviation between the operation timing of the virtual video and the audio output timing, or even a slight deviation is corrected by the virtual video data generation unit 10G or the main control unit 10A. There is no sense of incongruity when listening to

  For example, a speaker can be used as the audio output unit 10I.

  Next, an event method using the virtual video 14 using the event system 10 will be described.

  As shown in FIG. 3, the event method using virtual video mainly includes body motion information acquisition step S10, audio information acquisition step S11, motion data generation step S12, music data generation step S13, and virtual video data. It has generation step S14, virtual video display control step S15, and audio output step S16.

  In the body motion information acquisition step S10, body motion information DB1 related to body motion is acquired by the body motion information input unit 10C. For example, the body motion information DB 1 corresponds to information based on a record of body motion such as an artist's posture, acting (comte), choreography when singing a song.

  In the voice information acquisition step S11, the voice information DB 2 related to the voice when the spoken word / song is sung is acquired by the voice information input unit 10D. For example, the audio information DB 2 corresponds to information based on audio recording such as an artist's singing voice and spoken language.

  In the motion data generation step S12, the motion data generating unit 10E generates the motion data DB3 based on the body motion information DB1. The motion data DB 3 is data that can reproduce physical movements such as the posture and performance of an artist, for example.

  In the voice data generation step S13, the voice data DB 4 is generated by the voice data generation unit 10F based on the voice information DB2. The audio data DB 4 is data that can reproduce, for example, an artist's singing voice and spoken language.

  In virtual video data generation step S14, virtual video data DB5 is generated based on motion data DB3 and audio data DB4 by virtual video data generation unit 10G. For example, the virtual video data DB 5 corresponds to virtual video data related to a silhouette of a body in which an artist operates in accordance with sound.

  Here, in order to match the timing of the motion data DB3 and the audio data DB4, they are harmonized or synchronized by a predetermined method.

  In the virtual video display control step S15, as shown in FIG. 4, the virtual video display control unit 10H projects the virtual video 14 based on the virtual video data DB 5 on the display unit 12. For example, the silhouette or character of an artist who dances while singing a song is displayed on the screen as the display unit 12.

  The audio output step S16 is executed at the same timing as the virtual video display control step S15, and the audio output unit 10I outputs audio based on the audio data DB4. As a result, optimal sound is output at a timing that matches the movement of the virtual video 14, and the degree of completion of events, concerts, and live using the virtual video 14 can be increased.

  The artist may be a single person or a group of multiple persons. In the case of a multi-person group, a silhouette or a character for dancing while a plurality of artists sing independently is displayed on the display unit 12.

  Here, as shown in FIG. 5, the artist himself 16 is made to appear and the virtual video 14 is set to be displayed in the vicinity of the artist himself 16, and the artist himself 16 sings a song at almost the same timing as the virtual video 14. A special performance may be provided for dancing. As a result, the performance can be executed with the artist and his part lined up, creating a novel production and exciting the event.

  The event corresponds to, for example, a concert or live performance by an artist. In this case, an invention of a concert holding method or a live holding method using the virtual image 14 can be created.

  According to the event method using the virtual image 14 or the concert live holding method of the present embodiment, the event can be progressed by displaying the virtual image 14 that closely resembles the physical motion and sound of an actual artist. . This gives the audience a fresh impression as if the artist is on the spot, and can excite the event.

  For artists, even if they are not at the event venue 17 (see FIG. 6), the virtual video data DB5 is generated and stored based on their own motion data DB3 and audio data DB4, so that an event based on their own self Can be held.

  The body motion information DB1 relating to the motion when singing a song is acquired in the body motion information acquisition step S10, and the motion data DB3 is generated based on the body motion information DB1 in the motion data generation step S12. The artist's movement can be expressed accurately.

  In the voice information acquisition step S11, voice information related to the voice related to the singing voice is acquired, and in the voice data generation step S13, the voice data DB 4 is generated based on the voice information, whereby the artist's singing voice is accurately expressed by the virtual video 14. be able to.

  In the virtual video data generation step S14, virtual video data DB5 that operates in accordance with the sound is generated based on the motion data DB3 and the sound data DB4. Then, by generating and displaying the virtual video 14 based on the virtual video data DB 5, it is possible to make it appear as if it is an effect by the artist himself, and the audience is more excited and the event can be successful.

  Here, by holding the concert by an event method using the virtual image 14, the concert can be energized in the same manner as when an artist appears. For artists, even if they are not in the concert hall 16, they can hold concerts based on their own personality.

  The person to be measured on which the motion data DB 3 is based is not limited to an artist. In addition to artists, various people and objects such as human beings, anime characters, deceased people, animals, machines, robots, etc. are targeted. The body motion information DB 1 for these items is preferably recorded in advance, but can be newly generated (reproduced) by a computer or the like.

  Similarly, the person to be measured that is the basis of the audio data DB 4 is not limited to an artist. In addition to artists, various people and objects such as human beings, anime characters, deceased people, animals, machines, robots, etc. are targeted. The audio information DB 2 of these items is preferably acquired in advance, but can be newly generated (reproduced) by a computer or the like. The sound data DB 4 is not limited to a live performance, but may be an artificially created sound or a sound that can be generated in a natural environment.

  Next, an event venue where an event using the virtual video 14 is held will be described.

  As shown in FIG. 6, the event venue 17 includes a display unit 12 that displays the virtual video 14 and a spectator area 20 that has a plurality of spectator seats 18 arranged in the vicinity of the display unit 12.

  In the event hall 17, a stage 22 may be appropriately arranged in the vicinity of the display unit 12. By providing the stage 22, the artist himself / herself may appear on the stage 22 and dance while singing a song, and at the same time, the virtual image 14 may be displayed on the display unit 12 to perform an event (concert or live).

  When performing a concert as an event, the event venue 17 becomes a concert venue. In addition, when performing live as an event, the event venue 17 becomes a live house or a live venue.

  As a result, it is possible to provide an event venue 17 that can display a virtual video 14 that closely resembles the artist's motion and voice. In addition to the display of the virtual video 14, the artist himself / herself can appear to create a novel effect.

  In particular, by providing a plurality of spectator seats 18 in the event venue 17, a large event can be held by accommodating many spectators.

  Next, a business model using the event method using the virtual video 14 will be described.

  Here, the virtual video data DB 5 generated by the event method and event system 10 of the first embodiment and the virtual video 14 displayed on the display unit 12 or at least a part of the content itself or an article to which the content is attached. Take up a business model for buying and selling.

  “Content” is defined to mean the content of music or other information. The content includes a design, a photograph, a picture, etc. in addition to a moving image / still image.

  The “article” of the present invention includes daily necessities, clothing, plants, animals, food, luxury goods, vehicles, etc. to which at least a part of the content is attached.

  For example, as shown in FIG. 20, the contents of the event are recorded by an imaging means (not shown) such as a camera to create a content 24 such as a moving image file. The content 24 itself or information related to the content 24 is stored in the artist terminal 26 using a dedicated application 28. Then, the content 24 or its information is uploaded from the artist terminal 26 to the content distribution server 30 using the dedicated application 28.

  The fan downloads the content or its information from the content distribution server 30 to the user terminal 32 using the dedicated application 34 via the Internet. Then, the desired content 24 is purchased from the user terminal 32 using the dedicated application 34. The billing process at this time can be performed for a billing management server 36 provided separately.

  In this way, the fan who has acquired the content 24 can enjoy the event by playing the content 24 using his / her terminal 32. This will delight fans and revitalize the music industry.

  The content 24 may be an article (goods) on which at least a part of the virtual video data DB 5 or the virtual video 14 is displayed. Here, as articles (goods), clothing, stationery, recorded media, and the like are applicable. The fans are pleased that at least a part of the virtual video 14 is displayed as a design on these articles. Note that an existing delivery route can be used to ship the article.

  Further, by obtaining the virtual video data DB5, it can be used for various purposes. Depending on the contents of the contract, the fan can freely edit the virtual video 14. Thereby, the novel content 24 can be sold to fans. As a result, the fans are delighted and the music industry is activated.

  Next, an embodiment of an event method using the virtual video 14 will be described. The following embodiments can also be applied to the concert holding method.

[First embodiment]
A first embodiment of an event method using virtual video will be described. Motion capture can be used as a method for generating the virtual video data 14. In the first embodiment, as an example of motion capture, an apparatus or system for executing the motion capture will be described.

  In the first embodiment, a known technique described in Japanese Patent Application Laid-Open No. 2007-18388 published at the JPO Digital Library is applied. The first embodiment will be described below based on the contents described in Japanese Patent Application Laid-Open No. 2007-18388 published at the JPO Electronic Library.

  In the first embodiment, FIG. 7 is a configuration diagram showing an outline of the configuration of the motion creation device 100. As shown in the figure, the motion creation device 100 is installed with a software for motion creation processing on a general-purpose computer and is activated to integrate hardware and software such as a CPU, ROM, RAM, and input / output ports (not shown). And is configured to function.

  The motion creation device 100 creates a motion of a human body image based on a music data input unit 102 that inputs music data, a music data analysis unit 104 that performs processing to analyze the input music data, and a result of analysis of the music data. A motion creation processing unit 106 that performs processing, a display processing unit 108 that performs processing for displaying the human body image on the display 101 so that the human body image operates with the created motion, and a motion database (hereinafter referred to as a motion graph) that registers a plurality of motions of the human body image , Referred to as motion DB) 110.

  The music data is generated in the form of audio data by recording in advance the artist's singing voice, speaking voice, and the like.

  The music data input unit 102 and the music data analysis unit 104 correspond to, for example, the audio information input unit 10D and the audio data generation unit 10F of the event system 10.

  The motion creation processing unit 106 corresponds to, for example, the body motion information input unit 10C, the motion data generation unit 10E, and the virtual video data generation unit 10G of the event system 10.

  The display processing unit 108 corresponds to the virtual video display control unit 10H of the event system 10, for example.

  The display 101 corresponds to, for example, the display unit 12 of the event system 10.

  The motion database 110 corresponds to the storage unit 10B of the event system 10, for example.

  FIG. 8 is an explanatory diagram showing an example of a human body model constituting one frame of a human body image. As shown in the figure, the human body model includes a vector R (= {rx, ry, rz}) indicating the direction of the human body, a vector t (= {tx, ty, tz}) indicating the position of the human body in space, 17 posture vectors (unit vectors) v0 to v16 indicating the directions of links connecting the elements of the human body model in the body center coordinate system {R, t} centered on the waist and the length of each link. It is configured as a human body posture S (f) (f is a frame number) consisting of nine scalar values 10 to l8 shown (see the following equations (1) to (3)).

  Here, in the body center coordinate system {R, t}, the z axis is a direction from the waist to the abdomen with the position of the waist as the origin, the y axis is the front direction, and the x axis is orthogonal to the z axis and the y axis. It was determined to be a direction. The nine scalar values 10 to 18 are the lengths of the right arms v1, v2, v3 and the left arms v5, v6, v7, the lengths of the right hand v4 and the left hand v8, the right feet v9, v10, v11, v12 and the left foot v13. , V14, v15, and v16 have the same length on the left and right.

  FIG. 9 is an explanatory diagram illustrating an example of a motion graph registered in the motion DB 110. As shown in the figure, the motion graph includes two frames whose human body postures S (f) are similar to each other in each frame on a plurality of motion data captured by motion capture (shown as motion data 1 to 6 in the figure). It is configured as a directed graph connecting frames (nodes).

For the connection between frames whose human body postures S (f) are similar to each other, for example, a correlation value is calculated according to the following equation (4) for any two frames on the motion data 1 to 6, and the calculated correlation value is What is larger than the threshold value is determined as frames fA and fB having similar human body postures S ^A (fA) and S ^B (fB), and the following equation (5) is determined between the two determined frames fA and fB. This can be done by concatenating the frames generated using the relational expression of the cubic interpolation that includes them.

  Here, the first term on the right side in Equation (4) indicates the correlation between the posture vectors v (fA) and v (fB) (i is the number of the posture vector) between the two frames fA and fB, and the second term on the right side. Indicates the correlation of the differential (motion) of the posture vectors v (fA) and v (fB) between the two frames fA and fB.

  “Αi” and “βi” are constants. In addition, “T” in Equation (5) indicates a transition time (the number of frames inserted between frames fA and fB). In Expression (5), only the orientation vectors v (fA) and v (fB) are described, but the frames R and FB are connected to each other by using the same cubic interpolation relational expression for the vectors R and t. Frames can be generated. When a motion graph is constructed in this way, motion of any pattern can be realized with a small amount of data by transitioning between nodes on the motion graph.

  The motion graph is registered in the motion DB 110 in such a manner that a key frame component MCkeyframe and a motion excitement component MCintensity are associated with each other for each frame.

  Here, in the key frame component MCkeyframe, the frames in which the operating speed of the right hand v4 and the left hand v8 are minimized are registered as key frames f and f (i and j are key frame numbers), respectively. A value obtained by applying a Gaussian function having a time width K centered on the time of the key frame f and a Gaussian function having a time width K centered on the time of the key frame f with respect to the frame f. A value obtained by adding together the values obtained by applying is set (see the following equation (6)).

  In the equation (6), “σ” indicates dispersion. An example of how the key frame component MCkeyframe is set is shown in FIG. As shown in the figure, the key frame component MCkeyframe is set to have a larger value as the frame f is closer to the key frames f and f, and the larger the value, the greater the correlation with the music rhythm (beat feature component ASCbeat) described later. Appear. This is based on the fact that it was confirmed that scenes where the movement of the hand stops during the dance (stopping action) tend to appear along with the rhythm of the music.

  The reason why the key frame component MCkeyframe has a temporal width K is that a person does not dance perfectly in accordance with the rhythm of the music, and a certain amount of deviation may occur. Further, the motion excitement component MCintity is obtained by applying a Gaussian function having a time width between the key frames f and f around the central time between the key frames f and f adjacent to the frame f, and Similarly, a value obtained by applying a Gaussian function with a time width between key frames f and f centered on a central time between adjacent key frames f and f is set for frame f. (Refer to the following formula (7)).

  In Expression (7), “v (i)” indicates the maximum speed of the right hand v4 between the adjacent key frames f and f, and “v (i)” is between the adjacent key frames f and f in the left hand v8. Shows the maximum speed of the left hand v8. FIG. 11 shows an example of how the motion excitement component MCintity is set.

  As shown in the drawing, the value of the motion excitement component MCintensity becomes larger as the frame f is closer to the central time (frame) between the adjacent key frames f and f and the central time (frame) between the adjacent key frames f and f. The maximum velocity v (i) of the right hand v4 between the adjacent key frames f and f and the maximum velocity v (i) of the left hand v8 between the adjacent key frames f and f are set so as to increase, and the music excitement described later A large correlation with the component ASC intensity appears. This is based on the fact that it was confirmed that the hands were moved fast when the music was exciting and the hands were moved only slowly when it was not.

  Next, an operation of the motion creation device 100 according to the embodiment configured as described above will be described. FIG. 12 is a flowchart illustrating an example of a main routine executed by the motion creation device 100. This routine is repeatedly executed every predetermined time (for example, every 3 seconds) when the operator gives an instruction to start motion creation.

  When the main routine is executed, the CPU of the motion creation device 100 first inputs music data (step S100). As the music data, data in any other format such as WAVE data or MIDI data may be input. The music data may be input by directly reading data from a recording medium such as a music CD, or may be input by capturing music being played from a microphone. Subsequently, the input music data is analyzed (step S110), and a motion is created by selecting an execution path Pathexec (trajectory on the motion graph) from the motion graph of the motion DB 110 based on the analysis result (step S120). The generated motion is displayed and output on the display 21 (step S130). The processes in steps S100 to S130 are repeatedly executed until the music is finished (step S140), and this routine is finished.

  In the embodiment, the music data analysis in step S110 is executed by a music data analysis process illustrated in FIG. In the music data analysis process, first, spectrum analysis is performed on the input music data (step S200). In the embodiment, the spectrum analysis is performed by Fourier transform using a Gaussian function that is said to be suitable for processing music data as a window function. Thereby, the spectrum power p (t, f) for each time component t and each frequency component f is derived.

  Subsequently, the beat interval and beat time of the input music are estimated based on the result of spectrum analysis (step S210). The beat interval is estimated by calculating the spectral power increment d (t, f) for each time component t and frequency component f by the following equation (8), and by the following equation (9). (T, f) is summed to calculate the spectrum power increment D (t), and the autocorrelation function RDD (τ) of the spectrum power increment D (t) is obtained by the equation (10), and it first shows a peak. It can be estimated by extracting the time τ_peak of the hour.

  FIG. 14 shows how the spectrum power increment d (t, f) is calculated. It should be noted that in the equation (8), the spectrum power above and below (f ± 1) of the target frequency f is included in the previous spectrum power PrevPow when a certain sound is sounded, and it is not necessarily sounded at a certain pitch. It is not based on the fact that the pitch goes up or down somewhat. The autocorrelation function RDD (τ) shown in the equation (10) indicates that the time τ_peak when the peak first arrives represents the period of the spectrum power increment D (t), and this time τ_peak represents the sounding interval, which is expressed as the beat interval. Can be considered.

  In the embodiment, the beat time is first estimated by setting a pulse train P (t) that generates a pulse with the estimated beat interval as a period, and the pulse train P (t) and the above-described spectrum power according to the following equation (11). A cross-correlation function RDP (τ) with the increment D (t) is obtained, a time τ_peak when it first shows a peak is extracted, and this time τ_peak is set as a temporary beat time.

  In the cross-correlation function RDP (τ) shown in Expression (11), the pulse train P (t + τ_peak) obtained by shifting the pulse train P (t) by the time τ_peak when the peak first arrives is similar to the spectrum power increment D (t). Therefore, a certain amount of beat time can be grasped from this time τ_peak. Then, when the peak of the spectrum power increment D (t, f) in a predetermined time width (for example, a time width of about 1/10 of the beat interval) around the set temporary beat time is extracted and indicated By setting this time as the beat time, an accurate beat time can be grasped.

  When the beat time is estimated in this way, a histogram Pist (t, f) of frequency components from the beat time t to the next beat time t + 1 is set (step S220). In the embodiment, for each frequency component f, the spectrum power p (t, f) is time-integrated from the beat time t to the next beat time t + 1, that is, one beat as an integration interval, and the integrated values are arranged in the order of frequency components. The peak of the integration value at that time is extracted, and the extracted peak is set as the histogram Pist (t, f) at the frequency component f and beat time t. An example of the frequency component histogram Pist (t, f) is shown in FIG.

  When the histogram Pist (t, f) of the frequency component is set, the beat feature component ASCbeat (t) is set based on the set histogram Pist (t, f) (step S230). The beat feature component ASCbeat (t) is set using the following equations (12) and (13).

  That is, for each frequency component f, the difference Pdiff (t, f) between the histograms Pist (t, f) and Pist (t + 1, f) of the frequency components at successive beat times t and t + 1 is obtained by the equation (12). The sum total over the frequency components 0 to 1 kHz according to the equation (13) is set as the beat feature component ASCbeat (t).

  Usually, there is a high possibility that a bar line (beat division) will appear at the time when the chord of the music changes, and when the chord changes, the frequency component changes greatly, and when the chord does not change, the frequency component hardly changes. It has been known. Therefore, the beat feature component ASCbeat (t) calculated as the temporal change degree of the frequency component histogram Pist (t, f) can be considered as the change degree of the chord at the beat time t.

  When the beat feature component ASCbeat (t) is set, the music excitement component ASCinttentity (t) is set based on the above-described frequency component histogram Pist (t, f) (step S240), and this routine is terminated. The setting of the music excitement component ASCintensity (t) is performed using equation (14).

  In equation (14), since the melody line in the audible area is considered to be about 400 to 600 Hz, the sum of the frequency component histograms Pist (t, f) over 400 to 1500 Hz is also taken into account. The component ASC intensity (t) was set. The reason why the music excitement component ASCinttentity (t) is set based on the histogram Pist (t, f) of the frequency component is based on the fact that the spectral power of the melody line of the music generally increases as the music rises.

  The motion creation process in step S120 is executed by the motion creation process illustrated in FIG. In the motion creation process, first, from the current beat time t along the path Path that can be selected from the current point on the motion graph until T time later (in the embodiment, 3 seconds, that is, 360 frames when the frame rate is 120 Hz). A correlation value MatchEvalbeat between the beat feature component ASCbeat (t + τ) (0 ≦ τ ≦ T) and the key frame component MCkeyframe described above is calculated using the following equation (15) (step S300), and a selectable path Path A process of selecting N (for example, five) paths Pathbeats in descending order of the correlation value MatchEvalbeat from the list (step S310).

  In this process, the correlation value MatchEvalbeat is calculated as a larger value as the point at which the beat feature component ASCbeat (t) occurs and the point at which the key frame component MCkeyframe is generated. It can be said that the path Pathbeat is selected so that a scene (stopping operation) stops.

  When N paths Pathbeat are selected in this way, a correlation value between the music excitement component ASC intensity (t + τ) and the above-described motion excitement component MC intensity (t + τ) is calculated by the following equation (16) for each of the N paths Pathbeat. S320), the path having the largest calculated correlation value among the selected N paths Pathbeat is set as the execution path Pathexec (step S330), and this routine is terminated.

  According to the first embodiment, each frame constituting the motion of the human body image is registered in the motion DB 110 in association with the key frame component MCkeyframe, the music data is input, the input music data is analyzed, and the beat feature component ASCbeat is calculated. Extracted and selected path Pathbeat with a large correlation value MatchEvalbeat between extracted beat feature component ASCbeat and key frame component MCkeyframe, and creates and displays a motion, so a more appropriate motion that matches the rhythm of music is created in real time can do. Moreover, in addition to the key frame component MCkeyframe, the motion excitement component MCtensity is also registered in the motion DB 110 in association with each frame, and the music excitement component MCintity is extracted from the input music data, and the correlation between the motion excitement component MCintity and the music excitement component ASCintensity Since the path with the largest value is selected, a motion corresponding to the size of the music can be created.

  The motion creation device 100 executes a path having the largest correlation value between the music excitement component ASCintensity and the motion excitement component MCintity from among N paths Pathbeat having a large correlation value MatchEvalbeat between the beat feature component ASCbeat and the key frame component MCkeyframe. Although the path Pathexec is set, the path having the largest correlation value MatchEvalbeat between the beat feature component ASCbeat and the key frame component MCkeyframe may be set as the execution path Pathexec regardless of the excitement of music.

  In the motion creation device 100, the beat feature component ASCbeat is set so that the correlation with the key frame component MCkeyframe appears greatly when the beat time at which a change occurs in the chord of music arrives. The beat feature component ASCbeat may be set so that the value is greatly correlated with the key frame component MCkeyframe when the beat time arrives. The beat feature component may be set using any other parameter indicating the beat feature. ASCbeat may be set.

  In the motion creation device 100, the key frame component MCkeyframe and the motion swell component MCintensity are set based on the motion speeds of the hands v4 and v8. However, the motions of the arms v1 to v3 and v5 to v7 are not limited to the hands v4 and v8. The key frame component MCkeyframe and the motion excitement component MCtensity may be set based on the speed, and the operation speed of other parts may be considered.

  In the motion creation device 100, motion (animation) of a human body image is created. However, the motion creation device 100 is not limited to a human body image, and may be applied to creation of motion of a human body robot. It can also be applied to create motions for non-human animal images such as dogs and cats and other anime character images, and motions for non-human animal robots such as dogs and cats and other character robots. It may be applied to the creation of When creating a motion from an image, the image used may be a two-dimensional image or a three-dimensional image.

  According to the first embodiment, the virtual video data DB5 (see FIG. 2) edited by synchronizing or synthesizing the motion data DB3 (see FIG. 2) and the audio data DB4 (see FIG. 2) can be acquired. .

  The first embodiment uses the motion capture technique, but is not limited to this, and a currently known motion capture technique can be applied.

  There are three types of motion capture technology: optical, magnetic, and mechanical. All of these methods can be used.

[Second Embodiment]
A second embodiment of the event method using virtual video will be described. Motion capture can be used as a method for generating the virtual video 14. In the second embodiment, as an example of motion capture, an apparatus or system for executing the motion capture will be described.

  In the second embodiment, a known technique described in Japanese Patent Application Laid-Open No. 2005-337983 published at the JPO Electronic Library is applied. The second embodiment will be described below based on the contents described in Japanese Patent Application Laid-Open No. 2005-337983 published at the JPO Electronic Library.

  In the second embodiment, as shown in FIG. 17, the motion capture system 200 has 18 gyro sensors attached to each part of the human body (hereinafter referred to as a person to be measured) and the skeleton data of the person to be measured is stored in a computer. The 18 gyro sensors are connected to the transmitter by wire, and the transmitter is attached to the person to be measured.

  The transmitting device wirelessly transmits data from each gyro sensor to a computer, receives the data by the computer, and organizes and records the data by a method corresponding to the purpose.

  The computer is equipped with posture control software from the waist down so that it can cope with general movements in which the measurement subject moves back and forth and from side to side.

  The gyro sensor is a three-dimensional sensor, and can accurately collect data in the rotation direction, so that it can also record subtle movements of the body of the subject and changes in each part.

  Since the mounting position of the gyro sensor is positioned based on the skeleton of the person to be measured, there is no possibility of deviation after the mounting, and the data does not need to be corrected.

  That is, the data input to the computer is directly calculated as data at the skeleton-corresponding position, so that the true form and the linkage can be reproduced, and the movement of the measurement subject can be reproduced accurately and smoothly.

  By replacing the measurement subject's skeleton size with a previously measured monkey skeleton size, the monkey's movement can be expressed and recorded.

  That is, in order to record the characters of the animation, the activity of the characters (for example, other animals, fictional animals, and imaginary animals) can be truly realized by using the data recorded on the human body.

  Although 18 gyro sensors are attached, the number is not limited to 18 and it is possible to change the attachment position and the number of attachments in order to adopt detailed partial movement.

  Therefore, it can be used as motion improvement data for various robots (nursing care, household, industrial and other various working robots replacing human bodies) from animated animals.

  In order to replace the recording of the movement of the person being measured (human) with the movement of the desired animal (for example, a monkey), it has been explained that the skeleton data of the person to be measured should be converted into the skeleton data of the monkey.

  More precisely, by converting the motion state of the same kind of animal (for example, a Japanese monkey) to the skeleton data of a Japanese monkey that is to be adopted for animation, it is possible to make the motion closer to the truth.

  As described above, the present invention can measure and record non-human animals and other moving objects, and analyze and improve the movement.

(Concrete example)
As shown in FIG. 18, gyro sensors 204a, 204b, 204c, 204d, 204e, 204f, 204g, 204h, 204i, 204j, 204k, 204l, 204m, 204n, 204o, 204p, 204q and 204r are attached respectively.

  Next, the transmitter 208 having a built-in computer is connected to each gyro sensor 204 (without alphabets when referring to all of them) by an electric wire 212. The transmitter 208 uses a belt or the like to measure the subject 202. Wear on the waist. The transmitting device 208 and the receiving device 214 with a computer are transmitted and received wirelessly.

  Therefore, the movement data detected in each part of the body of the person to be measured 202 is sent to the transmitting device via the electric wire 212, and the position is determined by calculation (determining the position from the points of X, Y, and Z). This calculated value is wirelessly sent to the receiving device 214 and input to the built-in computer installed in the receiving device 214, and an image of the motion of the skeleton is recorded from the relationship with the skeleton registered in the built-in computer. Since data of 100 to 500 per second is input from the gyro sensor 204, the movement of the skeleton is performed extremely smoothly.

  According to the specific example, 18 or more gyro sensors 204 are attached to the body of the person 202 to be measured, the three-dimensional data is sent, and the movement of the skeleton is displayed based on this data. Can be recorded accurately.

  Further, since the data of the gyro sensor 204 is wirelessly transmitted by computer processing (one point is obtained from the three-dimensional data), for example, the movement of the driver can be accurately recorded even during driving of an automobile or a motorcycle. Such a thing was impossible with the conventionally known motion capture.

  FIG. 19 shows the attachment position of the gyro sensor 204 on the main skeleton. That is, the gyro sensor 204 is attached to the head 216, the neck 218, the waist 220, the base of the leg 222, and the foot 224, respectively. In this case, although it is an example for measuring the movement of the important part of the human body, at least seven gyro sensors 204 are required for the human body as shown in FIG. In the case of humans, the minimum required number varies depending on the case of animals.

  In the second embodiment, the motion data DB 3 (see FIG. 2) can be acquired by the method described above. Virtual video data DB5 (see FIG. 2) can be obtained by synchronizing or synthesizing music data or audio data DB4 (see FIG. 2) with this motion data DB3.

  The best mode for carrying out the present invention has been described with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the gist of the present invention. Of course, it can be implemented in the form.

10 Event System 10A Main Control Unit 10B Storage Unit 10C Body Motion Information Input Unit (Body Motion Information Acquisition Unit)
10D voice information input unit (voice information acquisition means)
10E Motion data generator (motion data generator)
10F voice data generation unit (voice data generation means)
10G virtual video data generator (virtual video data generator)
10H Virtual video display control unit (virtual video display control means)
10I audio output unit (audio output means)
DESCRIPTION OF SYMBOLS 12 Display part 14 Virtual image | video 16 Artist person 17 Event hall 18 Audience seat 20 Audience area 22 Stage 24 Content 26 Artist terminal 28 Application 30 Content distribution server 32 User terminal 34 Application 36 Billing management server DB1 Body motion information DB2 Voice Information DB3 Motion data DB4 Audio data DB5 Virtual video data PG1 Motion data generation program PG2 Audio data generation program PG3 Virtual video data generation program S10 Physical motion information acquisition step S11 Audio information acquisition step S12 Motion data generation step S13 Audio data generation step S14 Virtual Video data generation step S15 Virtual video display control step S16 Audio output Step

Claims (11)

An event method using virtual images,
A body motion information acquisition step of acquiring body motion information related to body motion by the body motion information acquisition means;
An audio information acquisition step of acquiring audio information related to audio by the audio information acquisition means;
A motion data generation step of generating motion data based on the body movement information by means of motion data generation means;
A voice data generation step of generating voice data based on the voice information by voice data generation means;
A virtual video data generating step for generating virtual video data in which the motion data and the audio data are associated with each other by a virtual video data generating unit;
A virtual video display control step of projecting a virtual video based on the virtual video data on a display unit by a virtual video display control means;
An event method using virtual video, characterized by comprising:   The event method using a virtual image according to claim 1, further comprising an audio output step of outputting audio based on the audio data by an audio output means. The body movement information is information about movement when singing a song,
3. The event method using a virtual image according to claim 1, wherein the motion data is data related to an operation when singing a song. The audio information is information about singing voice,
The event method using a virtual image according to any one of claims 1 to 3, wherein the audio data is data related to a singing voice.   The event method using a virtual image according to any one of claims 1 to 4, wherein the virtual image data is data relating to a silhouette of a body that moves in accordance with sound.   6. The event method using a virtual image according to claim 1, wherein in the virtual image display control step, the virtual image is displayed in the vicinity of an artist.   The event method using a virtual video according to claim 1, wherein the virtual video is a two-dimensional video.   A concert holding method, wherein a concert is performed by an event method using the virtual image according to any one of claims 1 to 7.   An event venue that implements the event method using the virtual image according to any one of claims 1 to 7.   The event venue according to claim 9, wherein the event venue has a spectator seat. 8. At least a part of virtual video or virtual video data acquired by the event method using virtual video according to any one of claims 1 to 7 is sold as content or an article to which the content is attached. How to buy and sell content.

Images