JP4981370B2 - Movie generation system and movie generation method - Google Patents

Description

  The present invention generates a synthesized moving image by synthesizing an actually photographed subject image with a moving image including a three-dimensional human body image that can perform various operations and can freely change the viewing position. The present invention relates to a moving image generation system and a moving image generation method.

  Conventionally, a three-dimensional moving image is created using a motion capture technique (see Patent Document 1). Motion capture is a process of attaching a marker in a 3D coordinate system by attaching a plurality of markers for data acquisition to the required part of the human body and having the person perform movements such as dance and sports. Motion data (posture information) representing coordinate values and angles is acquired.

  If a motion object (three-dimensional human body image imitating a human body) representing a human body is created using the motion data acquired in this way, the motion object performs various operations with postures based on the motion data. You can create a video of the content. In addition, moving images based on motion data can freely change the direction in which the moving object is viewed by prescribing the position where the moving object is viewed, for example, generating a moving image when the moving object is viewed from the front, You can also create a video as seen from above.

  On the other hand, apart from the contents using the motion capture technology described above, it has been conventionally possible to extract only the image of the subject from the video including the photographed subject (actual video) and synthesize the extracted subject image with another image. It is made from. For example, there are a chroma key method, a roto scoping method, a difference matting method and the like as a method for extracting only a subject image from a live-action video.

  Further, as another image to be combined with the image of the subject extracted using the above-described method, there are images that are generated at any time other than previously prepared still images and moving images. is there. For example, in Patent Document 2, a video camera for photographing a singer is provided in addition to a processing unit that performs image composition, and a graphic image (for example, balloons fly around) corresponding to a detection result based on the image of the singer photographed by the video camera. It is disclosed that a video of a singer photographed with a video camera is synthesized with this graphic video. Note that Patent Document 2 also describes that the motion of a singer shooting with a video camera is detected, and for example, when the singer's right hand touches a balloon of a graphic image, the graphic image to be synthesized is changed.

Further, in Patent Document 3, a processing unit that performs image composition and a camera that captures an actor are provided, and an operator can obtain a CG operation unit based on three-dimensional model data of CG (computer graphics) characters accumulated in advance. It is disclosed that three-dimensional CG data is generated by adding the CG motion information input in step S3, and that the three-dimensional CG data of the actor photographed by the camera is combined with the generated three-dimensional CG data. In Patent Document 3, when a synthesized video is displayed, a virtual video that can be seen from the viewpoint of the operator (that is, the viewpoint of the CG character) is displayed so that the operator of the CG operation unit can easily understand. It is described to present.
Japanese Patent Laid-Open No. 10-222668 JP-A-5-2322861 JP 2000-23037 A

  The content described in Patent Document 2 is that a graphic image to be combined with a captured image is a two-dimensional image. Therefore, a three-dimensional depth and a three-dimensional image change are added to the combined image. There is a problem that it cannot be expressed.

  Also, the content described in Patent Document 3 is that the CG motion information input by the operator is added to generate 3D CG data, and the viewing direction of 3D CG data cannot be changed. However, there is a problem that the angle change with respect to the 3D CG data included in the synthesized video is scarce and the user is easily bored.

The present invention has been made in view of such circumstances, and by generating a moving image in which the moving object viewing direction can be changed as a composition target of the captured subject image, it is possible to generate a composite image with various angle changes. It is an object to provide a moving image generation system and a moving image generation method.
Another object of the present invention is to provide a moving image generation system that can generate a composite image in which the image content changes variously every time the position where the moving object is viewed is changed based on various conditions.
Furthermore, the present invention expands the range of use of the generated composite image by combining the generated composite image with karaoke, enabling the composite image to be stored in a storage medium, and enabling the composite image to be distributed over a network. An object of the present invention is to provide a moving image generation system that can be used.

In order to solve the above problems, the moving image generating system according to the present invention includes a posture based on posture information that defines a posture of a moving object in a three-dimensional coordinate system for each unit time, and viewpoint information that defines a position where the moving object is viewed. In a moving image generation system including a moving image generation unit that generates a moving image including a moving object with a specified viewpoint, an imaging unit that captures a subject, and an image extraction unit that extracts a subject image included in a video captured by the imaging unit And a synthesized moving picture generating means for generating a synthesized moving picture by synthesizing the subject image extracted by the image extracting means with the moving picture generated by the moving picture generating means.
In addition, the moving image generation method according to the present invention includes a moving image generation system based on posture information that defines a posture of a moving object in a three-dimensional coordinate system for each unit time and viewpoint information that defines a position where the moving object is viewed. In the moving image generation method for generating a moving image including a moving object with a specified viewpoint, the moving image generation system captures a subject, extracts a subject image included in the captured video, and converts the extracted subject image into the moving image. A synthesized moving image is generated by synthesis.

  In the present invention, the subject image extracted from the captured video is synthesized into a moving image including a moving object whose three-dimensional form is specified using posture information and viewpoint information. The viewing direction can be changed in various ways. As a result, a composite image with various angle changes can be generated, and a composite image applicable to various uses can be provided to various users. In particular, by photographing the user as the subject of the photographing means, it is possible to realize a new amusement service by allowing the user to appear in a moving image including a moving object whose viewpoint can be changed.

  The video generation system according to the present invention includes a music acquisition unit that acquires music, a music playback unit that performs playback processing of the music acquired by the music acquisition unit, and the synthetic video according to the playback processing of the music playback unit. And display processing means for performing the display processing.

  In the present invention, by performing display processing of the generated composite video in accordance with the playback processing of the acquired music, a composite video including a moving body that can perform various movements and change the angle freely can be processed. The user can be entertained in both hearing and vision.

  Further, in the moving image generating system according to the present invention, the music acquisition means is configured to acquire a music accompanied by a character representing lyrics, and a character that combines the character attached to the music with the synthesized moving image. The image processing apparatus includes a composing unit, and the display processing unit performs display processing of a synthesized moving image in which characters are synthesized by the character synthesizing unit.

  In the present invention, a character suitable for karaoke can be provided because the character representing the lyrics attached to the music is combined with the synthesized moving image and displayed. That is, since the telop (characters) of the lyrics is displayed in the synthesized video, the user can sing with reference to the telop. In particular, the user shoots the user singing with the shooting means and moves the moving object included in the video as a back dancer. If you create a human body image that mimics the situation, the user will be able to experience the situation of a singer with a back dancer that makes various movements. Can also be confirmed by watching the synthesized video, and can add new functions to entertain users in karaoke. Furthermore, when the present invention is applied to karaoke, in addition to the user who sings, the user who sees the composite video to be displayed can also enjoy watching the display reflected by the person (user) who is singing, and the user who sings and the user who watches Can enhance the sense of solidarity.

  Furthermore, in the moving image generation system according to the present invention, the music acquisition unit acquires a piece of music in which a plurality of viewpoint information is attached in the order of music progression, and the moving image generation unit is a reproduction process of the music reproduction unit. The moving image is generated based on the viewpoint information corresponding to the progress time of.

  According to the present invention, a music reproduction process in which a plurality of viewpoint information is added in the order of music progression is performed, and a moving image is generated based on the viewpoint information corresponding to the point in time of the reproduction processing, so that a synthetic video in which display processing is performed The position where the moving object is viewed changes as the music playback process proceeds. Therefore, the user can enjoy the change in the angle of the moving object in the synthesized moving image linked to the music reproduction process.

  Furthermore, the moving image generation system according to the present invention includes a selection unit that randomly selects one viewpoint information from a plurality of viewpoint information, and the moving image generation unit generates a moving image based on the selected viewpoint information. It is characterized by doing.

  In the present invention, since viewpoint information is randomly selected from a plurality of viewpoint information, and a moving image is generated based on the viewpoint information, a synthetic moving image in which the position where the moving object is viewed varies randomly is generated. It is possible to provide a user with a composite video with various viewpoint changes of moving objects.

  Also, the moving image generating system according to the present invention associates the image position detecting means for detecting the position of the subject image included in the video imaged by the imaging means with each position in the video and a plurality of viewpoint information. A position correspondence table; and means for selecting viewpoint information corresponding to the position detected by the image position detection means from the position correspondence table, wherein the moving image generation means is the viewpoint information selected from the position correspondence table. A moving image is generated based on the above.

  In the present invention, based on the position of the subject image in the captured video, a moving image in which the position where the moving object is viewed changes is generated, so the angle of the moving object in the composite image also varies depending on the position of the subject, A composite video in which the angle of the moving object changes in conjunction with a change in the position of the subject can be obtained. Therefore, if the subject is a user, if the user actively moves, the angle of the moving object in the synthesized moving image can be changed. Therefore, the user can freely control the angle of the moving object by controlling the position where the user reflects in the captured video. Note that it is preferable to change the position where the moving object is viewed from the viewpoint of obtaining a synthesized moving image having an optimal composition, with reference to the position of the detected subject image so as not to overlap the subject image.

  Furthermore, the moving image generation system according to the present invention is an operation in which the motion detection unit that detects the motion of the subject image in the video captured by the capturing unit, the motions related to the subject image, and the motions associated with the plurality of viewpoint information, respectively. A correspondence table; and means for selecting viewpoint information corresponding to the motion detected by the motion detection means from the motion correspondence table, wherein the moving image generation means is based on the viewpoint information selected from the motion correspondence table. It is characterized by generating a moving image.

  In the present invention, a moving image in which the position where the moving object is viewed changes is generated based on the operation of the subject image in the captured video, so the angle of the moving object in the synthesized moving image also changes in conjunction with the operation of the subject. It becomes like this. Therefore, if the subject is a user, the user can control the angle in the synthesized moving image by his / her movement, and can add a new function to entertain the user in the synthesized moving image service.

  Furthermore, in the moving image generating system according to the present invention, the posture information defines a posture of a three-dimensional coordinate system related to a plurality of moving objects, and the moving image generating unit is configured to generate a plurality of moving objects based on the posture information. It is characterized by producing | generating the moving image containing.

  In the present invention, since the posture information defines postures related to a plurality of moving objects, a moving image including a plurality of moving objects is generated, and the number of moving objects in the composite moving image is abundant and more dynamic content is generated. Can be provided to the user. For example, when the present invention is applied to karaoke and a user is photographed as a subject, and the moving body is a human body image that imitates a back dancer, the user can enjoy a singing atmosphere surrounded by a plurality of back dancers. It becomes possible and can enhance the fun of karaoke.

  The moving image generating system according to the present invention further includes means for storing background information that defines a position of a background in a three-dimensional coordinate system, and the moving image generating means includes a moving image including a background based on the stored background information. Is generated.

  In the present invention, in addition to moving objects, a moving image including a background is generated, so a background is also added to the synthesized moving image, and the details of the synthesized moving image are detailed to entertain the user's eyes. Can do. In particular, when the present invention is applied to karaoke and a user is photographed as a subject, the moving body is a human body image simulating a back dancer, and the background is a stage, the user can sing along with the back dancer on the stage. It becomes possible to enjoy in a pseudo manner, and the joy of karaoke can be further improved visually.

  Furthermore, the moving image generating system according to the present invention is characterized by comprising storage processing means for performing processing for storing the synthesized moving image generated by the synthesized moving image generating means in a storage medium.

  In the present invention, since the generated composite video can be stored in a storage medium, for example, if the composite video that the user himself appears is stored in a removable storage medium such as a DVD, and the storage medium is reproduced, You can enjoy a simulated experience at home, and can provide services that can be used at various events such as wedding reunions, alumni associations, and various auditions.

  Furthermore, the moving image generation system according to the present invention receives a moving image request signal transmitted through a network, and transmits a moving image to a transmission source of the moving image request signal when the receiving unit receives the moving image request signal. Moving image transmitting means for transmitting the synthesized moving image generated by the synthesized moving image generating means.

  In the present invention, since the generated synthesized video can be distributed to the requesting user via the network, the synthesized video can be widely distributed using the network, and the usage range of the generated synthesized video can be expanded. .

  In the present invention, the subject image extracted from the captured video is synthesized into a moving image including a moving object whose three-dimensional state is specified using viewpoint information in addition to posture information. The viewing direction of the moving object included in the moving image can be changed, and the viewing direction of the moving object in the synthesized moving image can be changed to entertain the user's eyes every time in various expression forms.

  In addition, in the present invention, by performing display processing of the generated composite video in accordance with the playback processing of the acquired music, a composite video in which the angle at which the moving object is viewed changes according to the progress of the music is displayed to the user. It can be displayed, and enjoyment can be provided to the user in both auditory and visual senses.

Furthermore, in the present invention, since the character representing the lyrics attached to the music is combined with the composite video and the display processing is performed, the lyrics are displayed in the composite video in which the moving direction of the moving object can be changed variously. A system suitable for karaoke can be realized, and a user who sings karaoke can enjoy a pseudo stage experience, and a user who sees the displayed synthesized video can enjoy display contents that acquaintances, friends, etc. appear on the screen.
Furthermore, in the present invention, since the music with a plurality of viewpoint information is used, the position of viewing the moving object in the synthesized video is changed according to the progress of the music playback process, and the music intro and excitement The user can be entertained with the display content that matches the playback status of the music such as the location.

  In the present invention, viewpoint information is randomly selected from a plurality of viewpoint information with different viewing positions, and a moving image is generated based on the viewpoint information. It is possible to provide the user with a synthetic video that does not get tired of changes.

Furthermore, according to the present invention, the position where the moving object in the synthesized moving image is viewed can be changed based on the position of the subject image in the shot video, and the position where the shot is taken can be changed for the user who is the shooting subject. Thus, a system capable of controlling the position of the moving object can be realized.
Furthermore, in the present invention, based on the operation of the subject image in the captured video, the position of viewing the moving object in the composite video can be changed, and the operation during the shooting is performed for the user who is the subject of the shooting. A system that can control the position of the moving object can be realized by changing it.

In the present invention, since the posture information defines postures related to a plurality of moving objects, it is possible to provide the user with a dynamic motion synthesized moving image in which the angles of the plurality of moving objects change.
In the present invention, since a moving image including a background in addition to a moving object is generated, moving objects and backgrounds whose viewpoint position can be changed appear in the combined moving image, and a combined moving image having a more various three-dimensional content is displayed. Can be created.

In the present invention, since the generated synthesized moving image is stored in the storage medium, it is possible to provide an opportunity for many people to see the generated synthesized movie through the storage medium.
Further, in the present invention, since the generated synthesized video can be distributed to the requesting user via the network, it is possible to provide an opportunity for many people to easily see the synthesized video using the network. .

  FIG. 1 shows an overall configuration of a karaoke system 1 to which a moving image generating system 10 according to the first embodiment of the present invention is applied. The karaoke system 1 uses the video generation system 10 of the present invention to generate a composite video in which a user U who sings karaoke music appears and displays it on the large display 2. A person who sings a new enjoyment of karaoke by making it possible to simulate the situation of singing in front of human back dancers 15a to 15c and allowing the surrounding users to enjoy the performance of the singing user U, The feature is that it can be provided to both viewers.

  The karaoke system 1 according to the first embodiment employs a chroma key method in order to capture a subject image obtained by photographing the user U, and has a blue wall on the back and the periphery of the place where the user U who sings karaoke music is located. A member 6 is provided. In addition, the karaoke system 1 arrange | positions the sub display 4 for the user U who sings to confirm a lyrics, an own picked-up image, etc. so that the wall member 6 may be opposed. In addition, the karaoke system 1 connects the karaoke music server 5 that distributes karaoke music to the video generation system 10 through the network NW, and is large enough to allow the surrounding users to check the generated synthesized video and the song of the user U. A display 2 and left and right speakers 3a and 3b are provided.

  A moving image generating system 10 according to the present embodiment applied to the karaoke system 1 corresponds to a range surrounded by a wavy line in FIG. 1 and includes a camera device 11, a chroma key device 20, a 3D moving image generating device 30, a distributing device 40, and a karaoke device. 41, a storage device 44, and a 3D moving image distribution server 45. In the moving image generation system 10, the camera device 11, the chroma key device 20, and the 3D moving image generation device 30 are indispensable, and the other parts (distribution device 40, karaoke device 41, etc.) are the types of services to be applied. This corresponds to a peripheral device that is optionally added depending on the device.

  Therefore, since the moving image generating system 10 of the first embodiment is applied to the karaoke system 1, the moving image generating system 10 includes the distribution device 40 and the karaoke device 41, and can further store the generated combined moving image in a storage medium (DVD). In addition, a storage device 44 and a 3D moving image distribution server 45 are provided to enable distribution through the network NW. Hereinafter, each of the devices 11 and 20 included in the moving image generation system 10 will be described from the added peripheral devices (distribution device 40 and karaoke device 41 and the like).

  The distribution device 40 performs processing for distributing the composite moving image generated by the chroma key device 20 and sending it to a plurality of distribution destinations. Specifically, the composite moving image received from the chroma key device 20 through the third video line V3 is sent to the large display 2 through the fourth video line V4, and sent to the karaoke device 41 through the fifth video line V5.

  FIG. 2 is a block diagram showing the internal configuration of the karaoke apparatus 41. The karaoke apparatus 41 includes a control unit 41a that performs various controls, a communication processing unit 41b, a karaoke music processing unit 41c, a music reproduction processing unit 41d, a memory unit 41e, a telop synthesis unit 41f, an input / output interface 41g, via an internal bus 41i. The infrared light receiving unit 41h is connected.

  The communication processing unit 41b corresponds to music acquisition means, is connected to the karaoke music server 5 via the network NW, and transmits a request signal for karaoke music specified by the user to the karaoke music server 5 based on the control instruction of the control unit 41a. To do. The karaoke song server 5 is configured to transmit the specified karaoke song with the reception of the request signal, and the communication processing unit 41b receives and acquires the karaoke song transmitted from the karaoke song server 5. Note that the karaoke music distributed by the karaoke music server 5 is obtained by adding character data (telop) representing lyrics to music data related to music.

  Furthermore, in the present embodiment, the communication processing unit 41b includes a sound file of a synthesized sound obtained by synthesizing karaoke music and user voice by a voice synthesizing unit 42a, which will be described later, and a video file of a synthesized moving image obtained by synthesizing a telop by the telop synthesizing unit 41f. Is also read out from the memory unit 41e, and the both are associated with each other and uploaded (transmitted) to the 3D moving image distribution server 45 via the network NW. Such upload processing is performed based on the control instruction of the control unit 41a, and when uploading, the date and information on the place where the karaoke was performed (for example, the name of the karaoke store) are added. Sent in the specified file format.

  The karaoke song processing unit 41c separates the karaoke song acquired by the communication processing unit 41b into song data and character data, and sends the song data to the song reproduction processing unit 41d, and the character data is converted into a telop synthesis unit. Processing to send to 41f is performed.

  The music reproduction processing unit 41d corresponds to music reproduction means, and sequentially performs the reproduction processing of the music data and sends the reproduced music sound to the voice synthesis unit 42a. User speech of the user U who sings karaoke music received by the microphone voice input unit 42b connected to the first voice cable A1 is also sent to the voice synthesis unit 42a. The voice synthesis unit 42a A process of synthesizing voice is performed, and the synthesized sound is sent to the amplifying unit 42c and also sent to the storage device 44 through the second voice cable A2. Furthermore, the voice synthesizer 42a also performs a process of converting the synthesized sound into a sound file for uploading and sending it to the memory unit 41e. The amplifying unit 42c amplifies the synthesized sound and outputs it from the left and right speakers 3a, 3b through the third audio cable A3.

  On the other hand, the telop synthesis unit 41f corresponds to a character synthesis unit and is connected to the moving image input unit 42d. The moving image input unit 42 acquires a combined moving image generated by the chroma key device 20 to be described later via the fifth video line V5, and performs processing for sending the acquired combined moving image to the telop combining unit 41f. Accordingly, when the telop synthesizing unit 41f receives the transmitted synthesized moving image, the telop synthesizing unit 41f performs a process of synthesizing the karaoke music telop with the synthesized moving image, and sends the synthesized movie (see FIG. 16) synthesized with the telop to the moving image interface unit 42e. The telop compositing unit 41f also performs processing for sending the composite video that has undergone the compositing process to the memory unit 41e as a video file for upload.

  The moving image interface unit 42e corresponds to display processing means, converts the received combined moving image into display data (data signal and scanning signal), and sends it to the sub display 4 through the sixth video line V6 at a predetermined timing. Process. The timing of the above-described display processing in the moving image interface unit 42e is synchronized with the playback processing in the music playback processing unit 41d by the control unit 41a, and the timing according to the playback status of the music is as shown in FIG. Display processing is performed so that a composite video with telop of karaoke music is displayed. The moving picture interface unit 42e sends the moving picture data of the synthesized moving picture to the storage device 44 through the seventh video line V7 in accordance with the timing at which the voice synthesis unit 41a sends the sound data to the storage device 44 through the second audio cable A2. Processing is also performed.

  The memory unit 41e temporarily stores a sound file and a moving image file for uploading in association with each other. Further, the input / output interface 41g is connected to a 3D moving image generating apparatus 30 described later through a network cable L1. The input / output interface 41g sends a playback start signal to the 3D video generation device 30 when the music playback processing unit 41d starts the playback process based on the control of the control unit 41a. Processing to send to the generation device 30 is performed.

  The infrared light receiving unit 41h receives infrared light including an operation instruction issued from the remote control device 43 operated by the user, and receives an operation instruction included in the received infrared light through the internal bus 40i. Process to send to. The control unit 41a controls the above-described units 41b and 41c according to the received operation instruction.

  Further, the storage device 44 shown in FIG. 1 corresponds to storage processing means for storing the generated composite moving image on a DVD as a storage medium. Specifically, a telop is sent from the karaoke device 41 through the seventh video line V7. The synthesized synthesized video is received, and the synthesized sound of the karaoke music sound and the user voice is received from the karaoke apparatus 41 through the second audio cable A2. The storage device 44 performs a process of storing (storing) the received synthesized moving image and synthesized sound on the DVD by the storage processing unit 44a. It is available for a fee as one of the service menus.

  FIG. 3 is a block diagram illustrating a main internal configuration of the 3D moving image distribution server 45. The 3D moving image distribution server 45 enables the combined moving image generated by the chroma key device 20 to be widely distributed through the network NW, and a website having a web page 47 as shown in FIG. 4 is provided on the network. In the 3D moving image distribution server 45, an MPU 45a, a communication interface 45b, a RAM 45c, a ROM 45d, and a hard disk device 45e are connected by an internal bus 45i.

  The communication interface 45b is connected to the network NW, and transmits and receives various signals and data files. In the present embodiment, the communication interface 45b receives a composite video and audio file (video file and sound file) uploaded from the karaoke apparatus 41. Then, processing to send to the content database 46 stored in the hard disk device 45e is performed. The communication interface 45b transmits the page data of the web page 47 to the access source that has accessed the website through the network NW under the control of the MPU 45a, receives the content request signal (video request signal), and receives the content (video file). And sound files) are also performed based on the control of the MPU 45a.

  The RAM 45c temporarily stores data, folders, and the like according to the processing of the MPU 45a, and the ROM 45d stores in advance a program that defines basic processing contents performed by the MPU 45a. The hard disk device 45e includes a server program 45f that defines basic processing of the server, a distribution program 45g that defines content distribution processing, page data 45h for web pages, and content that stores uploaded video files and sound files. A database 46 is stored.

  The distribution program 45g defines the contents of the control process of the MPU 45a related to content distribution, and performs the process of displaying the site page 47 of FIG. 4 on the terminal that is the website access source. The site page 47 is a state in which the content stored in the content database 46 is selected in the selection column 47a and the selection column 47a in which the creation date of the content and the karaoke shop where the content is created can be selected. A determination button 47b for determining content distribution and a cancel button 47c for canceling the selected content are provided. When the determination button 47b is selected at the access source terminal, a moving image request signal for requesting the selected content is transmitted to the 3D moving image distribution server 45.

  Therefore, when the distribution program 45g receives the video request signal by the communication interface 45b of the 3D video distribution server 45, the distribution program 45g reads the selected content (video file and sound file) from the content database 46 and communicates with the access source terminal. 45b is specified for transmission.

  Next, the camera device 11, the chroma key device 20, and the 3D moving image generating device 30 that are essential in the moving image generating system 10 will be described. The camera device 11 corresponds to photographing means for photographing a user U who sings karaoke music as a subject at a predetermined frame rate, and the photographed images are sequentially sent to the chroma key device 20 through the first video line V1.

  FIG. 5 is a block diagram showing an internal configuration of the chroma key device 20. The chroma key device 20 includes a first input unit 21, a second input unit 22, a subject image extraction unit 23, a synthesis unit 24, and an output unit 25. The first input unit 21 is connected to the first video line V <b> 1, and a captured video sent from the camera device 11 is input. The second input unit 22 is connected to the second video line V2, and receives the 3D moving image generated by the 3D moving image generating device 30 (see FIG. 14B). The subject image extraction unit 23 corresponds to an image extraction unit, and performs a process of extracting only an image (subject image) of the user U included in the captured video input to the first input unit 21 by the chroma key method. The image (see FIG. 14A) is sent to the combining unit 24.

  The synthesizing unit 24 corresponds to a synthetic moving image generating unit, and synthesizes the subject image sent from the subject image extracting unit 23 with the 3D moving image input by the second input unit 22 and displays the synthesized moving image (displayed on the large display 2 in FIG. 1). Generated content) at any time. The synthesizing unit 24 generates a subject image H extracted from the captured video W (indicated by a wavy line in the figure) as shown in FIG. The synthesized moving image frame as shown in FIGS. 15A and 15B is obtained by performing a process of combining so that the lower side Ga of the moving image frame of the 3D moving image G shown in FIG. Images G1, G2, etc. are generated. Further, the output unit 25 is connected to the third video line V3, and performs a process of outputting the synthesized moving image generated by the synthesizing unit 24 to the distribution device 40 through the third video line V3 as needed.

  FIG. 6 is a block diagram showing an internal configuration of the 3D moving image generating apparatus 30 (corresponding to moving image generating means). A general-purpose personal computer is applied to the 3D moving image generating apparatus 30 of the present embodiment. The 3D moving image generating apparatus 30 stores a plurality of moving image data D including moving image motion data (corresponding to posture information) obtained by the motion capture technology for each moving image frame, and a plurality of viewpoint information defining a position where the moving object is viewed. A table T is stored in advance, and based on the moving image data D and the viewpoint table T, a 3D moving image (see FIG. 14B) in which the viewpoint position with respect to the moving object is randomly changed is generated.

  The 3D moving image generating apparatus 30 includes a RAM 32, a ROM 33, a moving image output interface 34, an input / output interface 35, and a hard disk device 36 via an internal bus 30b in a control unit 31 (processor) that performs various control processes inside the computer main body 30a. Connected. The RAM 32 temporarily stores data, folders, and the like according to the processing of the control unit 31, and the ROM 33 stores in advance a program that defines basic processing contents performed by the control unit 31. The moving image output interface 34 is connected to the second video line V2, and performs processing to send the generated 3D moving image to the chroma key device 20 as needed. The input / output interface 35 is connected to the network cable L1 and receives a reproduction start signal, a reproduction end signal, and the like sent from the karaoke apparatus 41.

  The hard disk device 36 stores various programs, data, and the like. In the present embodiment, the system program 37 that defines a base process for operating the computer main body 30a is defined as a program, and a process related to generation of a 3D moving image is defined. A moving image generating program 38, moving image data D, a viewpoint table T, and the like are stored.

  The moving image data D stored in the hard disk device 36 is moving image content including three three-dimensional back dancers (back dancer images) as moving objects, and obtained by motion capture technology with markers attached to actual dancers. The content includes motion data created by defining the actual posture of the dancer in the three-dimensional coordinate system for each unit time (moving image frame) with the coordinates and angle of the marker. Such moving image data D specifies the position (viewing point) at which a moving object is viewed based on the viewpoint information included in the viewpoint table T when the 3D moving image is generated. A 3D video is generated. Note that the moving image data D of the present embodiment defines background information (stage background image position in the three-dimensional coordinate system) in addition to the posture information in the three-dimensional coordinate system regarding the three back dancers. Stuff).

  FIG. 7 schematically shows an image of a state related to generation of a 3D moving image. In the figure, three back dancers 15a to 15c and four stage backgrounds 16a to 16d (heart-shaped motif images) in an XYZ coordinate system composed of an X-axis, a Y-axis, and a Z-axis are represented as moving image data D. In the figure, a large number of cameras 17A, 17B and the like represent positions where the moving object is viewed in the three-dimensional coordinates XYZ defined by the viewpoint table T. Each camera 17A, 17B, etc. is provided with a UVW coordinate system which is a camera system coordinate different from the XYZ coordinate system, and each camera 17A, 17B of the present embodiment matches the imaging direction (viewpoint direction). The V axis to be directed is directed to the back dancer 15b located in the center. By selecting one of the plurality of cameras 17A, 17B, etc., the position where the moving body (back dancers 15a to 15c) is viewed in the XYZ coordinate system is determined, and the content captured by the determined camera is obtained. A 3D video is generated. Note that the number and positions of the cameras shown in FIG. 7 are examples, and the number and positions of the cameras can be set as appropriate according to the specifications.

  FIG. 8 shows the contents of the viewpoint table T. The viewpoint table T defines the coordinate values in the XYZ coordinates for each camera (the first camera 17A to the nth camera 17N), and the type (and coordinate value) of the camera in the table corresponds to the viewpoint information.

  The moving body (back dancer) included in the moving body data D will be described briefly. FIG. 9A shows one back dancer 15a (back dancer image) of three bodies created by three-dimensional computer graphics technology. As shown in FIG. 9B, the back dancer image is created by connecting a rod-like link member called bone B corresponding to the bone of the human body to a skin corresponding to the skin of the human body. . Furthermore, the points P1 to P17 attached to each part of the bone B shown in FIG. 9B correspond to the marker positions attached to the actual dancers, and there is a motion data value for each of these points P1 to P17. To do. In addition, the position and number of each point P1-P17 shown in FIG.9 (b) are examples, and can be suitably changed according to the position and number of the marker attached | subjected to an actual dancer.

  FIG. 10 is a diagram illustrating an image of each frame in a 3D moving image including a back dancer in which a three-dimensional state generated for each moving image frame (unit time) included in the moving image data D is specified. The 3D moving image is composed of moving image frames f1, f2, f3... At times t1, t2, t3..., And sequentially generates images corresponding to the moving image frames f1, f2, f3, etc. A moving image in which the back dancers 15a to 15c included in each of the moving image frames f1, f2, f3 and the like move is obtained. Note that the moving image data D of the present embodiment has 60 moving image frames per second (60 frames / second), but this value is merely an example, and the number of frames when the motion data is acquired. If it is within the range, it can be appropriately increased or decreased according to the required moving image quality. In FIG. 10, the stage background image included in the moving image data D is not shown.

  FIG. 11 schematically shows the contents of motion data M1a and M2a for one back dancer 15a in the first moving image frame f1 and the second moving image frame f2 at times t1 and t2 included in the moving image data D. . The motion data M1a and M2a have rotation angles and coordinate values for the X, Y, and Z axes in the XYZ coordinate system shown in FIG. 7 for each point P1 to P17 shown in FIG. 9B. Based on the motion data for each moving image frame, an image in which the postures of the back dancers 15a to 15c included in each moving image frame illustrated in FIG. 10 are specified is generated. In the image of FIG. 10, the viewpoint is specified with the central back dancer 15b viewed from the front. In FIG. 11, illustration of motion data relating to the remaining back dancers 15b and 15c and the stage backgrounds 16a to 16d is omitted, but the moving image data D naturally includes these motion data.

  Next, processing contents defined by the moving image generation program 38 will be described. The moving image generation program 38 defines the content of control processing performed by the control unit 31 and performs processing for randomly selecting a camera from the viewpoint table T in accordance with the input of the playback start signal from the karaoke apparatus 41. Specifically, if the viewpoint table T includes N cameras, random number generation processing is performed to generate a number corresponding to any one camera with a probability of 1 / N, and the generated random number is converted into a numerical value. Processing to select the corresponding camera is performed. For example, when “2” occurs in the random number generation process, the control unit 31 selects the second camera 17B from the viewpoint table T in FIG.

  The moving image generation program 38 defines that a 3D moving image is generated from the viewpoint of the selected camera. Note that the moving image generation program 38 of the present embodiment performs such camera selection processing every 8 seconds, and the generation of the 3D moving image and the camera selection processing are accompanied by the input of the reproduction end signal from the karaoke apparatus 41. To end.

  FIG. 12 is a first flowchart in which the flow of processing based on the moving image generation program 38 in the 3D moving image generation apparatus 30 is organized. Hereinafter, the processing procedure of the 3D moving image generating apparatus 30 will be described according to the first flowchart. First, the 3D moving image generating apparatus 30 determines whether or not a reproduction start signal is received from the karaoke apparatus 41 (S1). When the reproduction start signal is not received (S1: NO), the 3D moving image generating apparatus 30 waits for processing. Further, when the reproduction start signal is received (S1: YES), the 3D moving image generating apparatus 30 performs a process of randomly selecting one camera from a plurality of cameras (S2), referring to the viewpoint table T, A 3D moving image generation process is performed from the viewpoint of the selected camera (S3). Then, the 3D moving image generating apparatus 30 determines whether or not 8 seconds have passed (S4). If 8 seconds have not elapsed (S4: NO), the 3D moving image generating apparatus 30 returns to the 3D moving image generation processing step (S3), Thereafter, a 3D moving image is generated in which three back dancers perform a predetermined motion at a predetermined frame rate until 8 seconds elapse.

  If 8 seconds have elapsed (S4: YES), the 3D moving image generating apparatus 30 determines whether or not a playback end signal has been received from the karaoke apparatus 41 (S5), and if no playback end signal has been received (S5). : NO), the process returns to the stage (S2) of the camera selection process, and a new camera selection process is performed. By selecting this new camera, the viewpoint of the 3D moving image is automatically changed. Thereafter, steps S2 to S5 are repeated until a playback end signal is received, and when the playback end signal is received (S5: YES), the 3D moving image generating apparatus 30 ends the process.

  FIG. 13 is a time chart showing a viewpoint change state in a 3D moving image. Since the 3D moving image generating apparatus 30 performs the camera selection process as described above, in this embodiment, the camera is switched every 8 seconds from the start of the reproduction of the karaoke music. Accordingly, in the present invention, in addition to the angle of the 3D moving image G shown in FIG. 14B, a 3D image in which the positions of viewing the three-dimensional images of the back dancers 15a to 15c and the stage backgrounds 16a to 16d are switched every 8 seconds. Is generated by the 3D moving image generating apparatus 30.

  Finally, as shown in FIGS. 15A and 15B, three back dancers 15a to 15c and stage backgrounds 16a to 16c located behind the subject image H are obtained by combining processing in the chroma key device 20. It is possible to obtain frame images G1 and G2 of a composite image in which angles for viewing a 16d three-dimensional image are switched variously. Since the composite image composed of such frame images G1 and G2 is displayed on the large display 2, users around the user U who sings karaoke music can also enjoy watching the display contents on the large display 2. A sense of unity that can be enjoyed with the singing user U can be produced. FIG. 15A shows the case where the front sixth camera 17F is selected in FIG. 7, and FIG. 15B shows the case where the left second camera 17B is selected. .

  Further, FIG. 16 shows a frame image G10 of a combined moving image in which the telop 19 displayed on the sub display 4 is combined. The user U who sings a karaoke song will see the frame image G10 of this synthesized moving image. Therefore, since the user U appears in addition to the normal karaoke screen, the user U can confirm the swing according to the music, and the back dancers 15a to 15c and the stage backgrounds 16a to 16d that perform various realistic movements are simulated. You can enjoy the atmosphere of singing on stage. Furthermore, since the positions and angles at which the back dancers 15a to 15c and the stage backgrounds 16a to 16d are viewed are switched at random, the user U can enjoy a dynamic composition change every time like real camera work in a television program.

  Furthermore, in the karaoke system 1 of the present invention, a composite video as shown in FIG. 16 can be stored on a DVD in the storage device 44 and can be distributed over the network from the 3D video distribution server 45. ) Can be easily used for secondary use. As a result, the karaoke system 1 of the present invention can give the user a new enjoyment that has never existed, and can smoothly provide the content generated by singing karaoke, thereby providing a karaoke service. In addition to karaoke services, the present invention can also be used to secure new revenue sources for business entities, as well as various events such as wedding reunions and alumni associations, and auditions for discovering new singers. Can be used.

  The karaoke system 1 and the moving image generation system 10 of the first embodiment are not limited to the above-described contents, and various modifications can be applied. For example, the karaoke song acquisition method is not limited to the acquisition from the karaoke song server 5 through the network NW as shown in FIG. 1, but a storage medium (DVD, hard disk device, etc.) storing a large number of karaoke songs in the karaoke device 41. The karaoke device 41 may acquire the karaoke music by reading the karaoke music specified by the user from the storage medium.

  Further, when the user U who sings karaoke music can also confirm the display on the large display 2, the sub-display 4 may be omitted. In this case, the distribution device 40 can be omitted and a telop is synthesized on the large display 2. It is preferable to display the output moving image from the karaoke apparatus 41 so as to display the synthesized moving image. Further, when simplifying the specification, the storage device 44 and the 3D moving image distribution server 45 may be omitted. Furthermore, when the moving image generation system 10 is not applied to the karaoke system 1, the karaoke device 41 may be omitted and the processing may be limited to only the moving image.

  Furthermore, the time interval for selecting a camera at random by the 3D moving image generating apparatus 30 is not limited to 8 seconds (see step S4 in FIG. 12), and other times can be applied. Furthermore, the time interval for selecting the camera can also be changed randomly. For example, the first time interval is 3 seconds, the second time interval is 10 seconds, and the third time interval is 7 seconds. The content of the moving image may change at random time intervals so that the viewpoint changes randomly in time. In this case, it is necessary to determine the time interval after the camera selection process. For example, a random number is generated so that each second in the range of 1 to 15 seconds has a probability of 1/15. To determine the time.

  Further, the back dancers 15a to 15c corresponding to the moving objects can be appropriately increased or decreased in addition to the three bodies, and various forms such as animals and animated characters may be applied as moving objects in addition to the back dancers. Furthermore, it is possible to apply another stage background, and it is of course possible to omit it when simplifying the image content. In addition to the chroma key method, other methods may be applied as the method used for the composition processing.

  FIG. 17 schematically shows the data structure of a karaoke piece used in the process of the modification of the first embodiment. In this karaoke music piece, information for specifying a camera to be selected is attached in order of the musical piece progression at a predetermined time interval (for example, every 8 seconds) from the first (time 0) to the last (time Tn) of the music data. It has become. In addition, since the karaoke music creator of the data structure can determine the angle when the creator of the karaoke music is generated by the 3D moving image generating device 30, the creator of the karaoke music can use the karaoke music. There is an advantage that the angle of the displayed video content can be controlled. Further, in this modification, the karaoke piece having the above-described data structure is configured to be distributed from the karaoke piece server 5 shown in FIG.

  Furthermore, in this modified example, when the karaoke device 41 performs the karaoke song reproduction process, the karaoke device 41 performs processing to send the information of the attached camera as a camera designation signal to the 3D moving image generation device 30 in accordance with the song reproduction process. For example, when the karaoke apparatus 41 performs the reproduction process of the karaoke piece having the data structure shown in FIG. 17, the reproduction start signal and the camera designation signal for designating the fourth camera are sent to the 3D moving picture generating apparatus 30 at the start of reproduction of the piece of music. Then, a camera designation signal designating the sixth camera is sent to the 3D video generation device 30 after elapse of t10 seconds from the start of reproduction, and a camera designation signal designating the third camera is sent to the 3D video generation device 30 after elapse of t11 seconds from the start of reproduction. .

  FIG. 18 is a second flowchart showing a processing procedure performed by the 3D moving image generating apparatus 30 when the karaoke piece of the modification shown in FIG. 17 is used. A 3D moving image generation process performed by the modified 3D moving image generating apparatus 30 will be described with reference to the second flowchart. First, the 3D moving image generating apparatus 30 determines whether or not a reproduction start signal has been received from the karaoke apparatus 41 (S10). If no reproduction start signal has been received (S10: NO), the 3D moving image generating apparatus 30 waits for processing and receives a reproduction start signal. Is received (S10: YES), it is then determined whether a camera designation signal has been received from the karaoke apparatus 41 or not.

  When the camera designation signal is not received (S11: NO), the 3D moving image generating apparatus 30 waits for processing, and when the camera designation signal is received (S11: YES), the camera designated by the received camera designation signal is selected. Processing is performed (S12), and 3D moving image generation processing is performed from the viewpoint of the selected camera (S13). Then, the 3D moving image generating apparatus 30 determines whether or not a new camera designation signal has been received (S14). If a new camera designation signal is received (S14: YES), the process returns to the camera selection stage (S12). A camera designated by a new camera designation signal is selected.

  If a new camera designation signal has not been received (S14: NO), it is determined whether or not a playback end signal has been received from the karaoke apparatus 41 (S15). If a playback end signal has not been received (S15: NO) Returning to the 3D moving image generation stage (S13), a 3D moving image generation process is performed without changing the camera. When the playback end signal is received (S15: YES), the 3D moving image generating apparatus 30 ends the process.

  As described above, in the modification based on the karaoke music piece shown in FIG. 17, the 3D moving image generating apparatus 30 does not need to select a camera for determining the viewpoint at random, and the processing load can be reduced. In addition, since the karaoke song creation side can specify the viewpoint of the 3D movie part (back dancer image, background image, etc.) included in the synthesized movie to be displayed, the creation of movie content according to the intention of the creation side is realized. it can.

  FIG. 19 is a block diagram showing a main part of the moving image generating system 50 according to the second embodiment of the present invention. The moving image generating system 50 according to the second embodiment is characterized in that the viewpoint information of the 3D moving image is switched according to the position of the subject image in the video captured by the camera device 51. Therefore, one end of the first video line V1 of the camera device 51 is bifurcated to connect the first branch line V1a to the chroma key device 60, and the second branch line V1b is newly connected to the 3D moving image generating device 70. By connecting to the provided moving image input interface 77, the 3D moving image generating apparatus 70 acquires the captured video and can perform processing for detecting the position of the subject image included in the captured video. Note that the camera device 51 is fixed in a state in which it faces a predetermined imaging direction as in the first embodiment shown in FIG.

  In the 3D moving image generating apparatus 70 of the second embodiment, the moving image generating program 79 stored in the hard disk device 76 includes processing for detecting the position of the subject image, and the control unit 71 sets the contents defined by the moving image generating program 79. Based on this, the position detection process of the subject image is performed.

  FIG. 20A is a diagram for describing a subject image position detection process performed by the control unit 71 of the 3D moving image generating apparatus 70. When the control unit 71 acquires the captured video W1 of the camera device 51 by using the moving image input interface 77, the control unit 71 divides the entire range of the acquired captured video W1 into a lattice shape to form a total of 12 blocks B1 to B12. Then, the control unit 71 specifies a block having the largest area occupied by the subject image H in each of the blocks B1 to B12, and determines the specified block as a detected position (detection position) of the subject image H. In FIG. 20A, the block B11 having the largest area occupied by the subject image H is the detection position of the subject image H.

  FIG. 20B shows the contents of the position correspondence table 80 newly stored in the hard disk device 76 by the 3D moving image generating apparatus 70 of the second embodiment. The position correspondence table 80 has a content in which a plurality of types of cameras are associated in advance for each of the divided blocks B1 to B12 of the captured video W1. The association in the position correspondence table 80 is determined in consideration of the fact that the moving object (back dancer) and the background image are easy to see with respect to the position of the block occupied by the subject image H without overlapping. For example, when the subject image H occupies the right side (or lower right) block, the left side camera that is the opposite side or the upper left camera that is diagonally opposite is associated with the subject image H. When H occupies the left (or lower left) block, the right side camera that is the opposite side or the upper right side camera that is diagonally opposite is associated.

  The moving image generation program 79 stored in the hard disk device 76 defines a process for selecting a camera for defining a viewpoint using the position correspondence table 80 in FIG. Specifically, when the moving image generating program 79 identifies one block by the above-described detection processing of the subject image H, the control unit 71 performs processing for selecting a camera (viewpoint information) corresponding to the identified block from the position correspondence table 80. Stipulates what to do. Then, the moving image generation program 79 specifies that the control unit 71 specifies the position of the selected camera (the position of the XYZ coordinate system) based on the viewpoint table T (see FIG. 8) and performs the 3D moving image generation process. Yes.

  The other part (RAM 72, ROM 73, moving image output interface 74, input / output interface 75, etc.) in the 3D moving image generating apparatus 70 of the second embodiment and the system program 78, moving image data D, and viewpoint table T stored in the hard disk device 76 are This is equivalent to the first embodiment. In addition, since the camera device 51 and the chroma key device 60 of the moving image generation system 50 of the second embodiment are the same as those of the first embodiment, description thereof is omitted. Furthermore, the moving image generating system 50 of the second embodiment is also employed in the karaoke system 1 as shown in FIG. 1, and generates a combined moving image in which a user U who sings karaoke music is synthesized. Note that the distribution device 40, the karaoke device 41, and the like other than the moving image generation system 50 included in the karaoke system 1 are also equivalent to those in the first embodiment, and thus the description thereof is omitted, and the second embodiment is also equivalent to the first embodiment. The following description will be made using the reference numeral.

  FIG. 21 is a third flowchart illustrating 3D moving image generation processing in the 3D moving image generation apparatus 70 according to the second embodiment. First, the 3D moving image generating apparatus 70 determines whether or not a reproduction start signal has been received from the karaoke apparatus 41 (S20). If no reproduction start signal has been received (S20: NO), the process waits for the reproduction start signal. Is received (S20: YES), it is then determined whether or not there is an input of a captured video sent from the camera device 51 (S21).

  When there is no input of a captured video (S21: NO), the 3D moving image generating apparatus 70 waits for input of a captured video, and when there is an input of a captured video (S21: YES), as shown in FIG. A position detection process of the subject image H is performed (S22), a camera selection process corresponding to the detected position is performed using the position correspondence table 80 (S23), and a 3D moving image generation process is performed from the viewpoint of the selected camera. (S24). Then, the 3D moving image generating apparatus 70 determines whether or not a playback end signal has been received from the karaoke apparatus 41 (S25), and if it has not received a playback end signal (S25: NO), a stage of input determination processing for a captured video Returning to (S21), the processing is continued. When the playback end signal is received (S25: YES), the 3D moving image generating apparatus 70 ends the process.

  As described above, since the 3D moving image generating apparatus 70 according to the second embodiment generates a 3D moving image in which the viewpoint is changed according to the position of the subject image in the captured video, the synthesized moving image finally generated by the chroma key device 60 is generated. In the frame images G20, G21, etc., as shown in FIGS. 22A and 22B, the direction of viewing the images of the back dancers 15a to 15c and the stage backgrounds 16a to 16d changes every time the position of the subject image H changes. Become content.

  That is, in FIG. 22A, the subject image H is positioned on the left side in the frame image G20 due to the movement of the user U, so that the right end back dancer 15c is large when viewed from the right camera on the opposite side to the left side. The viewpoint has been changed so that On the other hand, in FIG. 22B, since the subject image H is positioned on the right side in the frame image G21 due to the movement of the user U, the left-end back dancer 15a is large from the viewpoint of the left camera on the opposite side to the right side. The viewpoint has been changed so that Therefore, the user U who sings karaoke music moves the position with respect to the camera device 51 that performs shooting, so that the viewpoints of the images of the back dancers 15a to 15c and the stage backgrounds 16a to 16d are switched, and the position is changed while singing. You can enjoy the visual enjoyment.

  In the second embodiment, the various modifications described in the first embodiment can be applied. FIGS. 23A and 23B show the processing contents of a modification specific to the second embodiment. This modification is characterized in that the viewpoint information of the 3D moving image is switched according to the operation of the subject image in the video shot by the camera device 51. For this reason, in the modified 3D moving image generating apparatus 70, the moving image generating program 79 stored in the hard disk device 76 includes processing for detecting the action of the photographed subject. The movement of the subject is detected based on the contents to be processed.

  The motion detection of the subject is performed by taking a difference between video frames that are temporally mixed in the captured video sequentially sent from the camera device 51. For example, FIG. 23A shows the content of the captured video W10 at time t20, and FIG. 23B shows the content of the captured video W11 at time t21 following time t20. The unit 71 detects a block in which the subject image H is located in each of the blocks B1 to B12 obtained by dividing the captured videos W10 and W11.

  Then, the control unit 71 determines whether or not there is a block where the subject H is located in the photographed video W10 at time t21 but not in the photographed video W10 at time t20 among the blocks where the detected subject image H is located. Judgment is made based on the difference for each block, and when there is such a block, it is determined that the subject has moved. For example, in the case of FIGS. 23 (a) and 23 (b), the subject image H is not located in the sixth block B6 in FIG. 23 (a), but in the sixth block B6 in FIG. 23 (b), the subject image H (user U The right hand) is located, the sixth block B6 is identified as a block having movement, and it is determined that the subject has moved.

  FIG. 24 shows the contents of the motion correspondence table 85 used in the modification relating to motion detection. The motion correspondence table 85 is newly stored in the hard disk device 76 by the 3D moving image generating apparatus 70 of the modification. is there. The operation correspondence table 85 has a content in which a plurality of types of cameras (corresponding to viewpoint information) are associated in advance for each of the blocks B1 to B12 that have operated. The correspondence in the motion correspondence table 85 is determined in consideration of the fact that the moving object (back dancer) and the background image are easy to see without overlapping with respect to the position of the block where the motion has occurred. Cameras may be associated so as to be linked to each other. For example, if a block on the right side of the block where the current subject H is located is newly detected by the operation of the subject H, a camera located on the right side of the camera in use may be associated with the subject H. If a block on the left side of the located block is newly detected, the camera on the left side of the camera in use may be associated. In the present invention, the blocks B1 to B12 described above correspond to the operation of the subject to be detected.

  In addition, the modified moving image generation program 79 stored in the hard disk device 76 defines a process for selecting a camera for determining a viewpoint based on the motion detection using the motion correspondence table 85 of FIG. Specifically, the moving image generation program 79 specifies a block in which an operation has been performed by the above-described operation detection process, and a process of selecting a camera (viewpoint information) corresponding to the specified block from the operation correspondence table 85.

  FIG. 25 is a fourth flowchart illustrating the processing contents performed by the 3D moving image generating apparatus 70 according to the modified example related to the motion detection described above. First, the 3D moving image generating apparatus 70 performs a determination process (S30) for receiving a reproduction start signal from the karaoke apparatus 41 and a determination process (S31) for inputting a photographed image sent from the camera apparatus 51, as shown in FIG. The same as the flowchart.

  When a captured video is input from the camera device 51 (S31: YES), the 3D moving image generating device 70 performs a subject motion detection process as shown in FIGS. 23A and 23B (S32). Then, it is determined whether or not there has been a change due to the operation (S33). When it is determined that there has been a change (S33: YES), the 3D moving image generating apparatus 70 uses the operation correspondence table 85 to perform a camera selection process corresponding to the block having the operation (S34). If it is determined that there is no change (S33: NO), camera selection processing corresponding to the block occupied by the subject image H is performed using the position correspondence table 80 of FIG. 20B (S35).

  Then, the 3D moving image generating apparatus 70 performs 3D moving image generation processing from the viewpoint of the selected camera (S36). Then, the 3D moving image generating apparatus 70 determines whether or not a playback end signal has been received from the karaoke apparatus 41 (S37), and if it has not received a playback end signal (S37: NO), a stage of input determination processing for a captured video Returning to (S31), the processing is continued. When the playback end signal is received (S37: YES), the 3D moving image generating apparatus 70 ends the process.

  As described above, the 3D moving image generating apparatus 70 according to the modified example of the second embodiment generates a 3D moving image in which the viewpoint is changed according to the motion of the subject, so that the synthesized moving image finally generated by the chroma key device 60 is also a karaoke piece. The direction of viewing the images of the back dancers 15a to 15c and the stage backgrounds 16a to 16d is switched according to the operation of the user U who sings. Therefore, the user U who sings karaoke music performs a swing motion while singing, so that the viewpoints of the images of the back dancers 15a to 15c and the stage backgrounds 16a to 16d in the synthesized video change, so that the swing motion In addition, each user can actively perform various swinging motions to change the viewpoint of the image and entertain the user who sees the composite video. Note that the fourth flowchart in FIG. 25 is a combination of processing related to motion detection and processing related to position detection (S34, S35), but only processing related to motion detection is performed and changes due to motion are performed. If there is no (S33: NO), a fixed viewpoint (default viewpoint) may be selected.

It is the schematic which shows the whole structure of the karaoke system to which the moving image production | generation system which concerns on 1st Embodiment of this invention is applied. It is a block diagram which shows the internal structure of a karaoke apparatus. It is a block diagram which shows the internal structure of a 3D moving image delivery server. It is the schematic which shows an example of the site page of a website. It is a block diagram which shows the internal structure of a chroma key apparatus. It is a block diagram which shows the internal structure of a 3D moving image production | generation apparatus. It is the schematic explaining the three-dimensional moving body in a tertiary coordinate system, the positional relationship of a viewpoint, etc. FIG. It is a chart which shows the contents of a viewpoint table. (A) Schematic diagram showing an image of a back dancer created by three-dimensional computer graphic technology, (b) is a schematic diagram showing points and bones corresponding to markers by motion capture technology. It is the schematic which shows the state of the back dancer contained in each moving image frame which comprises moving image data. It is a figure showing the content of the motion data corresponding to each moving image frame. It is a 1st flowchart which shows the processing method of the 3D moving image production | generation apparatus of 1st Embodiment. It is the time chart showing the camera selected with the reproduction progress of music. (A) is a schematic diagram showing an image of a subject image extracted from a captured video, and (b) is a schematic diagram showing a frame image of a 3D moving image generated by the 3D moving image generating device. (A) is the schematic which shows an example of the frame image which comprises the synthetic | combination moving image produced | generated, (b) is the schematic which shows the frame image of the synthetic | combination moving image from which a viewpoint differs from (a). It is the schematic which shows an example of the frame image of the synthetic | combination moving image with which the telop was synthesize | combined. It is a figure which shows the data structure of the karaoke musical piece used for the modification of 1st Embodiment. It is a 2nd flow chart which shows the processing method of 3D animation generation concerning the modification of a 1st embodiment. It is a block diagram which shows the structure of the moving image production | generation system which concerns on 2nd Embodiment of this invention. (A) is a figure of the picked-up image for demonstrating the process concerning the position detection of a to-be-photographed image, (b) is a table | surface which shows the content of a position corresponding | compatible table. It is a 3rd flow chart which shows the processing method of 3D animation generation concerning a 2nd embodiment. (A) is a schematic diagram showing an example of a frame image constituting a synthesized moving image, and (b) is a schematic diagram showing a frame image in a state where the viewpoint is changed as the subject image moves. FIG. 10A is a photographed video image for explaining processing related to motion detection according to a modification of the second embodiment, FIG. 10A is a diagram illustrating a front side state in continuous time, and FIG. It is a figure which shows the state of the rear side. It is a chart which shows the contents of an operation correspondence table. It is a 4th flow chart which shows the processing method of 3D animation generation concerning the modification of a 2nd embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Karaoke system 2 Large display 4 Sub display 5 Karaoke music server 6 Wall member 10 Movie generation system 11 Camera apparatus 15a-15c Back dancer 16a-16d Stage background 20 Chroma key apparatus 23 Subject image extraction part 24 Composition part 30 3D animation generation apparatus 38 Movie generation program 40 Distribution device 41 Karaoke device 41d Music playback processing unit 40f Telop synthesis unit 44 Storage device 45 3D movie distribution server 46 Content database 80 Position correspondence table 85 Operation correspondence table D Movie data T Viewpoint table H Subject image

Claims (12)

Movie generation means for generating a moving image including a moving object whose posture and viewpoint are specified based on posture information that defines a posture of the moving object in a three-dimensional coordinate system per unit time and viewpoint information that defines a position where the moving object is viewed In the video generation system provided,
Photographing means for photographing the subject;
Image extracting means for extracting a subject image included in the video imaged by the imaging means;
A moving image generating system comprising: a synthesized moving image generating unit configured to combine a subject image extracted by the image extracting unit with a moving image generated by the moving image generating unit to generate a combined moving image. Music acquisition means for acquiring music;
Music playback means for performing playback processing of the music acquired by the music acquisition means;
The moving image generating system according to claim 1, further comprising display processing means for performing display processing of the synthetic moving image in accordance with the reproduction processing of the music reproducing means. The music acquisition means is adapted to acquire a music accompanied by a character representing lyrics,
Character synthesis means for synthesizing the characters attached to the music into the synthesized video,
The moving image generation system according to claim 2, wherein the display processing unit performs display processing of a combined moving image in which characters are combined by the character combining unit. The music acquisition means is configured to acquire a music in which a plurality of viewpoint information is attached in the order of music progress,
4. The moving image generation system according to claim 2, wherein the moving image generation unit generates a moving image based on viewpoint information corresponding to a progress point of the reproduction process of the music reproduction unit. A means for randomly selecting one viewpoint information from a plurality of viewpoint information,
The moving image generating system according to any one of claims 1 to 3, wherein the moving image generating means generates a moving image based on selected viewpoint information. Image position detecting means for detecting the position of a subject image included in the video imaged by the imaging means;
A position correspondence table in which each position in the video and viewpoint information are associated with each other;
Means for selecting viewpoint information corresponding to the position detected by the image position detection means from the position correspondence table;
The moving image generation system according to any one of claims 1 to 3, wherein the moving image generation unit generates a moving image based on viewpoint information selected from the position correspondence table. An action detecting means for detecting an action of a subject image in a video taken by the photographing means;
An action correspondence table in which each action related to a subject image and a plurality of viewpoint information are respectively associated;
Means for selecting viewpoint information corresponding to the motion detected by the motion detection means from the motion correspondence table;
The moving image generating system according to any one of claims 1, 2, 3, and 6, wherein the moving image generating means generates a moving image based on viewpoint information selected from the operation correspondence table. In the posture information, the posture of a three-dimensional coordinate system related to a plurality of moving objects is defined,
The moving image generation system according to any one of claims 1 to 7, wherein the moving image generation unit generates a moving image including a plurality of moving objects based on the posture information. Means for storing background information defining the position of the background in the three-dimensional coordinate system;
The moving image generation system according to claim 1, wherein the moving image generation unit generates a moving image including a background based on the stored background information.   The moving image generation system according to any one of claims 1 to 9, further comprising a storage processing unit that performs a process of storing the combined moving image generated by the combined moving image generation unit in a storage medium. Receiving means for receiving a video request signal transmitted through the network;
A video transmission means for transmitting a video to a transmission source of the video request signal when the receiving means receives the video request signal;
The moving image generation system according to any one of claims 1 to 10, wherein the moving image transmission unit transmits the combined moving image generated by the synthetic moving image generation unit. The video generation system generates a video that includes a moving object whose posture and viewpoint are specified based on posture information that defines the posture of the moving object in the 3D coordinate system per unit time and viewpoint information that defines the position where the moving object is viewed. In the video generation method to
The video generation system includes:
Shoot the subject,
Extract the subject image included in the captured video,
A moving image generating method comprising combining the extracted subject image with the moving image to generate a combined moving image.

Images