JP2005333309A - Apparatus, method, system, and program for information processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To form a content having a video image and a voice based on user's performance. <P>SOLUTION: A movement detector 62 detects a user's movement from the video imaged by a camera 42, for example by a movement vector detection and tracking of the movement of an LED light grasped or mounted by the user and performed by various types of processes. A foot sensor input detector 66 receives the supply of a signal showing an operation input from a foot sensor 43. A video effect imparting processor 64 performs a predetermined effect on the video based on the detected result of the user's movement supplied from the movement detector 62, and the signal showing the operation input to the user's foot sensor 43 supplied from the foot sensor input detector 66. An effect sound generator 65 applies an effect sound to audio data of accompaniment formed by an accompaniment formation unit 67. This can be applied to a self-movie forming device. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information processing device, an information processing method, an information processing system, and a program, and in particular, provides an effect to a captured image based on information about a user's operation input during image capturing. The present invention relates to an information processing apparatus and information processing method, an information processing system, and a program that can generate audio data.

  As entertainment mainly based on virtual experiences, for example, there is LBE (Location Based Entertainment). LBE is, for example, recent high-density recording technology, virtual reality (virtual reality) technology that allows users to feel the real world created by computers, and virtual reality and the real world. Highly immersive, made possible by the development of mixed reality technology, which is a collective term for technologies that mix humans and make humans recognize them as a continuous 3D space with little discomfort. It is entertainment of the audience experience type. For example, in an amusement park, LBE has a video experience with a huge screen that allows viewers to be highly immersive, and experience-based content that allows the audience to become characters in the story. .

In addition, with recent development of information calculation technology, there is a technology that can process audio data and video data in substantially real time (for example, Non-Patent Document 1).
Shu Matsuda, DI PS: Real-time video processing objects for MAX, IPSJ, MUS36-9, 2000.

  However, LBE is not an entertainment that can be enjoyed over and over again because the story is scheduled and the interface is simplified so that anyone can participate regardless of age.

  In addition, content production by processing video data and audio data has become possible for many people due to the development of personal computers, etc., but depending on the technical contents, for example, expertise such as software programming Is often required. For example, when a content is to be created by highly editing video and audio, the required knowledge and technology are very advanced. On the other hand, if the content can be created by anyone with a simple interface, there is limited room for the user's originality to be included in the created content. The simpler the user interface, the less complex and simpler it becomes.

  Also, a special operation input method is used in the creation of so-called conventional interactive art in which content is created by measuring user gestures and mapping digital information based on the measurement results. In addition, it is difficult to say that usability is taken into consideration, such as being difficult to handle only by users who have deep knowledge of video and music. In addition, it is difficult to say that conventional interactive art takes into account the relationship between body movement and processing content and the relationship between acoustic processing and video processing in its expression technique.

  Users are looking for entertainment that can be actively enjoyed, including not only passive enjoyment but also self-expression. In response to these issues, the user can feel creativity and expressiveness while leaving the immersive feeling unique to conventional LBE content. Even a simple user interface can be enjoyed repeatedly, In consideration of the relationship between operation and processing content and the relationship between audio processing and video processing, it has been desired to provide a content creation device that can easily create original content.

  The present invention has been made in view of such a situation, and a user's movement (performance) is detected using a simple user interface that can be used sufficiently by a user who does not have special knowledge or skill, and sound is reproduced. By creating an image and adding an effect, it is possible to generate original content that has been subjected to advanced image processing and audio processing.

  An information processing apparatus according to the present invention includes an imaging unit that captures first video data, an operation information acquisition unit that acquires information about a user's operation, an operation input unit that acquires a user's operation input, and a first sound. The first video imaged by the imaging unit based on the voice generation unit that generates data, the information about the user's operation acquired by the operation information acquisition unit, and the user's operation input input by the operation input unit Video processing means for performing predetermined first processing on the data to generate second video data constituting the content, information on the user's motion acquired by the motion information acquisition means, and input by the operation input means The second audio data is generated by performing predetermined second processing on the first audio data generated by the audio generation unit based on the user's operation input. Audio processing means for generating voice data, display control means for controlling the display of the second video data generated by the video processing means, and audio for controlling the output of the second audio data generated by the audio processing means Output control means.

  The information processing apparatus is a space where a user can sufficiently operate, which is distinguished from the outside by a casing formed of a ceiling surface and a wall surface having an opening through which the user can enter and exit, and a floor portion installed at a lower portion of the housing. A plurality of operation input means are installed on the upper surface of the floor, and can be operated by being stepped on by a user's foot.

  The information processing apparatus may be a portable information processing apparatus that can be held by a user.

  Content output control means for controlling the output of the second video data generated by the video processing means and the second audio data generated by the audio processing means to another information processing apparatus Further, it can be provided.

  An illumination control means for controlling illumination can be further provided based on the information related to the user's action acquired by the action information acquiring means and the user's operation input input by the operation input means.

  In response to a user operation input input by the operation input means, a first process applied to the first video data by the video processing means and a second process applied to the first audio data by the audio processing means. This process may be a process having relevance.

  The information processing method of the present invention includes an audio generation step for generating first audio data, an imaging control step for controlling imaging of the first video data, an operation information acquisition step for acquiring information related to the user's operation, Imaging based on the operation input acquisition step for acquiring the user's operation input, information on the user's operation acquired by the process of the operation information acquisition step, and the user's operation input acquired by the process of the operation input acquisition step By performing a predetermined first process on the first video data whose imaging is controlled by the process of the control step and generating the second video data constituting the content, and the process of the operation information acquisition step Based on the acquired user's operation information and the user's operation input acquired by the operation input acquisition process The first audio data generated by the processing of the speech production step is subjected to a predetermined second processing, characterized in that it comprises a sound processing step of generating the second audio data constituting the content.

  The program according to the present invention includes a sound generation step for generating first sound data, an image pickup control step for controlling image pickup of the first video data, and an operation information acquisition step for acquiring information related to a user operation. Based on the operation input acquisition step for acquiring the user's operation input, information on the user's operation acquired by the process of the operation information acquisition step, and the user's operation input acquired by the process of the operation input acquisition step, A video processing step for performing a predetermined first process on the first video data whose imaging is controlled by the process of the imaging control step to generate the second video data constituting the content, and a process of the operation information acquisition step Information about the user's actions acquired by the user, and the user's An audio processing step of performing a predetermined second process on the first audio data generated by the process of the audio generation step based on the work input, and generating second audio data constituting the content. Processing is executed.

  In the information processing apparatus, the information processing method, and the program of the present invention, the first audio data is generated, the first video data is captured, information about the user's operation is acquired, and the user's operation input is acquired. The first video data is subjected to a predetermined first process based on the information related to the user's operation and the user's operation input, and the second video data constituting the content is generated. Based on the information related to the operation and the user's operation input, predetermined second processing is performed on the first audio data to generate second audio data constituting the content.

  The information processing system according to the present invention includes information including a first information processing apparatus that creates content and a second information processing apparatus that receives the content from the first information processing apparatus and distributes the content to the user. In the processing system, the first information processing apparatus includes an imaging unit that captures the first video data, an operation information acquisition unit that acquires information about a user's operation, and an operation input unit that acquires an operation input of the user. Imaging by the imaging means based on the voice generation means for generating the first voice data, the information on the user's motion acquired by the motion information acquisition means, and the user's operation input input by the operation input means A video processing unit that performs a predetermined first process on the first video data that is generated to generate second video data constituting the content, and an operation information acquisition unit. Based on the obtained information about the user's operation and the user's operation input input by the operation input means, the first sound data generated by the sound generation means is subjected to a predetermined second process, Audio processing means for generating second audio data constituting the content, display control means for controlling display of the second video data generated by the video processing means, and second audio generated by the audio processing means Content that is composed of audio output control means for controlling the output of data, second video data generated by the video processing means, and second audio data generated by the audio processing means, and second information And a second information processor that receives the content transmitted by the transmitter of the first information processor, and a receiver that receives the content transmitted by the transmitter of the first information processor. Storage means for storing the received content Ri, when receiving a content distribution request from a user, characterized in that it comprises a distribution control means for controlling the delivery to a user of the content stored by the storage means.

  The second information processing apparatus may be further provided with conversion means for converting the data format of the content corresponding to the type of the content distribution destination device whose distribution is controlled by the distribution control means. .

  The second information processing apparatus may be further provided with a billing processing unit that executes billing processing that occurs for the distribution of the content whose distribution is controlled by the distribution control unit.

  In the information processing system of the present invention, the first information processing apparatus generates the first audio data, the first video data is imaged, the information about the user's operation is acquired, and the user's operation input is acquired. The first video data is subjected to a predetermined first process based on the information related to the user's operation and the user's operation input, and the second video data constituting the content is generated. Based on the information related to the operation and the user's operation input, predetermined second processing is performed on the first audio data to generate second audio data constituting the content, and second information processing is performed. When the device receives the content transmitted from the first information processing device, stores the content, and receives a content distribution request from the user, the distribution of the stored content to the user is restricted. It is.

  According to the present invention, content composed of video and audio can be created. In particular, it is possible to acquire user performance and create video and audio based on it, and to apply effects to video and audio, so with a simple interface, original and advanced image processing and Content that has been subjected to audio processing can be generated.

  According to another aspect of the present invention, content composed of video and audio can be created and distributed to the user. In particular, after the user's performance is acquired, and the content generated by creating the video and audio based on the performance and the effect added to the video and audio is transmitted to the second information processing apparatus, Since it is delivered to the user, original and sophisticated content with advanced image processing and audio processing is generated with a simple interface, and the content is converted to data corresponding to the delivery destination device and delivered to the user. Or charging processing associated with content distribution can be performed.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, even if there is an embodiment that is described in the embodiment of the invention but is not described here as corresponding to the invention, the fact that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

  In the information processing apparatus according to claim 1 (for example, the content creation apparatus 2 according to FIGS. 3 and 9), an imaging unit that captures the first video data (for example, the camera 42 in FIGS. 2 and 3, or , The camera 301 in FIG. 9, operation information acquisition means (for example, the motion detection unit 62 in FIG. 3 or FIG. 9) that acquires information about the user's movement, and operation input means (for example, the user's operation input) The foot sensors 43-1 to 43-8 in FIG. 2 and FIG. 3 or the operation input unit 302 in FIG. 9 and voice generating means for generating first voice data (for example, accompaniment) (for example, FIG. 3 or The first accompaniment generation unit 67) of FIG. 9 and the information about the user's motion acquired by the motion information acquisition unit and the user's operation input input by the operation input unit are captured by the imaging unit. Image processing means (for example, as shown in FIG. 3 or FIG. 9) that performs predetermined first processing (for example, addition of an effect) on the image data to generate second image data constituting the content (for example, self-movie) Based on the information related to the user's motion acquired by the video effect applying processing unit 64) and the motion information acquiring unit, and the user's operation input input by the operation input unit, the first generated by the sound generating unit Audio processing means (for example, the sound effect of FIG. 3 or FIG. 9) that performs predetermined second processing (for example, addition of sound effects and effects) to the sound data to generate second sound data that constitutes the content Generating section 65), display control means for controlling the display of the second video data generated by the video processing means (for example, video display control section 69 in FIG. 3 or FIG. 9), audio processing The second sound output control means for controlling the output of the audio data (e.g., audio output control unit 70 of FIG. 3 or FIG. 9) generated by the means, characterized in that it comprises a.

  The information processing apparatus includes a casing (for example, the housing 21 in FIG. 2) having a ceiling surface and a wall surface having an opening through which a user can enter and exit, and a floor portion (for example, in FIG. It is possible to have a space that can be sufficiently operated by the user, which is distinguished from the outside by the floor 22), and the operation input means (for example, the foot sensors 43-1 to 43-8 in FIG. 2 and FIG. 3) A plurality of devices are installed on the upper surface of the device, and an operation input can be received by being stepped on by a user's foot.

  The information processing apparatus can be a portable information processing apparatus that can be held by a user (for example, the mobile phone 4, the PDA 5, a digital still camera, or a digital video camera).

  Another information processing apparatus (for example, the content distribution server 3 or the user) of the content composed of the second video data generated by the video processing unit and the second audio data generated by the audio processing unit Content output control means (for example, the network interface 71 of FIG. 3 or FIG. 9) for controlling the output to the portable telephone 4 or the PDA 5) possessed by the mobile phone can be further provided.

  Illumination control means (for example, the illumination control unit 63 in FIG. 3) for controlling illumination based on the information related to the user's motion acquired by the motion information acquisition means and the user's operation input input by the operation input means. Further, it can be provided.

  The information processing method according to claim 7 is an information processing method of an information processing apparatus (for example, the content creation apparatus 2 described in FIGS. 3 and 9) that creates content (for example, self-movie), and includes a first sound An audio generation step for generating data (for example, accompaniment) (for example, the process of step S4 in FIG. 7 or step S64 in FIG. 10) and an imaging control step for controlling the imaging of the first video data (for example, FIG. 7 step S5 or the process of step S65 of FIG. 10), an operation information acquisition step of acquiring information related to the user's action (for example, step S6 of FIG. 7 or the process of step S66 of FIG. 10), An operation input acquisition step (for example, step S8 in FIG. 7 or step S67 in FIG. 10) for acquiring a user operation input, and operation information acquisition Based on the information about the user's operation acquired by the processing of the step and the user's operation input acquired by the processing of the operation input acquisition step, the first video data whose imaging is controlled by the processing of the imaging control step A video processing step (for example, step S9 in FIG. 7 or step S68 in FIG. 10) for generating predetermined second processing (for example, adding an effect) and generating second video data constituting the content ) And information on the user's motion acquired by the processing of the motion information acquisition step, and the user's operation input acquired by the processing of the operation input acquisition step, the first generated by the processing of the voice generation step A second process (for example, addition of sound effects and effects) is applied to the audio data of the first audio data to form the content. Audio processing step (e.g., step S10 in FIG. 7, or the process of step S69 of FIG. 10) for generating audio data, characterized in that it comprises a.

  Also in the program according to claim 8, the embodiment (however, an example) to which each step corresponds is the same as the information processing method according to claim 7.

  The information processing system according to claim 9 includes a first information processing apparatus (for example, the content creation apparatus 2 described in FIGS. 3 and 9) that creates content (for example, a self-movie), and a first information processing. An information processing system including a second information processing apparatus (for example, the content distribution server 3 described in FIG. 6) that receives content from an apparatus and distributes the content to a user. Imaging means for capturing the first video data (for example, the camera 42 in FIG. 2, FIG. 3 or the camera 301 in FIG. 9) and the operation information acquiring means for acquiring information about the user's motion (for example, FIG. 3 or the motion detection unit 62 in FIG. 9 and an operation input means for acquiring a user's operation input (for example, the foot sensors 43-1 to 43-8 in FIGS. 2 and 3 or the operation input in FIG. 9). 302), voice generation means for generating first voice data (for example, accompaniment) (for example, accompaniment generation section 67 in FIG. 3 or FIG. 9), information on the user's action acquired by the action information acquisition means, Then, based on the user's operation input input by the operation input means, predetermined first processing (for example, addition of an effect) is performed on the first video data imaged by the imaging means to constitute the content. Information about the user's motion acquired by the video processing means (for example, the video effect imparting processing unit 64 in FIG. 3 or FIG. 9) for generating the second video data, the motion information acquisition means, and input by the operation input means Based on the user's operation input, the first sound data generated by the sound generating means is subjected to predetermined second processing (for example, addition of sound effects and effects). Audio processing means for generating the second audio data constituting the content (for example, the sound effect generating unit 65 in FIG. 3 or FIG. 9) and a display for controlling the display of the second video data generated by the video processing means Control means (for example, the video display control unit 69 in FIG. 3 or FIG. 9) and sound output control means (for example, the sound in FIG. 3 or FIG. 9) for controlling the output of the second sound data generated by the sound processing means. Output control unit 70), the content composed of the second video data generated by the video processing means and the second audio data generated by the audio processing means, is transmitted to the second information processing apparatus. And the second information processing apparatus receives the content transmitted by the transmission means of the first information processing apparatus. Receiving means (for example, the network interface 220 in FIG. 6), storage means for storing the content received by the receiving means (for example, the HDD 218 in FIG. 6), and storage means when receiving a content distribution request from the user Distribution control means (for example, the CPU 211 in FIG. 6) for controlling distribution of the content stored by the user to the user.

  The second information processing apparatus converts the data format of the content in accordance with the type of content distribution destination device whose distribution is controlled by the distribution control unit (for example, the image processing unit 241 in FIG. An audio processing unit 242) can be further provided.

  The second information processing apparatus can further include a billing processing unit (for example, a billing processing unit 243 in FIG. 6) that executes billing processing that occurs for the distribution of content whose distribution is controlled by the distribution control unit. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing a system configuration of a self-movie distribution system to which the present invention is applied.

  For example, the network 1 which is a wide area network including the Internet and a telephone line network includes a self-movie creation device 2-1 to 2-n (where n is a positive integer), a self-movie distribution server 3, and a mobile phone 4 A PDA (Personal Digital Assistant) 5, a personal computer 6, and a home server 7 are connected to the home server 7, and a television receiver 8 is connected.

  The self-movie creation devices 2-1 to 2-n have acquired the user's performance (motion and operation inputs to various sensors in the device) and acquired the user's video by the process described later. Create self-movies that can be distributed to users by generating audio data based on performance, applying predetermined effects to captured images, and controlling lighting as necessary. The created self-movie is transmitted to the self-movie creating server 3 via the network 1.

  The self-movie distribution server 3 stores the self-movie transmitted from the self-movie creating apparatuses 2-1 to 2-n, and when the self-movie distribution (download) is requested by the user, the self-movie distribution server 3 Via the mobile phone 4, the PDA 5, the personal computer 6, and the home server 7, and charging processing for the distributed self-movie is executed.

  The mobile phone 4, PDA 5, personal computer 6, or home server 7 has a user who has created a self-movie in the self-movie distribution server 3 in any of the self-movie creation devices 2-1 to 2 -n. It is used when accessing via the network 1 and requesting download of a self-movie.

  The user accesses the self-movie distribution server 3 via the network 1 using, for example, the mobile phone 4, the PDA 5, the personal computer 6, or the home server 7, and the self-movie creation devices 2-1 to 2-2. Request download of self-movie created in any of n. Then, the user can view the downloaded self-movie using the mobile phone 4, the PDA 5, the personal computer 6, or the television receiver 8 connected to the home server 7, and further the mobile phone. Using a portable telephone 4, a PDA 5, a personal computer 6 and the like, for example, using a device (such as a portable telephone, a PDA, or a personal computer, not shown) that is owned by a friend using e-mail or wireless communication, Self-movies can also be exchanged.

  Hereinafter, when it is not necessary to individually distinguish the self-movie creating apparatuses 2-1 to 2-n, they are simply referred to as the self-movie creating apparatus 2.

  FIG. 2 is a diagram for explaining an external configuration of the self-movie creating apparatus 2 according to the first embodiment.

  The self-movie creation device 2 is moved by a user to create a self-movie by a cylindrical wall 21 having an opening serving as an entrance for the user to enter and a casing 21 made of a ceiling and a circular floor 22. A possible space (for example, a cylindrical space having a diameter of 1800 mm and a height of 3000 mm, in which a person with a height of 180 cm can horizontally spread his hand or jump, can be configured).

  An operation input unit 41 provided with keys, buttons, touch pads, or the like for a user to perform operation input is provided in front of the opening of the housing 21. In addition, the operation input unit 41 is provided with a receiving unit 51 that can receive wireless information such as infrared rays. In addition to performing operation input using the keys, buttons, touch pad, or the like of the operation input unit 41, the user uses, for example, the portable phone 4 or the PDA 5 capable of wireless communication by infrared communication or Bluetooth, It is possible to supply desired information to the self-movie creating apparatus 2. For example, before instructing the start of self-movie creation (hereinafter referred to as play start), the user inputs a desired text using the mobile phone 4 or the like, and performs wireless communication such as infrared communication. And can be transmitted to the self-movie creating apparatus 2 in advance. Then, the self-movie creating apparatus 2 can receive the play start command from the user and can display the received text superimposed on the captured image by a process described later.

  A camera 42 that images the user is provided above the vertical plane of the operation input unit 41. The height of the center position of the camera 42 is preferably about 1400 mm so that a person with a height of 140 cm to 180 cm can enter the frame of the image captured by the camera 42. The video imaged by the camera 42 can be used not only as a material for a self-movie but also for detecting a user's movement. The detection result of the detected user movement is used as a parameter of a process for applying an effect (effect) to video or audio or adding a sound effect to the audio by a process described later.

  In order to detect a user's movement using the video imaged by the camera 42, for example, the user operates with an LED light or wearing it, analyzes the captured video data, The movement of the LED light may be tracked, or the user may operate without wearing anything, and the captured image is subjected to image processing to obtain a motion vector, and the magnitude of the motion vector For example, a predetermined position such as the tip of the user's hand or the apex of the head may be detected and the amount of movement may be tracked.

  Further, in order to detect the movement of the user, instead of using the video data captured by the camera 42, for example, by providing a plurality of infrared sensors and photo sensors inside the housing 21, based on the sensor input. The user's body movement may be detected, or the user may be equipped with a sensor that can directly detect the user's own movement, and the user's movement is detected based on the sensor input. It may be. When a plurality of infrared sensors or the like are provided in the housing 21 to detect the user's movement, the space in which the user can move is a columnar shape, and the sensors are located near the center of the space. Since the assumed user distances are equal, it is suitable for detecting the user's movement.

  In addition, in order to detect a user's motion, when a motion vector is detected from the captured image or a sensor input such as an infrared sensor provided in the housing 21 is used, the user has no restrictions. However, there is a case where the movement of the user is shrunken if there is no restriction. On the other hand, when the user operates with the LED light attached or attached, the user is conscious of the operation of swinging the LED light toward the camera 42, and the user's own operation and video, There is an advantage that it is easy to recognize the relationship between the effect applied to the sound or the sound effect added to the sound.

  In other words, when the user has an LED light or is put on the device to operate, the user can “draw a picture with a light brush, play a sound” rather than just dancing freely. It is possible to have a simple image that is simpler and more effective than when it is allowed to operate without any restrictions. Furthermore, since the video imaged by the camera 42 is a material for a self-movie, the created video image is rich in change due to the presence of LED light in the video image. Also occurs.

  In addition, since LED radiates | emits a fixed light quantity omnidirectionally, there also exists an advantage that the position in the acquired image is easy to detect. In addition, in the captured image, when the information on which the position of the LED light is to be tracked is focused on the luminance information emitted from the LED rather than the hue (RGB) of the LED, the color of the LED light Since the position can be detected without changing the detection method even if is changed, for example, a plurality of color LED lights are prepared, and the user desires a desired color LED light, or It may be possible to attach it.

  The floor 22 is provided with foot sensors 43-1 to 43-8. When the foot sensors 43-1 to 43-8 are stepped on by the user, that is, when an operation input is received, a predetermined effect is applied to the captured video or the created audio, or a predetermined effect is applied to the created audio. Sound is given. For example, different effects and sound effects may be assigned to each of the foot sensors 43-1 to 43-8, or the same effect may be assigned to some of the foot sensors 43-1 to 43-8. Alternatively, the same sound effect may be assigned. Also, the type of effect or sound effect assigned to the foot sensors 43-1 to 43-8 is randomly changed for each user's self-movie creation process (hereinafter referred to as play). Therefore, even if the user is playing many times in such a way that the user cannot easily learn the correspondence between the foot sensors 43-1 to 43-8 and the effects or sound effects, it is surprising. A self-movie may be created. Alternatively, the user may be able to set the type of effect or sound effect assigned to the foot sensors 43-1 to 43-8. Then, as described above, based on the detection result of the user's motion detected by the video captured by the camera 42, the effect intensity and the sound effect parameters related to the captured video and the created audio Etc. are controlled. Detailed examples of effects and sound effects will be described later.

  In FIG. 2, it has been described that eight foot sensors 43-1 to 43-8 are provided, but it goes without saying that the number of foot sensors may be any number other than eight. Alternatively, a touch pad or the like may be prepared inside the housing 21 so that an operation input can be received instead of the foot sensor.

  In addition, a mirror capable of projecting the user's whole body is installed inside the housing 21 (however, a position that does not interfere with the imaging process by the camera 42, the foot sensor, and other operations) so that the user can check his / her own performance. You may make it do.

  A monitor 44 is provided in front of the opening of the housing 21. The monitor 44 displays an image created by applying an effect to the image captured by the camera 42 based on the operation input to the foot sensors 43-1 to 43-8 and the detection result of the user's movement. It is made like that.

  In addition, inside the housing 21, a speaker 45-that outputs a sound to which an effect is applied and an effect sound is added according to an operation input to the foot sensors 43-1 to 43-8 and a detection result of a user's movement. 1 to 45-6 are provided. The speakers 45-1 to 45-6 are adapted to be able to reproduce sound by the latest Dolby Digital method, the speaker 45-1 being a center speaker in a 20 Hz to 20 kHz band, and the speaker 45 being a front left and right speaker. -2 and a speaker 45-3, a speaker 45-4 and a speaker 45-5 which are rear left and right speakers, and a speaker 45-6 which is a subwoofer dedicated to a low frequency of 120 Hz or less. In addition, in FIG. 2, although demonstrated as what can output an audio | voice with the speakers 45-1 thru | or 45-6, the number of speakers may be any number other than six, Furthermore, it goes without saying that any reproduction system other than Dolby Digital may be used.

  Also, illuminations 47-1 to 47-5 are provided on the upper surface (near the ceiling surface) inside the housing 21. The illuminations 47-1 to 47-5 may be configured to illuminate the inside of the housing 21 with a constant brightness, and the operation input to the foot sensors 43-1 to 43-8 and the detection result of the user's movement Thus, control such as changing the color of the illumination or blinking may be performed. In addition, in order to detect the movement of the user, when the user is provided with an LED light and the movement thereof is detected, the brightness of the illumination by the illuminations 47-1 to 47-5 is determined in the captured image. Therefore, it is necessary to control the LED light so as not to interfere with the detection of luminance.

  Next, FIG. 3 is a block diagram showing the internal configuration of the self-movie creating apparatus 2 according to the first embodiment described with reference to FIG.

  The self-movie creating apparatus 2 uses a performance section for acquiring a user's performance, real-time effects for generating video data to which a predetermined effect is applied and audio data to which an effect or a sound effect is applied, using the acquired performance. The section is roughly divided into three sections: a rendering section for rendering video data and audio data to which a predetermined effect is applied.

  Here, a description will be given assuming that the user's movement is detected by, for example, motion vector detection or tracking of movement of the LED light held or attached by the user based on the video imaged by the camera 42.

  The operation input unit 41 outputs a control signal indicating the start of play to the image processing unit 61, the video effect applying processing unit 64, the foot sensor input detection unit 66, and the accompaniment generation unit 67 when play start is instructed by the user. To supply. In addition, the operation input unit 41 can receive input of information (for example, a user ID) necessary for uniquely distinguishing the created self-movie as necessary.

  The receiving unit 51 receives text data to be displayed on the self-movie transmitted from the mobile phone 4 or PDA 5 owned by the user and supplies the text data to the video effect applying processing unit 64.

  The video imaged by the camera 42 is supplied to the image processing unit 61. When receiving a control signal indicating the start of play, the image processing unit 61 performs various signal processing (image correction processing) such as A / D conversion, gamma correction, and white balance adjustment on the supplied video. Then, the processed video data is supplied to the motion detector 62 and the video effect applying processor 64.

  The motion detection unit 62 detects the user's motion from the captured image by, for example, motion vector detection or tracking of the movement of the LED light held or attached by the user, and the detection result is displayed as the illumination control unit 63, This is supplied to the video effect applying processing unit 64 and the sound effect generating unit 65.

  For example, when tracking the movement of the LED light held or attached by the user, the motion detection unit 62 has a luminance equal to or higher than a predetermined threshold on the XY plane with the center point of the captured image as coordinates (0, 0). The position of the LED light can be recognized based on the position of the coordinates having the signal intensity, and the change amount of the position of the LED light in a predetermined unit time can be detected as the amount of movement of the user.

  In addition, when receiving a control signal indicating the start of play, the foot sensor input detection unit 66 receives a signal indicating an operation input from the foot sensors 43-1 to 43-8, and receives the foot sensor 43-1 to 43-8. A signal indicating which one of the operation inputs 43-8 has received the operation input is supplied to the illumination control unit 63, the video effect applying processing unit 64, and the sound effect generating unit 65.

  When the accompaniment generator 67 receives a control signal indicating the start of play, the accompaniment generator 67 loops a predetermined chord progression dance music composed of three tracks such as rhythm, bass, and melody based on 8 bars. Then, an accompaniment is generated and supplied to the sound effect generator 65. The accompaniment generation unit 67 indicates, for example, an accompaniment chord currently being played, a current tension value, a currently playing section, or the like necessary for generating sound effects and adding effects to audio data. Information is supplied to the sound effect generator 65.

  In generating music, if the pitch is determined at random, a dissonant sound will be generated, so an accompaniment must be generated based on the chord progression rule. The accompaniment generation unit 67 can generate a different piece of music (music score generation) each time using a Stochastic Markov model in which a random fraction is controlled by a Markov chain based on a predetermined chord progression rule.

  The illumination control unit 63 indicates the detection result of the user's motion supplied from the motion detection unit 62 and the operation input to the user's foot sensors 43-1 to 43-8 supplied from the foot sensor input detection unit 66. Based on the signal, the lights 47-1 to 47-5 are controlled. As an example of the association between the detection result of the user's motion and the operation input to the user's foot sensors 43-1 to 43-8 and the control of the lights 47-1 to 47-5, for example, the detected motion Depending on the amount of movement or the amount of change in the detected movement, the illumination intensity or blinking speed may be changed, or it will light up according to the position of the operated foot sensor. You may change the kind of illumination to make.

  The video effect provision processing unit 64 detects the user motion detection result supplied from the motion detection unit 62 and the operation input to the user foot sensors 43-1 to 43-8 supplied from the foot sensor input detection unit 66. Video data supplied from the image processing unit 61 is superimposed on the video data supplied from the receiving unit 51 as necessary, and a predetermined effect is applied to the video data supplied as a result. Is supplied to the self-movie storage unit 68.

  With reference to FIG. 4 and FIG. 5, the application of the effect to the video will be described.

  FIG. 4 shows an image 91 that is one frame of the captured video and images 101 to 108 to which a single effect is applied.

  Here, independent effects are associated with each of the foot sensors 43-1 to 43-8, and the intensity of the effect is determined by the amount of motion of the user detected by the motion detector 62. It will be described as being done. Further, the intensity of the effect may be linearly changed with respect to the user's motion amount detected by the motion detection unit 62, or the user's motion amount detected by the motion detection unit 62 may be set to a predetermined quantum. The quantization step may be used to quantize and relate to the effect intensity. For users, it is better to roughly quantize the motion amount detection value and associate it with the effect strength than to change the effect strength linearly with respect to the motion amount detection value. There is an advantage that it is easy to recognize the relationship between and the effect strength.

  For example, when the foot sensor 43-1 is operated, the video effect provision processing unit 64 converts the brightness and contrast of the captured image 91 to generate the image 101. At this time, the detected amount of user movement is linked to the brightness and contrast values. For example, the greater the detected amount of motion, the higher the brightness in the image and the higher the degree of contrast. It is made to do.

  When the foot sensor 43-2 is operated, the video effect imparting processing unit 64 divides the captured image 91 into a plurality of cells, and remaps each cell, thereby providing an effect like a jigsaw puzzle. A given image 102 is generated. At this time, among the detected user movements, the horizontal movement is associated with the amount of change in cell remapping, and the vertical movement is associated with the speed of cell position change. For example, the greater the amount of motion detected, the greater the speed and amount of change in cell position.

  When the foot sensor 43-3 is operated, the video effect imparting processing unit 64 divides the captured image 91 into a predetermined number of layers (four in FIG. 4) and moves the divided layers. Then, the image 103 to which the alternation effect is given is generated. At this time, the detected amount of movement of the user and the amount of movement of each layer are linked, for example, the amount of movement of the layer increases as the amount of detected movement increases. .

  When the foot sensor 43-4 is operated, the video effect imparting processing unit 64 divides the captured image 91 into a predetermined number of images and generates an image 104. At this time, the detected amount of user motion and the number of divisions of the image (the number of divisions is a power value) are linked. For example, the larger the amount of detected motion, the larger the number of divisions of the image. Has been made.

  In addition, when the foot sensor 43-5 is operated, the video effect imparting processing unit 64 reduces the captured image 91 to a predetermined scale and rotates the image 91 to a predetermined angle, thereby giving a kaleidoscopic effect. The generated image 105 is generated. At this time, among the detected user movements, the horizontal movement is associated with the rotation angle, and the vertical movement is assigned to the value of the scale. For example, the detected movement amount The larger the is, the smaller the scale and the faster the rotation speed.

  When the foot sensor 43-6 is operated, the video effect imparting processing unit 64 performs a predetermined delay process on a part of the image data of the captured image 91 to generate a so-called ghost. The image 106 is generated. At this time, the detected amount of movement of the user and the length of the delay time are linked. For example, the larger the amount of detected motion, the longer the delay time, and an image in which the ghost can be clearly observed.

  When the foot sensor 43-7 is operated, the video effect imparting processing unit 64 distorts the image by multiplying the image data of the captured image 91 by a sine wave, and generates the image 107. At this time, the detected amount of movement of the user and the magnitude of the amplitude of the sine wave to be multiplied are linked, for example, the larger the amount of detected motion, the larger the amplitude of the sine wave, The amount of distortion is increased.

  When the foot sensor 43-8 is operated, the video effect imparting processing unit 64 detects an edge (contour) of the captured image 91, changes the hue, and generates the image 108. At this time, the detected amount of movement of the user and the amount of change in hue are linked. For example, the amount of change in hue increases as the amount of movement generated increases, and the distortion of the edge increases. Has been made.

  In addition, when a plurality of foot sensors among the foot sensors 43-1 to 43-8 are operated simultaneously, the video effect imparting processing unit 64 performs parallel processing and applies the effect repeatedly, as shown in FIG. Such an image can be generated.

  For example, FIG. 5A includes a foot sensor 43-4 associated with the effect that generates the image 104 described with reference to FIG. 4 and a foot sensor 43-5 associated with the effect that generates the image 105. It is the image 111 produced | generated when operated simultaneously. In the image 111, the horizontal movement of the detected amount of movement of the user is associated with the rotation angle, and the vertical movement is assigned to the scale value. Further, the image is generated by dividing the image into the number of divisions based on the detected user movement.

  5B includes a foot sensor 43-2 associated with the effect for generating the image 102 described with reference to FIG. 4 and a foot sensor 43-4 associated with the effect for generating the image 104. It is the image 112 produced | generated when operated simultaneously. In the image 112, with respect to the captured image 91, among the detected user movements, the movement in the horizontal direction is associated with the amount of change in cell remapping, and the movement in the vertical direction represents a change in the position of the cell. Further, the image is divided into the number of divisions based on the detected user movement, and is generated.

  In FIGS. 4 and 5, for the sake of explanation, an example has been described in which the user's hand wearing the LED light is in front of the camera 42, but the image captured during play is usually It goes without saying that many parts of the user's body are included, including the user's head, upper body, and the like.

  Returning to the description of FIG.

  The sound effect generation unit 65 receives the detection result of the user's movement supplied from the movement detection unit 62 and the operation input to the user's foot sensors 43-1 to 43-8 supplied from the foot sensor input detection unit 66. On the basis of the signal shown, the accompaniment sound data supplied from the accompaniment generator 67 is subjected to a predetermined effect or, for example, a sound effect including a sound of an instrument is given, and the resulting sound data is The self-movie storage unit 68 is supplied. The sound effect generator 65 obtains information indicating an accompaniment chord currently being played, a current tension variable, a currently playing section, or the like from the accompaniment generator 67, and generates a sound effect based on the information. Or execute effects.

  Here, regarding the sound effect generation, the foot sensors 43-1 to 43-8 are associated with the addition of independent sound effects, and the sound effect parameters and the like are detected by the motion detection unit 62. It is assumed that it is determined based on the amount of movement. In addition, various parameters may be linearly changed with respect to the user motion amount detected by the motion detection unit 62, or the user motion amount detected by the motion detection unit 62 may be quantized to a predetermined value. Quantization may be performed in steps and associated with various parameters.

  For example, when the foot sensor 43-1 is operated, the sound effect generating unit 65 adds noise to the supplied accompaniment. At that time, the detected amount of movement of the user is associated with the frequency of the bandpass filter of white noise. For example, when the amount of detected movement of the user per unit time increases, the cutoff frequency of the filter increases, and the movement When the detection amount decreases, the cutoff frequency of the filter gradually returns to the initial value.

  Further, for example, when the foot sensor 43-2 is operated, the sound effect generating unit 65 adds a predetermined sample sound to the supplied accompaniment. At that time, the detected magnitude of the horizontal movement of the user is associated with the reproduction position in the sample, and the speed of the movement is associated with the reproduction speed of the sample. Therefore, the user can play the sample back and forth like a record by shaking the hand holding or wearing the LED light to the left and right, and further control the playback speed according to the speed of the left and right movements of the hand. be able to. The given sample is randomly loaded from a sample collection stored in advance.

  For example, when the foot sensor 43-3 is operated, the sound effect generating unit 65 is based on the detected magnitude of the lateral movement of the user based on a pattern having a predetermined number of steps (for example, 16 steps). Then, any pitch in the scales defined by the currently output accompaniment chord is selected and added to the accompaniment for output. Therefore, the user virtually plays a melody that always rides the chord progression (does not become dissonant with accompaniment) by swinging the hand holding or wearing the LED light to the left or right (virtually playing the instrument) )be able to.

  For example, when the foot sensor 43-4 is operated, the sound effect generating unit 65 performs the scale of the scale defined by the currently output accompaniment chord based on the detected magnitude of the lateral movement of the user. One of the pitches is selected, and a timbre (FM amount) is selected based on the detected magnitude of the vertical movement of the user, and is added to the accompaniment and output. Therefore, the user can synthesize a melody always on the chord progression (which does not become dissonant with accompaniment) with a tone changed by the vertical movement of the hand by swinging the hand holding or wearing the LED light to the left or right. Performance (virtual instrument playing).

  Also, for example, when the foot sensor 43-5 is operated, the sound effect generating unit 65 adds a sound like a scratched record to the accompaniment by outputting the filtered white noise in a constantly changing pattern. And output. The cutoff of the filter is controlled according to the detected magnitude of the lateral movement of the user.

  For example, when the foot sensor 43-6 is operated, the sound effect generating unit 65 is based on the detected magnitude of the lateral movement of the user based on a pattern having a predetermined number of steps (for example, 16 steps). Select one of the scales defined by the currently output accompaniment chord, and change the sound appearance density with respect to the time axis based on the detected vertical movement of the user And add it to the accompaniment for output. Therefore, the user can always play the chord progression (does not become dissonant with the accompaniment) by swinging the hand holding or wearing the LED light left and right, with a performance density that is changed by the vertical movement of the hand, You can play (virtually play a musical instrument).

  For example, when the foot sensor 43-7 is operated, the sound effect generating unit 65 adds a sample voice of a singing voice fragment (one or two human voices) to the supplied accompaniment. At that time, based on the detected magnitude of the lateral movement of the user, one of the musical scales defined by the currently output accompaniment chord is selected. Therefore, the user can add a melody singing voice that always rides the chord progression (does not become dissonant with the accompaniment) by shaking the hand holding or wearing the LED light to the left or right. Note that the singing voice samples to be given are randomly loaded from a sample collection stored in advance.

  For example, when the foot sensor 43-8 is operated, the sound effect generation unit 65 changes the current tension variable of the accompaniment supplied from the accompaniment generation unit 67 based on the detected magnitude of user movement. . The tension variable controls the range of the range of multiple musical instruments that make up the accompaniment.For example, if the amount of motion detected per unit time of the user increases, the range of the accompaniment increases and the generated sound becomes When it becomes brighter and the amount of motion detection decreases, the accompaniment sound range becomes narrower and the generated sound has a calm atmosphere.

  In addition, here, by adding sound created based on the user's operation of the foot sensors 43-1 to 43-8 and the detected value of the amount of movement of the user to the automatically created accompaniment, the user can perform the performance. Can give a predetermined sound effect according to the accompaniment or perform a virtual performance. For example, an automatically created accompaniment or a randomly selected pre- Various effect processes may be performed on the stored sampling sound source based on the operation of the foot sensors 43-1 to 43-8 by the user and the detected value of the amount of movement of the user.

  Similarly to the case of image processing, when a plurality of foot sensors are operated at the same time, sound effects given by operation inputs of the respective foot sensors may be output simultaneously.

  In addition, the application of the effect to the captured image and the sound effect applied to the accompaniment that are executed when a predetermined foot sensor is operated may be related to each other. For example, according to the above-described example, when the foot sensor 43-5 is operated, the video effect imparting processing unit 64 reduces the captured image 91 to a predetermined scale, rotates it to a predetermined angle, The sound effect generator 65 adds a sound like a scratched record to the accompaniment by generating the filtered white noise in a constantly changing pattern. Output. That is, when the user operates the foot sensor 43-5, the user can feel “rotation” from the output video and audio.

  When the foot sensor 43-2 is operated, the video effect imparting processing unit 64 divides the captured image 91 into a plurality of cells, and remaps each cell, thereby providing an effect like a jigsaw puzzle. The given image 102 is generated, and the sound effect generator 65 adds a predetermined sample sound to the supplied accompaniment. At this time, the video effect imparting processing unit 64 associates the horizontal movement of the detected user movement with the amount of change in cell remapping, and the vertical movement is used as the cell position change speed. Therefore, for example, the greater the amount of motion detected, the greater the speed and amount of change in cell position. The sound effect generation unit 65 associates the detected magnitude of the horizontal movement of the user with the reproduction position in the sample, and associates the speed of movement with the reproduction speed of the sample. That is, when the user operates the foot sensor 43-2, the user feels that he / she is operating the “change amount per unit time” of the output video and audio according to the magnitude and speed of the movement. You can enjoy the speed of change in audio.

  In this way, the relationship between the application of the effect to the captured image and the sound effect added to the accompaniment (not only the relationship of processing but also the sensory relationship given to the user) By associating those having “good” with the operation of the same foot sensor, the user can easily create content that matches the “user's image”.

  Next, the self-movie storage unit 68 buffers the effect-added video data supplied from the video effect addition processing unit 64 and the sound effect-added audio data supplied from the sound effect generation unit 65. The video data to which the effect has been applied is supplied to the video display control unit 69, and the audio data to which the sound effect has been applied is output to the audio output control unit 70, and the video data to which the effect has been applied and the sound data to which the sound effect has been applied, Is encoded in a predetermined format and supplied to the network interface 71.

  The video display control unit 69 controls display of video data supplied from the self-movie storage unit 68 on the monitor 44. The audio output control unit 70 controls the audio output from the speakers 45-1 to 45-6 of the audio data supplied from the self-movie storage unit 68.

  The network interface 71 may use a self-movie composed of encoded video and audio and information that can uniquely distinguish the self-movie (for example, information such as a user ID input by the operation input unit 41). Or a self-movie ID may be generated based on the current time and its own device ID) to the self-movie distribution server 3 via the network 1 so that the user can Information such as the URL of a web page to be accessed when a self-movie download request is made to the movie distribution server 3 is transmitted to the mobile phone 4 held by the user.

  Further, in the self-movie creating apparatus 2, the user is instructed to start playing so that the user can understand to some extent the relationship between his / her actions and the effects given to video, audio, lighting, and the like. Then, for example, an effect is applied to the captured video for a predetermined time such as 30 seconds or 1 minute based on the detection result of the user's motion and the operation input to the foot sensors 43-1 to 43-8. An image is displayed on the monitor 44, a sound effect is added to the generated accompaniment, and sound is output from the speakers 45-1 to 45-6. However, a “practice mode” that is not stored as a self-movie is set. When the video data with the effect applied is displayed for a predetermined time such as 30 seconds or 1 minute, and the audio data with the sound effect is output as audio To be stored as a self-movie, via the network 1, is supplied to the self movie creation server 3 may be a "self-movie mode".

  Here, it is needless to say that the time set as the “practice mode” and the time set as the “self-movie mode” can be set as appropriate. For example, if the time set as the “practice mode” is long, the user can better understand the relationship between his / her actions and effects given to video, audio, lighting, and the like. If the time set as is short, there is an effect that the unexpectedness in the self-movie creation increases.

  Further, in the above description, the created self-movie has been described as being transmitted to the self-movie distribution server 3 via the network 1, but the mobile phone that the user holds the created self-movie. 4 or PDA 5, for example, using wireless communication such as Bluetooth, or using a system such as e-mail via the network 1, directly to the user without going through the self-movie distribution server 3. It may be supplied.

  FIG. 6 is a block diagram showing a configuration of the self-movie distribution server 3.

  A CPU (Central Processing Unit) 211 of the self-movie distribution server 3 controls the overall operation of the self-movie distribution server 3. Further, the CPU 211 responds to an operation input by the administrator of the self-movie distribution server 3 from the input unit 214 including the mouse 231 and the keyboard 232 via the internal bus 213 and the input / output interface 212. Then, a program stored in a ROM (Read Only Memory) 215 is loaded into a RAM (Random Access Memory) 216 and executed. Alternatively, the CPU 211 loads a program installed in the HDD 218 to the RAM 216 and executes it, and causes the output unit 217 such as the display 233 and the speaker 234 to output the execution result. Further, the CPU 211 controls the network interface 220, communicates with the outside, and executes data exchange.

  The HDD 218 stores the self-movie transmitted from the self-movie creation device 2 via the network in association with the information specific to the self-movie, and is further required for authentication processing or billing processing as necessary. The personal information of the user is stored.

  The CPU 211 is connected to the drive 219 as necessary via the internal bus 213 and the input / output interface 212, and the magnetic disk 221, the optical disk 222, the magneto-optical disk 223, or the like mounted on the drive 219 as necessary. Information can be exchanged with the semiconductor memory 224.

  Further, the CPU 211 is connected to the image processing unit 241, the audio processing unit 242, the billing processing unit 243, and the authentication processing unit 244 via the internal bus 213.

  Based on the control of the CPU 211, the image processing unit 241 reads out video data of a self-movie that is requested to be downloaded by the user from the HDD 218, and a device (for example, the mobile phone 4, the PDA 5, Depending on the type of personal computer 6 or home server 7), the video format, resolution, etc. are converted.

  Based on the control of the CPU 211, the audio processing unit 242 reads out the audio data of the self-movie requested to be downloaded from the user from the HDD 218, and the device (for example, the mobile phone 4, PDA 5, Depending on the type of the personal computer 6 or the home server 7), the audio format, the number of channels, and the like are converted.

  For example, the format of the self-movie transmitted from the self-movie creating apparatus 2 is a format that can be commonly viewed on the mobile phone 4, the PDA 5, the personal computer 6, and other devices such as 3GPP (3rd Generation Partnership Project). In this case, the self-movie distribution server 3 may read the self-movie requested by the user from the HDD 218 and transmit it without performing data conversion. However, the format of data that can be processed, the resolution of video that can be displayed, the speaker system that outputs audio data (for example, monaural, stereo, 5.1ch, etc.) differ depending on the device to which the self-movie is distributed. When the format of the self-movie transmitted from the self-movie creating apparatus 2 is not suitable for viewing the self-movie on the destination device, the image processing unit 241 and the audio processing unit 242 are based on the control of the CPU 211. Then, a process of converting the self-movie transmitted from the self-movie creating apparatus 2 into a format that can be viewed on the output destination device is executed.

  Based on the control of the CPU 211, the charging processing unit 243 refers to the user's personal information stored in the HDD 218 as necessary, or based on information necessary for charging transmitted from the user requesting the self-movie. Then, the accounting process is executed by accessing a server of another financial institution (for example, a credit card or electronic money system service provider).

  Based on the control of the CPU 211, the authentication processing unit 244 refers to the personal information of the user stored in the HDD 218 as necessary, and uses information necessary for authentication transmitted from the user requesting the self-movie. Execute the authentication process.

  Next, the self-movie creating process 1 executed by the self-movie creating apparatus 2 described with reference to FIG. 3 will be described with reference to the flowchart of FIG.

  Here, for example, for a predetermined time such as 30 seconds or 1 minute after the start of play is instructed by the user, the video data with the effect and the audio data with the sound effect are displayed on the monitor 44. Is output from the speakers 45-1 to 45-6, but is not stored as a self-movie, and is set to “practice mode”, and then video data on which an effect has been applied for a predetermined time such as 30 seconds or 1 minute. And the sound data to which the sound effect is added are displayed on the monitor 44, output from the speakers 45-1 to 45-6, stored as a self-movie, and supplied to the self-movie creation server 3 via the network 1. In the following description, the “self movie mode” is set.

  In step S <b> 1, the receiving unit 51 determines whether or not text data is input from a portable device possessed by the user, such as the mobile phone 4 or the PDA 5, by infrared communication or the like.

  If it is determined in step S1 that text data has been input, in step S2, the receiving unit 51 supplies the input text data to the video effect applying processing unit 64. The video effect imparting processing unit 64 converts the input text data into image data such as a bitmap that can be superimposed on the video, and temporarily stores it in a page buffer (not shown) or the like included therein.

  If it is determined in step S1 that no text data has been input, or after the processing in step S2, the operation input unit 41 has received an operation input for starting play from the user in step S3. Determine whether. If it is determined in step S3 that no play start operation input has been received, the process of step S3 is repeated until it is determined that a play start operation input has been received.

  When it is determined in step S3 that an operation input for starting play has been received, the operation input unit 41 sends a control signal indicating the start of play to an image processing unit 61, a video effect application processing unit 64, a foot sensor input detection unit 66, And since it supplies to the accompaniment production | generation part 67, the accompaniment production | generation part 67 produces the audio | voice data corresponding to the accompaniment used as the basis of the audio | voice data of a self movie in step S4. In addition, the accompaniment generation unit 67 may generate an accompaniment so that the transition from the “practice mode” to the “self movie mode” can be notified by a change in the accompaniment.

  In step S <b> 5, the image processing unit 61 starts acquisition of the video imaged by the camera 42, performs various signal processing such as A / D conversion, gamma correction, and white balance adjustment, and performs the motion detection unit 62. , And the video effect provision processing unit 64.

  In step S6, the motion detection unit 62 tracks the movement of the LED light held or worn by the user based on the captured image, detects the magnitude of the motion vector of the video data, etc. The movement of the user is detected, and the detection result is supplied to the illumination control unit 63, the video effect applying processing unit 64, and the sound effect generating unit 65.

  In step S <b> 7, the illumination control unit 63 controls illumination based on the detected user movement. For example, the illumination control unit 63 controls the illumination by changing the illuminance of the illumination, the blinking speed, or the like in accordance with the magnitude of the detected movement or the amount of change in the magnitude of the detected movement. To do.

  In step S8, the foot sensor input detection unit 66 determines whether any of the foot sensors has been operated based on the signals supplied from the foot sensors 43-1 to 43-8.

  If it is determined in step S8 that any one of the foot sensors has been operated, the foot sensor input detection unit 66 sends a signal indicating which of the foot sensors has been operated to the illumination control unit 63, the video effect applying processing unit. 64 and the sound effect generator 65. In step S9, the video effect applying processor 64 is described with reference to FIGS. 4 and 5 based on the signal supplied from the foot sensor input detector 66. As described above, the effect of the video corresponding to the operated foot sensor is obtained from the captured video supplied from the image processing unit 61 according to the detected user movement (supplied from the receiving unit 51 as necessary). And is supplied to the self-movie storage unit 68.

  In step S10, the sound effect generation unit 65 uses the sound effect corresponding to the operated foot sensor based on the signal supplied from the foot sensor input detection unit 66, based on the detected user movement. The sound data corresponding to the accompaniment supplied from the accompaniment creation unit 67 is added to the sound effect as described above to generate self-movie sound data, which is supplied to the self-movie storage unit 68.

  In step S <b> 11, the illumination control unit 63 performs illumination based on the signal supplied from the foot sensor input detection unit 66, for example, by changing the type of illumination to be turned on corresponding to the operated foot sensor. Control.

  In addition, although the process of step S6 thru | or step S11 was demonstrated as a process step performed sequentially here, these processes are performed in parallel and processed in real time.

  In step S12, the self-movie storage unit 68 determines whether or not a predetermined time has already elapsed since the start of play. If it is determined in step S12 that the predetermined time has not elapsed since the start of play, the self-movie storage unit 68 supplies the video data to the video display control unit 69 and displays the video data on the monitor 44 for audio output control. The audio data is supplied to the unit 70 and is output from the speakers 45-1 to 45-6, but is not stored inside, and the process returns to step S6, and the subsequent processes are repeated.

  If it is determined in step S12 that a predetermined time has elapsed since the start of play, in step S13, the self-movie storage unit 68 supplies video data to the video display control unit 69 to display on the monitor 44, and the audio data is displayed. Audio data is supplied to the output controller 70 and output from the speakers 45-1 to 45-6, and the video data and audio data are stored, that is, for transmission to the self-movie distribution server 3. Record and record.

  In step S14, each part of the self-movie creating apparatus 2 determines whether or not the play end time has come. If it is determined in step S14 that the play end time has not been reached, the process returns to step S6, and the subsequent processes are repeated.

  If it is determined in step S14 that the play end time is reached, in step S15, the self-movie storage unit 68 supplies the recorded video data and recorded audio data, that is, the generated self-movie, to the network interface 71. To do. The network interface 71 transmits the supplied self-movie together with information capable of uniquely distinguishing the self-movie to the self-movie distribution server 3 via the network 1, and the processing ends.

  Through such processing, the user performance is detected by two types of methods, that is, an operation input to the foot sensors 43-1 to 43-8 and a motion amount detected from the captured image, and the detected user is detected. Based on the performance of the video, it is possible to apply effects to the captured video and add sound effects to the created accompaniment. Can be generated. The user can express himself / herself in the creation of the self-movie and can enjoy it actively.

  Note that the information that can uniquely distinguish the self-movie may be information input by the user, or information that the self-movie creation device 2 creates from the date / time information and the device ID, for example. In the case where the self-movie creating apparatus 2 creates, the user may be notified so that it can be used when downloading.

  Next, processing executed by the self-movie distribution server 3 described with reference to FIG. 6 will be described with reference to the flowchart of FIG.

  In step S31, the CPU 211 determines whether or not the network interface 220 has received recorded video data and recorded audio data, that is, the created self-movie, from any of the self-movie creating apparatuses 2. To do.

  If it is determined in step S31 that a self-movie has been received, in step S32, the CPU 211 is information necessary for individually distinguishing the self-movie, for example, in the user ID or the self-movie creating apparatus 2. Along with the created information and the like, the self movie is stored in the self movie storage area of the HDD 218.

  If it is determined in step S31 that a self-movie has not been received, or after the processing in step S32 is completed, the CPU 211 causes the network interface 220 to be sent from any of the self-movie creation devices 2 in step S33. Then, it is determined whether or not a signal indicating a self-movie download request has been received. The signal indicating the self-movie download request includes, for example, information necessary for user authentication, information regarding a device that is a download destination, and information necessary for billing processing. If it is determined in step S33 that a self-movie download request has not been received, the process returns to step S31, and the subsequent processes are repeated.

  If it is determined in step S33 that a self-movie download request has been received, in step S34, the CPU 211 controls the authentication processing unit 244 to execute authentication processing.

  In step S <b> 35, the CPU 211 determines whether the user who requested the download of the self-movie is authenticated as a correct user based on the result of the authentication process supplied from the authentication processing unit 244.

  If it is determined in step S35 that the user has been authenticated, in step S36, the CPU 211 searches the HDD 218 for a corresponding self-movie that the user requested to download, and reads it out.

  In step S <b> 37, the CPU 211 determines the data format of the read self-movie and whether the destination device is, for example, the mobile phone 4, the PDA 5, the personal computer 6, or the home server 7. Based on the above, the destination device determines whether or not data conversion is necessary. Specifically, when the self-movie transmission destination device is a small device such as a mobile phone 4 or a PDA 5, for example, even if the audio data is 5.1ch Dolby digital data, it is reproduced only in stereo. The video quality such as the size and resolution of the video that can be displayed on the monitors of these devices is also lower than that in, for example, the personal computer 6 or the television receiver 8. . In addition, when a self-movie is transmitted to the personal computer 6 or the home server 7, the video display and audio output functions possessed by these devices are different. Therefore, the CPU 211 determines whether or not data conversion is necessary based on the information regarding the download destination device included in the signal indicating the self-movie download request.

  If it is determined in step S37 that data conversion is necessary, in step S38, the CPU 211 controls the image processing unit 241 and the audio processing unit 242 to set the data format corresponding to the transmission destination device to the self format. Convert the movie.

  If it is determined in step S37 that data conversion is not necessary, or after the process of step S38 is completed, in step S39, the CPU 211 controls the charging processing unit 243 to execute the charging process.

  In step S <b> 40, the CPU 211 controls the network interface 220 to transmit the self-movie to the self-movie download request source device via the network 1, and the process ends.

  If it is determined in step S35 that the user is not authenticated as the correct user, in step S41, the CPU 211 controls the network interface 220 to send an error message to the self-movie download request source device via the network 1. The message is sent and the process is terminated.

  Through such processing, the self-movie transmitted from the self-movie creating apparatus 2 is distributed to the user, and billing processing is executed.

  In the process described with reference to FIG. 8, it has been described that the authentication process and the charging process are performed for each self-movie download request. However, the authentication process and the charging process may be omitted, for example, monthly or yearly. It may be made to be performed.

  In the above description, as described with reference to FIGS. 2 and 3, the self-movie creating apparatus 2 is a large-sized device in which the housing 21 and the floor 22 constitute a space in which the user can sufficiently operate. Although it has been described that the operation input to the foot sensors 43-1 to 43-8 provided on the floor 22 is associated with the video effect and the audio sound effect application, the device captures the video and the user Any device that can receive operation input, can process image data and audio data, and can output video and audio, for example, a portable telephone 4, a PDA 5, a personal computer 6, or a digital still A camera or a digital video camera can also be used as the self-moving creation apparatus 2.

  Next, FIG. 9 is a functional block diagram for explaining a configuration example of the second embodiment of the self-movie creating apparatus 2. The self-movie creation device 2 is, for example, the above-described mobile phone 4, PDA 5, personal computer 6, and the like as long as the functions illustrated in the functional block diagram of FIG. 9 can be realized by hardware or software. Alternatively, any device other than a digital still camera or a digital video camera may be used, or a device (original device) for creating a self-movie may be used.

  Note that portions corresponding to those in FIG. 3 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  That is, the self-movie creating apparatus 2 in FIG. 9 is, for example, a mobile phone 4, a PDA 5, a personal computer 6, a digital still camera, or a digital camera, instead of the operation input unit 41 and the foot sensors 43-1 to 43-8. An operation input unit 302 such as a button, key, or dial included in a video or the like is used to receive a user's operation input. For example, the mobile phone 4, PDA 5, personal computer 6, digital still camera is used instead of the camera 42. Alternatively, a camera 301 such as a digital video camera is used to capture an image. Instead of the monitor 44, for example, a mobile phone 4, a PDA 5, a personal computer 6, a digital still camera, or a digital video camera is used. Monitor 303 This is used to display the video with the effect applied, and instead of the speakers 45-1 to 54-6, the speakers 304-1 and 304-2 are used for the output of the sound to which the sound effect is given. Except that the sensor input detection unit 66, the illumination control unit 63, and the illuminations 47-1 to 47-5 are omitted, the configuration is basically the same as that described with reference to FIG. .

  The buttons, keys, dials, etc. of the operation input unit 302 are not limited to text data input and play start command input, but during play, the foot sensors 43-1 to 4-3 described with reference to FIGS. Similarly to 43-8, it is assumed that the effect of a predetermined video is associated with the type of sound effect added to the accompaniment (or the effect applied to the accompaniment or sample sound source). When a plurality of buttons and keys are operated at the same time, effects or audio data associated with the simultaneously operated buttons and keys are added at the same time (combined and processed). You may do it.

  For example, when the self-movie creating device 2 in FIG. 9 is a small device capable of operation input while being held by the user, such as the mobile phone 4 or the PDA 5, The camera 301 has an image of itself or another person while operating the operation input unit 302 such as a button, key, or dial to apply an effect to the captured image, and a sound effect on the automatically created accompaniment Etc. can be added to create a self-movie.

  The created self-movie is displayed on the monitor 303 in the “practice mode” and the “self-movie mode”, and is output from the speakers 304-1 and 304-2 as well as stored in the “self-movie mode”. Then, it is transmitted from the network interface 71 to the self-movie distribution server 3 via the network 1.

  Next, the self-movie creating process 2 executed by the self-movie creating apparatus 2 described with reference to FIG. 9 will be described with reference to the flowchart of FIG.

  In steps S61 to S66, basically the same processing as that in steps S1 to S6 described with reference to FIG. 7 is executed.

  That is, when text data is input from the operation input unit 302, the input text data is supplied to the video effect applying processing unit 64 and converted into video data such as a bitmap that can be superimposed on the video. Are temporarily stored in a page buffer (not shown) or the like. Then, in response to a play start operation input, audio data corresponding to the accompaniment that is the basis of the audio data of the self-movie is created, acquisition of the captured video is started, and based on the captured video, the user's Motion is detected.

  In step S67, the video effect imparting processing unit 64 and the sound effect generating unit 65 are used to add a predetermined video effect and a sound effect to the accompaniment of the audio data based on the signal supplied from the operation input unit 302. It is determined whether any of the associated buttons or keys has been operated.

  If it is determined in step S67 that any button or key has been operated, in step S68, the video effect imparting processing unit 64 performs the process shown in FIGS. 4 and 5 based on the signal supplied from the operation input unit 302. As described above, the effect of the video corresponding to the operated button or key is determined based on the captured video supplied from the image processing unit 61 in accordance with the detected user movement (if necessary, the operation (The video on which the text data supplied from the input unit 302 is superimposed) is supplied to the self-movie storage unit 68.

  In step S <b> 69, the sound effect generation unit 65 uses the sound effect corresponding to the operated button or key based on the signal supplied from the operation input unit 302, and accompaniment based on the detected user movement. The sound data corresponding to the accompaniment supplied from the creating unit 67 is added with sound effects as described above or subjected to sound effects to generate self-movie sound data, which is supplied to the self-movie storage unit 68.

  In addition, although the process of step S66 thru | or step S69 was demonstrated as a process step performed sequentially here, these processes are performed in parallel and processed in real time.

  In steps S70 to S73, basically the same processing as that in steps S12 to S15 described with reference to FIG. 7 is executed, and the processing ends.

  That is, it is determined whether or not a predetermined time has elapsed since the start of play, and when it is determined that the predetermined time has not elapsed, the created video is displayed on the monitor 303 and the created audio data Are output from the speakers 304-1 and 304-2, but are not stored internally, and the process returns to step S66, and the subsequent processes are repeated. If it is determined that a predetermined time has elapsed since the start of play, similarly, the created video is displayed on the monitor 303, and the created audio data is output from the speakers 304-1 and 304-2. At the same time, the created self-movie is stored. Then, it is determined whether or not it is the play end time, and if it is determined that it is not the play end time, the process returns to step S66, and the subsequent processes are repeated to determine that the play end time is reached. In this case, the self-movie generated and stored is transmitted to the self-movie distribution server 3 via the network 1, and the process is terminated.

  Through such processing, the user's performance is detected by two types of methods, that is, an operation input to a button or a key and a motion amount detected from the captured video, and based on the detected user's performance, Since the effect was applied to the captured video and sound effects can be added to the created accompaniment, even users who do not have special knowledge or technology can use original, advanced image processing and A self-movie that has been subjected to sound processing can be created using familiar devices such as the mobile phone 4, the PDA 5, the personal computer 6, a digital still camera, or a digital video camera.

  Furthermore, for example, in an open space such as a club or a party venue, the entire venue can be made to function as a self-movie creating apparatus. In other words, it captures video at any location in the venue, detects the motion of the video, and supports the effects of a given video and the addition of sound effects to accompaniment (or effects on audio data) For example, a plurality of operation input units such as sensors, pads, foot sensors, buttons or keys are provided. Then, through the same processing as described above, an effect is applied to the captured image and audio data is generated based on the motion of the image and the operation input to the operation input unit, and the entire open space is used. It is also possible to display the generated video on a large screen monitor so that it can be viewed, and to output the generated audio data from a speaker. At this time, as in the case of the first embodiment, it is also possible to control the illumination of the entire room, and the number of users to be imaged is not limited to one, and a plurality of cameras are prepared. Thus, a plurality of imaging points can be prepared.

  That is, the present invention prepares a space closed by the casing 21 as in the case of the first embodiment, or a small device that can be gripped by the user as in the case of the second embodiment. The present invention can be applied to other than using the above. In other words, as long as the information processing apparatus can capture video and detect the user's performance, the present invention can be applied to the user's performance and the effect on the captured video and created audio. It is possible to create a self-movie that has been subjected to advanced image processing and audio processing, in association with the provision of.

  In addition, since the degree of freedom of operation input is high, a user who creates a self-movie for the first time can create a self-movie with a high degree of effect even if simple operation and operation input are performed, A user who has played many times can create a self-movie with a high-level effect intended by the user. In other words, the user can express themselves and actively enjoy the creation of a self-movie, so that even a simple user interface can be enjoyed repeatedly.

  The series of processes described above can also be executed by software. The software can be a computer in which the program constituting the software is installed in dedicated hardware, or various types of functions can be executed by installing various types of programs. It is installed from a recording medium on a computer or the like.

  As shown in FIG. 6, this recording medium is distributed to provide a program to the user separately from the computer, and includes a magnetic disk 221 (including a flexible disk) on which the program is recorded, an optical disk 222 (CD-ROM). (Including Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), magneto-optical disk 223 (including MD (Mini-Disc) (trademark)), or semiconductor memory 224 .

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  In the present specification, the term “system” represents the entire apparatus constituted by a plurality of apparatuses.

It is a figure which shows the system configuration | structure of a self-movie delivery system. It is a figure which shows the external appearance structure of 1st Embodiment of the self-movie production apparatus. It is a block diagram which shows the internal structure of the self-movie production apparatus of a 1st form. It is a figure for demonstrating the effect given to an image | video. It is a figure for demonstrating the effect given to an image | video. It is a block diagram which shows the structure of a self-movie delivery server. 6 is a flowchart for explaining self-movie creation processing 1; It is a flowchart for demonstrating the process of a self-movie delivery server. It is a block diagram which shows the internal structure of the self-movie production apparatus of a 2nd form. It is a flowchart for demonstrating the self-movie production | generation process 2. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Network 2 Self movie production apparatus 3 Self movie delivery server 41 Operation input part 42 Camera 43 Foot sensor 44 Monitor 45 Speaker 47 Illumination 62 Motion detection part 63 Illumination control part 64 Image effect provision process part 65 Sound effect generation part 66 Foot sensor input Detection unit 67 Accompaniment creation unit 68 Self movie storage unit 211 CPU
241 Image processing unit 242 Audio processing unit 243 Charge processing unit 244 Authentication processing unit

Claims (11)

In an information processing apparatus that creates content,
Imaging means for imaging first video data;
Action information acquisition means for acquiring information relating to user actions;
An operation input means for acquiring the operation input of the user;
Voice generating means for generating first voice data;
Based on the information about the user's motion acquired by the motion information acquisition unit and the user's operation input input by the operation input unit, the first video data captured by the imaging unit is predetermined. Video processing means for performing the first processing and generating second video data constituting the content;
Based on the information related to the user's motion acquired by the motion information acquiring means and the operation input of the user input by the operation input means, the first voice data generated by the voice generating means Audio processing means for performing second predetermined processing to generate second audio data constituting the content;
Display control means for controlling display of the second video data generated by the video processing means;
And an audio output control means for controlling the output of the second audio data generated by the audio processing means. The information processing apparatus includes:
A housing composed of a ceiling surface and a wall surface having an opening through which the user can enter and exit;
A space that can be sufficiently operated by the user, and is distinguished from the outside by a floor installed at a lower portion of the housing;
The information processing apparatus according to claim 1, wherein a plurality of the operation input units are installed on an upper surface of the floor portion and receive an operation input by being stepped on by the user's foot. The information processing apparatus according to claim 1, wherein the information processing apparatus is a portable information processing apparatus that can be held by the user. Controls output of the content composed of the second video data generated by the video processing unit and the second audio data generated by the audio processing unit to another information processing apparatus. The information processing apparatus according to claim 1, further comprising content output control means. The apparatus further comprises illumination control means for controlling illumination based on the information related to the user's action acquired by the action information acquiring means and the operation input of the user input by the operation input means. The information processing apparatus according to claim 1. In response to the user's operation input input by the operation input means, the first processing applied to the first video data by the video processing means, and the first audio by the audio processing means. The information processing apparatus according to claim 1, wherein the second process applied to the data is a process having relevance. In an information processing method of an information processing apparatus for creating content,
A voice generation step of generating first voice data;
An imaging control step for controlling imaging of the first video data;
An operation information acquisition step for acquiring information about the user's operation;
An operation input acquisition step of acquiring the user's operation input;
Imaging is controlled by the process of the imaging control step based on the information related to the user's action acquired by the process of the operation information acquisition step and the operation input of the user acquired by the process of the operation input acquisition step. A video processing step of performing a predetermined first process on the first video data and generating second video data constituting the content;
Based on the information related to the user's motion acquired by the processing of the motion information acquisition step and the operation input of the user acquired by the processing of the operation input acquisition step, it is generated by the processing of the voice generation step. An information processing method comprising: an audio processing step of performing predetermined second processing on the first audio data to generate second audio data constituting the content. A program for causing a computer to execute content creation processing,
A voice generation step of generating first voice data;
An imaging control step for controlling imaging of the first video data;
An operation information acquisition step for acquiring information about the user's operation;
An operation input acquisition step of acquiring the user's operation input;
Imaging is controlled by the process of the imaging control step based on the information related to the user's action acquired by the process of the operation information acquisition step and the operation input of the user acquired by the process of the operation input acquisition step. A video processing step of performing a predetermined first process on the first video data and generating second video data constituting the content;
Based on the information related to the user's motion acquired by the processing of the motion information acquisition step and the operation input of the user acquired by the processing of the operation input acquisition step, it is generated by the processing of the voice generation step. An audio processing step of performing predetermined second processing on the first audio data to generate second audio data constituting the content, and causing the computer to execute the processing, program. A first information processing device for creating content;
An information processing system comprising: a second information processing device that receives the content from the first information processing device and distributes the content to a user;
The first information processing apparatus includes:
Imaging means for imaging first video data;
Action information acquisition means for acquiring information about the user's action;
Operation input means for acquiring the user's operation input;
Voice generating means for generating first voice data;
Based on the information about the user's motion acquired by the motion information acquisition unit and the user's operation input input by the operation input unit, the first video data captured by the imaging unit is predetermined. Video processing means for performing the first processing and generating second video data constituting the content;
Based on the information related to the user's motion acquired by the motion information acquiring means and the operation input of the user input by the operation input means, the first voice data generated by the voice generating means Audio processing means for performing second predetermined processing to generate second audio data constituting the content;
Display control means for controlling display of the second video data generated by the video processing means;
Audio output control means for controlling the output of the second audio data generated by the audio processing means;
Transmission that transmits the second video data generated by the video processing means and the content composed of the second audio data generated by the audio processing means to the second information processing apparatus Means and
The second information processing apparatus
Receiving means for receiving the content transmitted by the transmitting means of the first information processing apparatus;
Storage means for storing the content received by the receiving means;
An information processing system comprising: a distribution control unit that controls distribution of the content stored in the storage unit to the user when a distribution request for the content is received from the user. The second information processing apparatus
The information processing apparatus according to claim 9, further comprising: a conversion unit that converts a data format of the content corresponding to a type of a distribution destination device of the content whose distribution is controlled by the distribution control unit. system. The second information processing apparatus
The information processing system according to claim 9, further comprising a billing processing unit that executes billing processing that occurs for the distribution of the content whose distribution is controlled by the distribution control unit.

Images