JP2000115672A - Video data processing method, video data processor and video data reproducing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily reproduce changes in the appearance of a background or the like which accompanies the movement of a camera device in composited video images. SOLUTION: At photographing, photographing data including the information of the position and direction, etc., of the camera device 300 are recorded in a recording medium (DVD-R) 1, together with video data. Then, by using the photographing data obtained by reproducing the DVD-R 1, computer graphic video images are generated. Thus, the movement of the camera device 300 at photographing is reflected on the computer graphic video image, and by compositing the computer graphic video image and the video image obtained by photographing, realistic composited video images are generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video data processing method and apparatus for recording, reproducing and outputting video data obtained by photographing a subject with a camera device.

[0002]

2. Description of the Related Art A chroma key method is known as a technique for synthesizing a plurality of images. For example, a person in a space with a blue background is photographed by a camera device, and the obtained image of the person is combined with an image of a virtual space generated by computer graphics or the like. This allows
It is possible to create a composite image that looks as if a person is in a virtual space. Such a video synthesizing technique is used for producing a movie or a television program.

[0003]

When a person is photographed together with a real background around the person (that is, when photographing normally), the position and direction of the camera device are adjusted in accordance with the movement of the person. When it is changed, the appearance of the scene seen through the lens of the camera device changes accordingly. For example, if the camera device is moved to the left so as to follow a person moving to the left, the appearance of the background seen through the lens changes accordingly. Similarly, if the camera device is rotated leftward so as to follow a person moving leftward, the appearance of the background seen through the lens changes accordingly. Changing the camera angle for a person changes the appearance of the person, and also changes the appearance of the background behind the person.

In order to reproduce such a change in the appearance of the background due to the movement of the camera device in the composite image as described above, a virtual background image is planned in accordance with the movement of the camera device at the time of shooting. Needs to be changed. Traditionally,
The motion pattern of the camera device at the time of shooting is planned in advance, shooting is performed according to the plan, a virtual background image is created according to the plan, and then the image obtained by shooting and the virtual background image are Had been synthesized. However, such a work has a problem that it takes a lot of trouble and time.

[0005] Further, in such a conventional method, since it is necessary to shoot and create a background image under a thorough plan, a composite image of a person and a virtual background image is used in a live TV program or the like. It is extremely difficult to produce in real time and change the background video in real time with the movement of the camera device.

In addition, the inventor of the present application has proposed a camera that can obtain motion information representing the motion of the camera device when the subject is being photographed, together with an image generated by photographing the subject with the camera device. They came to the idea that the change in the appearance of the background due to the movement of the device can be relatively easily reproduced in the synthesized video. However, in order to realize this, it is necessary to temporally associate the video with the motion information of the camera device and convert the data into data.
This is unresolved. Further, in order to temporally correlate the video and the motion information of the camera device into data,
It is necessary to determine the data format in consideration of the recording format of a recording medium such as a DVD-R for recording the data, and this has not been solved.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and a first object of the present invention is to easily reproduce a change in appearance such as a background caused by movement of a camera device in a composite image. It is an object of the present invention to provide a video data processing method and a video data processing device which enable the above.

A second object is to provide a video data processing method and a video data processing apparatus capable of temporally associating video data with motion information of a camera device so as to match the recording format of a recording medium and converting the data into data. Is to provide.

[0009]

According to a first aspect of the present invention, there is provided a video data processing method comprising the steps of: capturing video data generated by capturing a subject with a camera device; Inputting shooting data including position information indicating the position of the camera device when the camera is turned on, a video data storing procedure for storing a predetermined amount of input video data, and a predetermined amount stored in the video data storing procedure. And an output procedure for outputting the video data to which the predetermined amount of video data is temporally associated, and for outputting the video data to which the video data is added.

[0010] This makes it possible to temporally associate the video data with the position information of the camera device and convert it into data. Therefore, if a virtual background image or the like is created using the position information of the camera device corresponding to the video data, it is possible to easily reproduce a change in the appearance of the background due to the movement of the camera device in the composite image. it can.

According to a second aspect of the present invention, there is provided a video data processing method comprising:
In addition to the respective steps of the video data processing method according to the first aspect of the present invention, there is provided a recording procedure for recording the video data, to which the photographing data has been added, output in the output procedure, on a recording medium. As a result, the video data and the position information of the camera device can be temporally associated and converted into data, and the data can be recorded in accordance with the recording format of the recording medium.

According to a third aspect of the present invention, there is provided a video data processing method comprising:
A video data group generating procedure for generating a video data group by arranging a predetermined amount of stored video data; and And a photographing data arranging procedure for arranging photographing data temporally corresponding to the image data of the image data adjacent to the image data group. This makes it possible to temporally associate the video data with the position information of the camera device and convert it into data.

According to a fourth aspect of the present invention, there is provided a video data processing method comprising:
In addition to the steps of the video data processing method according to any one of claims 1 to 3, the video data output in the output procedure and to which the imaging data is added is received, and the received data is referred to as the imaging data. A separation procedure for separating video data, a video generation procedure for generating a first video corresponding to the separated video data, and a video stream generated based on other video data according to position information included in the separated shooting data. And a synthesizing procedure for synthesizing the first video and the second video. In this manner, the second video can be moved using the position information of the camera device corresponding to the video data. Therefore, for example, when the second video is a virtual background video, it is possible to easily create a change in the appearance of the background due to the movement of the camera device.

Further, in order to solve the above-mentioned problems, a video data processing apparatus according to each of the inventions according to claims 5 to 8 realizes each procedure of the video data processing method according to each of the inventions described above. The operation or action of each of these means is the same as that of each procedure described above.

According to a ninth aspect of the present invention, there is provided a video data reproducing apparatus comprising:
The video data generated by photographing the subject with the camera device and the photographing data including the position data representing the position of the camera device at the time of photographing the subject are stored in a storage medium in which the video data and the photographing data are taken. There are provided a reading means for reading data, and an output means for outputting the read video data and the photographing data while synchronizing them. Thereby, it is possible to easily obtain the video data and the photographing data temporally corresponding to the video data.
Therefore, if a virtual background video or the like is created using this photographing data, a change in the appearance of the background due to the movement of the camera device can be easily reproduced in the composite video.

[0016]

Embodiments of the present invention will be described below with reference to the accompanying drawings. In the embodiments described below, a case is described in which the video data processing method and the video data processing device according to the present invention are applied to a video data processing system.

The video data processing system includes a video data recording system and a video data reproduction system.
A video data recording system is a system that acquires video data and photographing data, converts these data into recording data having a predetermined data format, records the data on a recording medium, or outputs the data to the outside. The video data recording system can record and output audio data together with video data and shooting data. On the other hand, the video data reproducing system reads the recording data recorded on the recording medium by the video data recording system, or receives the recording data output from the video data recording system, and extracts the video data and the photographing data from the recording data. Then, the system creates a composite video in real time using these data. Further, the video data reproducing system can reproduce audio data together with video data.

1. Video data and shooting data The video data is, as shown in FIG.
Is data obtained by photographing a subject b such as a person. Each piece of video data corresponds to an image of one field almost in the same manner as forming a normal television screen. The video data is output from the camera device 300 for, for example, 60 fields in 0.4 second to 1 second. As described later, since the camera provided in the camera device 300 is a digital camera, the video data is digital data.

The photographing data is data such as information on the movement and optical settings of the camera device 300 and information on the photographing site, and includes the following data. Note that the photographing data is digital data, and the numerical values in parentheses indicate the size (number of bytes) of each data.

Position data (12 bytes) Direction data (4 bytes) Tilt data (2 bytes) Distance data (4 bytes) Brightness data (4 bytes) Color data (6 bytes)

The position data is data representing the position of the camera device 300 at the shooting site in three dimensions. For example,
The camera device 300 in FIG. 1 is located at a point C in a three-dimensional space. In this case, the position data of the camera device 300 is (xc, yc, zc).

The direction data is data representing the direction of the camera device 300 in three dimensions. The direction data includes data representing the visual line polar angle (bank) of the camera device 300 and data representing the visual line azimuth angle (heading) of the camera device 300. Become. The line-of-sight polar angle is an angle between the line of sight of the camera device 300 and the Z-axis in three-dimensional space. This means the angle between the direction of the camera device 300 and the horizontal plane, and specifies the vertical direction of the camera device 300. The line-of-sight azimuth is an angle formed between the line of sight of the camera device 300 and the X-axis in the three-dimensional space.
This means the direction of the camera device 300 on a horizontal plane, and specifies the left-right direction of the camera device 300. For example, as shown in FIG. 2, the position of the camera device 300 is set to a point C (the origin of the coordinates shown in FIG.
Assuming that the line of sight of 00 is a line D, the line-of-sight polar angle is the angle B between the line D and the Z axis, and the line of sight azimuth is the angle H between the line D and the X axis.

As shown in FIG. 1, the inclination data is obtained from the camera device 30 with the line of sight of the camera device 300 as the central axis.
This is data representing the rotation angle (tilt angle) P of 0, and represents the tilt (pitch) of the camera device. Further, the distance data includes, as shown in FIG.
This is data representing the distance Q to.

The brightness data is data representing the intensity (brightness) of the light source 5 provided at the shooting site. The color data is data representing the balance of the color (RGB) of the light source 5.

In synchronization with the output of the video data, the photographing data is transmitted from the camera device 300 for 60 seconds to 0.4 seconds.
Output twice. That is, the photographing data is output each time the video data corresponding to one field is output. Therefore,
The movement of the camera device 300 and the change in the state of the light source at the shooting site are accurately reflected in the shooting data.

The data size of the photographing data is predetermined in relation to a data format to be described later, and is set so that the total data size of the photographing data corresponding to the video data for 60 fields is less than 2 kilobytes. Have been. For example, the size of each data included in the shooting data is set to a predetermined number of bytes as described above. With this setting, the total data size of the shooting data corresponding to the video data for 60 fields is 1920 bytes, which is less than 2 kilobytes.

The photographing data may include, in addition to the data described above, data relating to the zoom, pan, iris, shutter speed, lens, and the like of the camera device. However, when the data size is limited as in the present embodiment, it is desirable to preferentially include information that changes at the time of shooting in the shooting data. For example, since the position and direction of the camera device frequently change when shooting is performed, the priority of including data on the position and direction of the camera device in the shooting data is high. On the other hand, since it is rare that the lens of the camera device is frequently replaced during shooting, the priority of including data on the lens in the shooting data is low.

2. Data Format As described above, the video data recording system converts video data and shooting data into a predetermined data format,
Record on a recording medium. In this embodiment, a DVD-R is used as a recording medium. The video data recording system is shown in FIG.
As shown in FIG.
Convert to 0. The data format of the recording data 10 conforms to the DVD-R recording format based on the DVD standard.

In FIG. 3, the recording data 10 is composed of an array of a plurality of cells 11. Further, each cell 11
Is constituted by an array of a plurality of VOBUs 12. The cell 11 is a data group based on the DVD standard, and the VOBU 12 is a finer data group based on the DVD standard. Further, each VOBU 12 is configured by an array of shooting data packs 13, a plurality of video packs 14, and a plurality of audio packs 15. The photographing data pack 13 is arranged at the head of each VOBU 12, and photographing data is arranged therein. In each video pack 14, video data is arranged, and in each audio pack 15, audio data is arranged. Each pack has a header 16 and ID 1 for identifying each pack.
7 and a data body (for example, the photographing data 18 in the case of the photographing data pack 13).

Each VOBU 12 includes video data for 60 fields. And each VOBU12
In the photographing data pack 13 included one by one,
Photographing data temporally corresponding to the video data for 60 fields included in the OBU 12 is included. That is, each VOBU 12 generates a video data group by arranging video data for 60 fields, and captures data (that is, captures) that temporally corresponds to the video data for 60 fields included in the video data group. The photographing data, which is sometimes output simultaneously with the video data from the camera device 300, is arranged adjacent to the head of this video data group. As described above, the video data and the photographing data temporally corresponding to the video data are
By arranging the data in the OBU 12, the video data and the photographed data can be deleted or copied integrally, and the data editing process can be made more efficient and simpler.

Further, the size of the photographing data pack 13
That is, the data size of photographing data that can be included in the photographing data pack 13 is limited to less than 2 kilobytes. By limiting the size of the photographing data pack 13 to a relatively small size, recording and reproduction efficiency can be improved.

3. Video Data Recording System FIG. 4 shows a video data recording system according to an embodiment of the present invention. First, the configuration of the video data recording system will be described with reference to FIG.

As shown in FIG. 4, the video data recording system includes a recording device 100 and a camera device 300.

The camera device 300 is a digital camera 30
1 and a camera position detector 302. The digital camera 301 is a photographing digital camera for photographing a subject. The digital camera 301 outputs video data for 60 fields in 0.4 second to 1 second during shooting. The digital camera 301 outputs a synchronization signal indicating an output cycle of the video data together with the video data. Also, the digital camera 301 is provided with a microphone (not shown), and the microphone is used to record the sound being captured. And the digital camera 301
Generates and outputs audio data by converting the audio into digital data.

Further, the digital camera 301 measures the distance between the digital camera 301 and the subject, and outputs the measurement result as distance information. The digital camera 3
The distance between the object 01 and the subject can be measured by a well-known distance measuring method such as triangulation using the detection light and an optical sensor (both not shown). Furthermore, the digital camera 301 has the brightness of the light source at the shooting site and the RGB values.
The balance is detected, and the information is output as brightness information and color information. For example, the digital camera 301
Includes a light amount detector (not shown), and detects the brightness of the light source and the RGB balance by the light amount detector.

The camera position detector 302 is a device for measuring the position, direction (line-of-sight polar angle and line-of-sight azimuth) and inclination of the digital camera 301 at the shooting site. The camera position detector 302 detects the position of the digital camera 301,
The direction and tilt measurement results are output as position information, direction information, and tilt information. The digital camera 30
The position, direction, and inclination of 1 can be measured by, for example, providing a potentiometer on a base supporting the digital camera 301 and using the potentiometer, or providing an ultrasonic sensor on the digital camera 301 and the like, and measuring by using this Alternatively, the measurement can be performed using a known method such as a method using a detection light and an optical sensor.

On the other hand, the recording apparatus 100 includes a photographing data encoder 101, a synchronization signal detector 102, MPEG encoders 103 and 104, a packing CPU 105, a buffer 106, an encoder 107, a data output terminal 108,
It includes a waveform converter 109, an optical pickup 110, and a spindle motor 111.

The photographing data encoder 101 receives the position information, direction information and tilt information output from the camera position detector 302, and the distance information, brightness information and color information output from the digital camera 301, and receives these information. The information is converted into various data having the above-mentioned predetermined number of bytes, that is, position data, direction data, inclination data, distance data, brightness data, and color data to generate photographing data. Further, the photographing data encoder 101 stores the photographing data in accordance with the synchronization signal output from the synchronization signal detecting unit 102.
Output to. Thus, the photographing data is output 60 times in a period of 0.4 second to 1 second in synchronization with the video data.

The synchronizing signal detecting section 102 detects a synchronizing signal output together with video data from the digital camera 301 and outputs the synchronizing signal to the photographing data encoder 101.

The MPEG encoder 103 receives video data output from the digital camera 301 via the synchronization signal detection unit 102, and converts the video data into an MPEG signal.
(Moving Picture Experts Group). On the other hand, the MPEG encoder 104
The audio data output from the digital camera 301 is received, and the audio data is transferred to an MPEG (Moving Picture E-mail).
xperts Group).

The packing CPU 105 stores photographing data,
The recording data is generated by arranging (packing) the video data and the audio data according to the data format described above. The buffer 106 is a memory for temporarily storing shooting data, video data, and audio data when the packing CPU 15 generates recording data.

The encoder 107 performs processing such as adding an error detection code, scrambling processing, adding an error correction code, interleaving processing, 8/16 modulation, and NRZI (Non Return to Zero Inverse) conversion processing to the recording data. And converts the recording data into an NRZI signal. Further, the encoder 107 has a function of converting the recording data into a transmission signal according to a predetermined communication protocol.

The waveform converter 109 is a circuit for performing a so-called recording strategy, and is a circuit for processing the waveform of the NRZI signal output from the encoder 107 in order to improve the shape of pits formed on the DVD-R1. It is.

The optical pickup 110 irradiates the DVD-R1 with a light beam, and switches the irradiation level of this light beam based on the NRZI signal output from the waveform conversion unit 109, thereby converting the recording data into a DVD-R1. R
1 is recorded. Spindle motor 111
Is a motor for rotating the DVD-R1.

Next, the operation of the video data recording system will be described. First, the camera device 300 is installed at the shooting site. Then, the DVD-R1 is set in the recording device 100, and the recording operation of the recording device 100 is started. Thus, the recording apparatus 100 performs the initialization operation and rotates the DVD-R1.

When the photographing is started, the digital camera 30
1 outputs distance information, brightness information, color information, video data, a synchronization signal, and audio data, and the camera position detector 302 outputs position information, direction information, and tilt information. The position information, the direction information, the inclination information, the distance information, the brightness information, and the color information are obtained by the imaging data encoder 101 using the position data, the direction data, the inclination data, the distance data, the brightness data, and the color each having a predetermined number of bytes. It is converted to shooting data including data,
Output to packing CPU 105. Video data is
After passing through the synchronization signal detection unit 102, the MPEG encoder 1
03 and output to the packing CPU 105. At this time, the synchronization signal is
Is supplied to the imaging data encoder 101, thereby synchronizing the output of the imaging data and the output of the video data. The audio data is encoded by the MPEG 104 and output to the packing CPU 105. The shooting data, video data, and audio data are arranged (packed) by the packing CPU 105 according to the above-described data format, and output to the encoder 107 as recording data. Further, the recording data is converted into an NRZI signal by the encoder 107 and supplied to the optical pickup 110 via the waveform converter 109. Thereby, the NRZI signal is transmitted to the optical pickup 11
0 is recorded as a pit on the DVD-R1.

The recording data is converted by the encoder 107 into a transmission signal according to a predetermined communication protocol, and the transmission signal is output from the encoder 107 to the outside via the data output terminal 108. For example, if the data output terminal 108 is connected to the data input terminal 205 provided in the playback device 200 of the video data playback system, a transmission signal can be sent to the playback device 200.

Next, the operation of the packing CPU 105 will be described in detail with reference to FIG. First, packing C
The PU 105 receives one of the photographing data output from the photographing data encoder 101, the video data output from the MPEG encoder 103, and the audio data output from the MPEG encoder 104 (step 11). Next, the packing CPU 10
5 determines whether the received data is photographing data, video data, or audio data (step 12). If the received data is shooting data, packing C
The PU 105 sends the photographing data to the buffer 106 and stores it in a predetermined area in the buffer 106 (Step 13). On the other hand, when the received data is video data or audio data, the packing CPU 105
This video data or audio data is sent to the buffer 106, and is temporarily stored in a predetermined area in the buffer 106 (step 14). Next, the packing CPU 105
Determines whether all the shooting data, video data, and audio data for forming one VOBU have been acquired (step 15). That is, it is determined whether or not the video data for 60 fields and the photographing data and audio data corresponding to the video data have all been stored in the buffer 106. When all the data is stored in the buffer 106, the packing CPU 105 reads the data from the buffer 106, arranges the data according to the above-described data format, and generates print data (step 16). On the other hand, when all these data are not stored in the buffer 106, the packing CPU 105 repeats the processing of steps 11 to 15.

4. Video Data Reproduction System FIG. 6 shows a video data reproduction system according to an embodiment of the present invention. Here, the configuration of the video data reproduction system will be described with reference to FIG.

The video data reproducing system includes a reproducing device 200 and a video synthesizing device 400 as shown in FIG.

The reproducing apparatus 200 includes the optical pickup 20
1, spindle motor 202, signal extraction unit 203, decoder 204, data input terminal 205, data processing unit 20
6, buffer 212, additional information synthesizing section 213 and D /
A conversion units 214 and 215 are provided.

The optical pickup 201 irradiates a DVD-R1 having pits corresponding to recording data with a light beam, converts the reflected light into an electric signal, and outputs the electric signal. The spindle motor 202 is a motor for rotating the DVD-R1. The signal extraction unit 203 is a circuit that extracts a reproduction signal including recording data and a control signal for performing tracking servo control, focusing servo control, and the like, from the electric signal output from the optical pickup 201.

The decoder 204 performs 8/16 demodulation, deinterleave processing, error detection processing, error correction processing, and the like on the reproduction signal output from the signal extraction unit 203, and converts the reproduction signal into recording data. It is.
Further, the decoder 204 has a function of converting a transmission signal input from the data input terminal 205 into recording data.

The data processing unit 206 includes the input control unit 20
7, output control unit 208, photographing data decoder 209, M
PEG decoders 210 and 211 are provided. The input control unit 207 reads the recording data output from the decoder 204 for each VOBU, and performs separation processing of a photographing data pack, a video pack, and an audio pack arranged in the VOBU. Photographing data decoder 209
Converts the shooting data included in the shooting data pack into data suitable for the background video generation unit 401 of the video synthesis device 400. The MPEG decoder 210 decodes the video data included in the video pack according to the MPEG system. The MPEG decoder 211 decodes audio data included in the audio pack according to the MPEG system.
The output control unit 208 outputs the captured data, video data, and audio data converted or decoded by the decoders 209, 210, and 211 according to the synchronization signal output from the background video generation unit 401.

The buffer 212 is a memory for temporarily storing photographed data, video data, and audio data converted or decoded by the decoders 209, 210, and 211 provided in the data processing unit 206.

The additional information synthesizing unit 213 includes the data processing unit 2
This is to add additional information such as subtitles to the video data output from the device 06. The D / A converter 214 is a circuit that performs digital-to-analog conversion of video data to which additional information has been added. The D / A converter 215 is a circuit that performs digital-to-analog conversion of audio data output from the data processor 206.

On the other hand, the video synthesizing device 400 includes a background video generation unit 401 and a video synthesizer 402. The background image generation unit 401 is configured by, for example, an image generation computer. The background video generation unit 401
The image data output from the data processing unit 206 of the playback device 200 is received, and a background video is generated using the image data. For example, when the background image is generated by a plurality of polygons, the background image generation unit 401 performs the operation of the following Expression 1 using the position data, the direction data, and the inclination data included in the shooting data, so that each polygon is calculated. The vertex coordinates (Xa, Ya, Za) are calculated.

[0058]

Xa = cosΦcosΘ (Xb + Δx) + sinΦcosΘ (Yb + Δy)
-sinΘ (Zb + Δz) Ya = (-sinΦcosΨ + cosΦsinΘsinΨ) (Xb + Δx) + (cosΦc
osΨ + sinΦ sinΘsinΨ) (Yb + Δy) + cosΘsinΨ (Zb + Δz) Za = (sinΦsinΨ + cosΦsinΘcosΨ) (Xb + Δx) + (-cosΦs
inΨ + sinΦ sinΘcosΨ (Yb + Δy) + cosΘcosΨ (Zb + Δz) where Xa, Ya, Za are the vertex coordinates of the polygon after the movement, Xb, Yb, Zb are the vertex coordinates of the polygon before the movement,
Δx, Δy, and Δz represent the difference between the previous position coordinates and the current position coordinates of the camera device, Θ represents the visual line polar angle of the camera device, Φ represents the visual line azimuth angle of the camera device, and Ψ represents the camera tilt.

The background image generation section 401 enlarges or reduces the background image using the distance data included in the photographing data. Further, the background video generation unit 401 adds a shadow to the background video using the brightness data and the color data included in the shooting data, and sets the brightness and color of the background video.

Further, background video generating section 401 outputs a synchronizing signal for synthesizing the background video and the video data to data processing section 206 of reproducing apparatus 200. On the other hand, the video synthesizer 402 synthesizes a video corresponding to video data and a background video.

Next, the operation of the video data reproducing system will be described. First, the video synthesizing device 400 is started, and the DVD-R1 on which the recording data is recorded by the video data recording system is set in the reproducing device 200, and the reproducing operation of the reproducing device 200 is started. Thereby, the initialization operation is performed in the playback device 200 and the DV
D-R1 rotates.

Subsequently, a DVD is read from the optical pickup 201.
A light beam is emitted toward −R1, and a signal corresponding to the reflected light of the light beam is output to the signal extraction unit 203. Then, a reproduction signal is extracted from this signal by the signal extraction unit 203 and output to the decoder 204. Thus, the reproduced signal is converted into recording data by the decoder 204 and output to the data processing unit 206.

The recording data output to the data processing unit 206 is read for each VOBU by the input control unit 207 and separated into a photographing data pack, a video pack, and an audio pack. Further, the photographing data included in the photographing data pack is converted by the photographing data decoder 209 into data suitable for the background video synthesizing unit 401, and the image data included in the video pack is converted by the MPEG decoder 2.
The audio data included in the audio pack is decoded by the MPEG decoder 211. The converted or decoded shooting data, video data, and audio data are temporarily stored in the buffer 212. The photographing data, video data, and audio data stored in the buffer 212 are transmitted from the buffer 212 to the data processing unit 2 at predetermined timing described later.
In response to the synchronization signal output from the background video generation unit 401, the data is output from the data processing unit 206 for each field.

Subsequently, the photographing data output from the data processing unit 206 is input to the background video generation unit 401.
As a result, the background image is generated by the background image generation unit 401. The video data output from the data processing unit 206 is input to the video synthesizer 402 via the additional information synthesis unit 213 and the D / A conversion unit 214. In this process, an image corresponding to the image data is generated. Then, this video is combined with the background video generated by the background video generation unit 401. Also, the data processing unit 206
Is output to a speaker or the like via the D / A converter 215.

On the other hand, the video data reproducing system has a data input terminal 205 of the reproducing device 200 and a data output terminal 1 provided in the recording device 100 of the video data recording system.
08, the transmission signal output from the recording device 100 can be taken into the reproducing device 200. In this case, the transmission signal output from the recording device 100 is input to the decoder 204 from the data input terminal 205 of the reproducing device 200, and is converted into recording data by the decoder 204. Then, the recording data is output to the data processing unit 206. Thereafter, the same processing as the above-described processing is performed on the recording data, and the generation of the composite video and the reproduction of the audio are performed.

Next, the output timing of the photographing data, video data and audio data output from the data processing section 206 will be described with reference to FIG. FIG. 7 shows the output timings of the synchronization signal output from the background video generation unit 401 and the shooting data, video data, and audio data output from the data processing unit 206, respectively. As shown in FIG. 7, the synchronization signal has one clock cycle from one fall to the next fall. The video data is output for each field in synchronization with the synchronization signal. Similarly, the audio data is output in synchronization with the synchronization signal. On the other hand, photographing data is also output periodically according to the clock cycle of the synchronization signal. However, the photographing data is output at a point earlier than the video data and the audio data by 1/2 clock cycle of the synchronization signal. This takes into account the time required for the background video generation unit 401 to receive the imaging data output from the data processing unit 206 and to finish generating the background video using this imaging data. Accordingly, the timing at which the video data and the audio data are output from the data processing unit 206 and the timing at which the background video is output from the background video generation unit 401 can be matched.

Thus, according to the embodiment of the present invention, the background image is generated using the photographing data, so that the change in the appearance of the background due to the movement of the camera device 300 at the time of photographing can be recognized in the composite image. It can be easily reproduced.

For example, a person is photographed using the chroma key method, and video data obtained at this time from the camera device 300 and photographed data temporally corresponding to the video data are recorded on the DVD-R 1 by the recording device 100. I do. Next, the DVD-R 1 on which the recording data has been recorded
Reproduce by 00. Then, the video synthesizing device 400 generates a virtual background video using the photographed data obtained from the reproducing device 200, and synthesizes the background video with the video corresponding to the video data output from the reproducing device 200. As a result, shooting effects such as movement of the camera device, changes in the camera angle, and zoom-up appear not only in the actually shot video but also in the virtual background video, and the live-action video and the virtual video are integrated. Realistic composite images can be created.

Further, since the movement of the camera device 300 and the like at the time of photographing are recorded as photographing data on the DVD-R1,
By simply playing back the DVD-R1, a realistic composite video in which the video of the person and the background video are integrated can be automatically created.

Further, the recording apparatus 100 according to the above-described embodiment can directly output recording data including video data and photographing data to the outside via the data output terminal 108. Further, the reproducing device 200 can directly take in the recorded data via the data input terminal 205. This makes it possible to generate a composite video almost simultaneously with shooting. Therefore, it is possible to easily create a special effect such as producing a composite video of a person and a virtual background video in real time in a live TV program and changing the background video in real time with the movement of the camera device.

When recording the photographing data on the DVD-R1, the photographing data can be recorded in accordance with the recording format of the DVD standard by arranging the photographing data at the head of each VOBU. Thereby, DVD-R
As a recording medium suitable for generating a composite video, its usability can be improved, and it can contribute to digitization of movie shooting and television program production, improvement of video synthesis technology, and improvement of computer graphic technology.

In the above-described embodiment, an example has been described in which a background video is generated using photographed data, but the present invention is not limited to this. For example, it is also possible to generate a person or an animated character created by computer graphics using the photographing data.

In the above-described embodiment, the photographing data corresponding to the video data for 60 fields is included in the photographing data pack and is arranged in the VOBU. However, the present invention is not limited to this. For example, sampling data is generated by sampling shooting data corresponding to video data for 60 fields at a predetermined rate,
This sampling data may be arranged in the VOBU. Thereby, the amount of photographing data can be reduced. Also, by calculating the average of each numerical value included in the shooting data corresponding to the video data for 60 fields, average value data is generated, and this average data is referred to as a VOBU.
It may be arranged inside. This can also reduce the amount of shooting data.

In the above-described embodiment, the digital camera 301 for outputting digital video data is provided in the camera device 300. However, the present invention is not limited to this, and an analog camera for outputting analog video data is provided. You may. In this case, analog video data may be converted from analog to digital.

Furthermore, in the above-described embodiment, the recording apparatus 100 and the reproducing apparatus 200 are provided in separate systems. However, the present invention is not limited to this, and the two may be integrated.

Further, in the above-described embodiment, the case where the DVD-R is used as the recording medium for recording the recording data has been described as an example, but the present invention is not limited to this, and the present invention is not limited to this.
Other recording media such as a RAM can be used.

In the above embodiment, step 11 in FIG. 5 is a specific example of the input procedure.
4 is a specific example of the video data storage procedure. Step 16 is a specific example of the addition procedure. The output procedure is
This is realized by a series of output operations from the packing CPU 105 to the optical pickup 110 or the data output terminal 108.

[0078]

As described above in detail, according to the video data processing method of the first aspect of the present invention, it is possible to easily obtain the position information of the camera device temporally corresponding to the video data.
A change in the appearance of the background due to the movement of the camera device can be easily reproduced in the composite video.

According to the video data processing method of the second aspect of the present invention, the video data and the position information of the camera device can be time-correlated and converted into data. Can be recorded. Furthermore, if the recording medium is reproduced, it is possible to easily obtain the position information of the camera device temporally corresponding to the video data, and easily change the appearance of the background due to the movement of the camera device in the composite video. Can be reproduced.

According to the video data processing method of the third aspect of the present invention, the video data and the position information of the camera device can be time-correlated and converted into data. For example, DVD
And the like adopt a recording format in which a video data group (for example, what is called a pack) including a predetermined amount of video data is created and the video data group is arranged in a recording data stream. The video data and the position information of the camera device can be time-correlated and converted into data so as to conform to such a recording format.

According to the video data processing method of the present invention, for example, it is possible to easily create a change in the appearance of the background due to the movement of the camera device, and to create a more realistic synthesized image. it can.

According to the video data processing apparatus of the present invention, the same effects as those of the above-described inventions can be obtained.

According to the video data reproducing apparatus of the ninth aspect, it is possible to easily reproduce, for example, a change in the appearance of the background due to the movement of the camera device in the composite video.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram showing a position and the like of a camera device in an embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a direction of a camera device in the embodiment of the present invention.

FIG. 3 shows a format of data recorded on a DVD-R in the embodiment of the present invention.

FIG. 4 is a block diagram showing a recording apparatus according to the embodiment of the present invention.

FIG. 5 is a flowchart illustrating an operation of the recording apparatus according to the embodiment of the present invention.

FIG. 6 is a block diagram showing a playback device according to the embodiment of the present invention.

FIG. 7 is a timing chart showing output timings of shooting data, video data, and audio data in the playback device according to the embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 1 DVD-R (recording medium) 100 recording device 101 photographing data encoder 103, 104 MPEG encoder 105 packing CPU 106 buffer (video data storage means) 108 data output terminal (output means) 110 optical pickup (recording means) 200 playback device 201 Optical pickup (reading means) 205 Data input terminal 206 Data processing unit 212 Buffer 300 Camera device 301 Digital camera 302 Camera position detector 400 Image synthesizing device 401 Background image generating unit 402 Image synthesizing unit

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/225 H04N 5/92 H 5/92 F term (Reference) 5C022 AA01 AB00 AB62 AB68 AC00 AC69 AC72 AC80 CA00 5C053 FA09 FA14 FA24 GA11 GB05 GB37 HA29 JA03 JA21 KA03 KA21 KA24 LA01 LA06 5D044 AB07 BC06 CC04 DE11 DE17 DE52 EF05 FG18 FG21 GK11 5D090 AA01 BB04 CC01 CC04 GG11 GG21 GG36

Claims (9)

[Claims] An input step of inputting video data generated by photographing a subject with a camera device and photographing data including position information indicating a position of the camera device when photographing the subject; A video data accumulation procedure for accumulating a predetermined amount of the input video data, and adding the shooting data temporally corresponding to the predetermined amount of video data to the predetermined quantity of video data accumulated in the video data accumulation procedure And an output step of outputting the video data to which the shooting data has been added. 2. The video data processing method according to claim 1, further comprising a recording step of recording, on a recording medium, the video data to which the shooting data has been added, output in the output step. 3. The method according to claim 1, wherein the adding step includes a step of generating a video data group by arranging the stored predetermined amount of video data. A photographing data arranging step of arranging the corresponding photographing data adjacent to the video data group.
2. The video data processing method according to 1. 4. A separating step of receiving the video data to which the shooting data has been added and output in the output step, and separating the received data into shooting data and video data; And a position setting step of setting a position of a second image generated based on other image data according to the position information included in the separated shooting data. 4. The video data processing method according to claim 1, further comprising a synthesizing procedure for synthesizing the first video and the second video. 5. An input means for inputting video data generated by photographing a subject with a camera device and photographing data including position information indicating a position of the camera device when photographing the subject. Video data storage means for storing a predetermined amount of the input video data; and adding the photographing data temporally corresponding to the predetermined amount of video data to a predetermined amount of video data stored by the video data storage means. A video data processing apparatus comprising: an adding unit that performs the above operation; and an output unit that outputs the video data to which the photographing data is added. 6. The recording apparatus according to claim 5, further comprising recording means for recording the video data to which the photographing data is added on a recording medium.
2. The video data processing device according to claim 1. 7. An image data group generating means for generating a video data group by arranging said stored predetermined amount of video data, said adding means comprising: 7. A photographing data arranging means for arranging the corresponding photographing data adjacent to the video data group.
2. The video data processing device according to claim 1. 8. Separation means for receiving video data to which the photographing data has been added and output by the output means, and separating the received data into photographing data and video data; Image generation means for generating a first image corresponding to the following; position setting means for setting a position of a second image generated based on other image data according to the position information included in the separated shooting data; 8. The video data processing device according to claim 5, further comprising a synthesizing unit for synthesizing the first video and the second video. 9. A storage medium in which video data generated by photographing a subject with a camera device and photographing data including position data representing a position of the camera device when photographing the subject are recorded. A video data reproducing apparatus comprising: a reading unit that reads out the video data and the photographing data; and an output unit that outputs the read video data and the photographing data while synchronizing them.