JP2011035638A - Virtual reality space video production system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system capable of synthesizing virtual space video images using three-dimensional CG with real space video images photographed by a camera to display the video images in real time, and to further provide a system by which restriction on a range of a camera operation is sharply mitigated, and whose device configuration is made into small scale to sharply reduce video production costs. <P>SOLUTION: This virtual reality space video production system is at least provided with a virtual space video creation means 12 for creating the virtual space video images using a computer a real space video photographing means 10 for photographing the real space video images using the camera a video synthesis display means 13 for synthesizing the virtual space video images with the real space video images to be displayed on a display screen, and a camera information detection means 11 for detecting camera operation information based on an operation of a user, inputs the camera operation information to the virtual space video creation means, and repeatedly creates virtual space video images reflecting the camera operation information for every video frame. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a virtual reality space video production system using a camera, and as a system for synthesizing and displaying a real video and a virtual video corresponding to the shooting position in real time using a real camera and a computer system for CG. The present invention relates to a system in which a user efficiently and effectively performs video production and video viewing using CG.

  Due to recent advances in video technology, the video usage environment, particularly the screening environment, has entered our daily lives as being easier than before. While this situation increases the demand for high-quality video software, more efficient production techniques are required.

  The development of information processing technology in terms of hardware such as personal computers has greatly reduced the time and labor required for processing and processing large volumes of data such as high-definition video. However, the video production process (photographing → processing → editing) has not changed significantly compared to the conventional method.

  However, in this production process, the processing process, especially the synthesis of live-action video and computer graphics (hereinafter referred to as CG), is increasingly being used in movies and TV programs, but it takes time and money. It has become work.

Here, the synthesis processing method is generally as follows.
First, in live-action shooting, a subject such as a person is photographed against a background of a single color such as blue, called a blue back, and a human space is cut out from the blue back area by a video composition device and extracted as a key signal. There is a technique called chroma key composition, in which video is synthesized as an electronic signal.
In recent years, a system called a virtual studio is known that uses a method of generating a CG image linked to the operation of a live-action camera and synthesizing it with the live-action image of the camera.

  Conventionally, there is a virtual studio system based on real-time three-dimensional CG as a means for solving such a problem in the composition of video, and its technical development has been tackled in various fields, but by using this virtual studio system, The time required for shooting and processing (compositing live action and CG) can be greatly reduced.

  In particular, because the processing time required for video composition has been shortened, the final video can be checked in real time during shooting, so video production that requires accurate composition of the virtual space, and TV programs that require immediate distribution. It is used in production sites.

Japanese Patent Publication No. 2004-56742 Japanese Patent Publication No. 2007-42073 Japanese Patent Publication No. 2008-33219

However, the conventional three-dimensional virtual studio system as mentioned in the background art has restrictions on the shooting environment. First, when shooting, the range of camera operation may be limited to only a part of the studio, and in order to properly place in the virtual space, the camera in the real space is at an accurate position and speed. The control device has been quite large, for example, using a large camera using a crane.

  That is, the camera operation range is greatly limited and the camera system itself has a large apparatus configuration as compared with the case of normal live-action shooting, and there is a problem that the cost is increased.

  On the other hand, in the case of video production with a small budget, a large-scale device configuration could not be adopted, and shooting was performed with the camera position fixed as much as possible. There was a problem that it would be a video in.

  For this reason, there is a video production system that can provide higher video effects with an inexpensive device configuration even when shooting with existing virtual studio systems, where there are many restrictions on the arrangement of shooting equipment and the range in which the camera can be moved. It was sought after.

The present invention has been made in order to solve the above-described problems, and an object of the present invention is to create a virtual space image by 3D CG and capture a real space image taken by a camera and the image. An object of the present invention is to provide a video production system capable of displaying a video by synthesizing a virtual space video suitable for the shooting position in real time.
A further object of the present invention is to provide a system capable of reducing the video production cost by reducing the system configuration of the system.

In the present invention, in order to achieve the above object, a virtual reality space video production system according to claim 1 includes virtual space video creation means for creating a virtual space video using a computer,
Real space video photographing means for photographing real space video using a camera;
Video composition display means for synthesizing the virtual space image and the real space image and displaying on the screen;
At least camera operation information detecting means for detecting camera operation information based on a user's operation, inputting the camera operation information to the virtual space image creating means, and generating a virtual space image reflecting the camera operation information as a video frame It is characterized by being repeatedly created every time.

  Here, the virtual reality space image refers to an image obtained by synthesizing a virtual space image created by CG using a computer and a real space image obtained by photographing a subject that is an entity of the real space.

  In the present invention, information on the position and orientation of the camera and the lens operation when the user operates the camera (hereinafter referred to as camera operation information) is reflected when creating a virtual space image as a CG drawing parameter. As a result, a virtual space image linked with the operation of the camera is created.

  As a result, the composite image of the real space image and the virtual space image looks as if the subject, which is a substance, enters the virtual space. Further, since the virtual space video reflecting the camera operation information is repeatedly created and displayed for each video frame by the computer, it is displayed as a moving image in real time.

The virtual reality space video production system according to claim 2,
The camera operation information detecting means acquires a position information of the camera in the real space based on a common reference point in the virtual space and the real space, and a shooting for acquiring the shooting direction information of the camera Direction acquisition step, camera setting information acquisition step for acquiring camera setting information including the focal length and lens aperture value of the camera, and further output the camera operation information acquired in these steps, via wireless communication or wired cable And a camera operation information output step for inputting to the virtual space video creating means.

The camera operation information mainly includes at least information relating to camera movement, that is, a change in position, information relating to a change in the shooting direction (angle) of the camera, and information relating to camera settings such as zooming of the screen and focus / lens aperture.
Then, by outputting these camera operation information to a computer for CG creation, a virtual space image is created using the information as CG drawing parameters.

The virtual reality space video production system according to claim 3,
The camera operation information detection unit includes a module with a GPS function, and the camera position acquisition step acquires camera position information using a GPS function.

The virtual reality space video production system according to claim 4,
The camera operation information detection means includes an IC tag equipped with a position sensor function,
In the camera position acquisition step, camera position information is acquired using a position sensor function of the IC tag.

The virtual reality space video production system according to claim 5,
The camera operation information detection means includes a gyro sensor (angular velocity sensor),
In addition, the photographing direction acquisition step is characterized by detecting the angular velocity when the camera is moved by the gyro sensor and acquiring the photographing direction information of the camera.

The virtual reality space video production system according to claim 6,
The video composition display means is characterized in that the subject of the real space video is a person or object and a blue back screen is used as the background, and the virtual space video is used as a background for this.

The virtual reality space video production system according to claim 7,
The camera includes an image processing processor and a display monitor, and the video composition display means performs the composition processing for each video frame using the image processing processor and displays the composite video on the display monitor. And

According to the present invention, a virtual reality space video production system that synthesizes and displays a virtual space image and a real space image is used to create, display, and confirm a virtual image linked to the camera operation by the user in real time. be able to.
In addition, the system configuration of the present invention is a system in which the camera system includes a position sensor and a direction sensor, and acquires and outputs information on the position and orientation of the camera with a simple configuration. Can be built.

It is a system diagram showing an example of a system configuration in an embodiment of the present invention. It is a flowchart which shows an example of the operation procedure in the embodiment of this invention. It is a flowchart which shows an example of the operation procedure in the embodiment of this invention.

  Embodiments of a virtual reality space video production system according to the present invention will be described below in detail with reference to the drawings.

FIG. 1 shows a basic configuration of a system according to an embodiment of the present invention.
In the embodiment of the present invention, the virtual reality video production system 1 is roughly divided into two device configurations, one of which is a camera system unit 2 that performs shooting of live-action video and composite display of the video. This is a CG system unit 3 that creates a virtual space image by three-dimensional CG. These camera system unit 2 and CG system unit 3 are connected by a communication interface capable of high-speed processing in order to communicate video information in real time.

(Camera system part)
First, in the camera system unit 2 surrounded by a dotted line in FIG. 1, the subject 4 is photographed by the user 4 using the real space video photographing means 10. The captured video is synthesized with a virtual space video, which will be described later, using the video synthesizing unit 13, and the synthesized video is displayed on the screen of the monitor 24 by the video display unit 14.

  Here, the user refers to a person who operates the camera and takes a video. The video is usually a moving image, but may be a still image. The video type is not particularly limited in terms of analog or digital type, data signal type, specification standard such as compression / non-compression, recording apparatus, recording medium, and recording time, and is used according to the purpose as appropriate. Furthermore, the camera device main body, the photographing method, and the like can be used as appropriate.

The main device configuration of the camera system unit 2 is to connect a photographing camera device main body, a video display monitor device, a position sensor device such as the GPS or IC tag sensor, a computer device equipped with an image processor, and them. Includes an interface.
However, the system of the present invention is not limited to the above-described apparatus configuration, and the configuration can be appropriately changed or an apparatus can be added as necessary.

  The real space video photographing means 10 uses a blue back screen as the background of the subject 5 when the user 4 operates the camera body to photograph the subject 5. As described above, since only a person or an object as a subject is cut out from the background as a video area by performing chroma key processing, a monochrome background such as blue or green is used.

  When shooting, for example, the user 4 may move the position by moving the camera to change the camera angle or change the distance between the camera and the subject depending on the intention of the production. In order to utilize such camera operation information in this system, the camera operation information detection unit 11 sequentially detects camera operation information when shooting is being performed, and this information is transmitted to a virtual space image creation unit 12 described later. Output.

  Since the user 4 operates the camera to shoot, the camera 4 determines settings related to camera shooting such as the camera position, shooting viewpoint angle (angle), and lens operations (zoom, pan, focus, etc.). According to this setting, the real space video photographing means 10 performs photographing. Then, such setting information relating to the camera is signaled as camera operation information by the camera operation information detecting means 11.

(Camera operation information detection means)
Here, the main implementation procedure of the camera operation information detection means 11 will be described with reference to FIG. Steps A1 to A4 in FIG. 3 are flowcharts showing a procedure for detecting camera operation information.
First, in step A1 (camera position acquisition), the spatial position of the camera at the time of shooting is acquired as position information. As a method for acquiring the position of the camera, for example, there is a method in which an IC tag having a GPS function and a position sensor described later is attached to the camera. In addition, this invention is not limited to these methods, You may use another method.

Here, the spatial position information of the camera refers to a single reference point that is common to the virtual space and the real space, and the camera position based on the reference point is the three-dimensional position of each of the virtual space and the real space. It is the information applied to the coordinates.
For example, when a position that is 1 meter away from the object as a subject is set as a reference point, if the camera is at that reference point first, the virtual space creates a background image on which the subject is placed based on that position. . When the camera moves from the reference point by a predetermined direction / distance, a background image is created again based on the position moved in the same direction / distance from the reference point in the virtual space.

  In this way, camera position information is acquired so that the synthesized video can be seen by the viewer without any discomfort by linking the movement of the camera position and the movement of the background video in the virtual space.

  In the next step A2 (shooting direction acquisition), information on the shooting direction of the camera, that is, the orientation of the camera lens is acquired. The method for acquiring the shooting direction of the camera is not particularly limited in the present invention. For example, there is a method in which a sensor for detecting the rotation speed and the rotation direction, such as a gyro sensor described later, is attached to the camera.

  In step A3 (camera setting information acquisition), operation information belonging to camera work such as zoom, pan, and focus is acquired as camera setting information. In order to acquire this information, a mechanical part that senses and acquires a mechanical operation such as a lens aperture or an operation button of the camera may be provided, and a known means may be used.

  In the next step A4 (camera operation information output), each piece of information acquired in the steps A1 to A3 is converted into a signal as a set of camera operation information and output via a communication interface such as wireless communication or a wired cable. Then, it is transmitted to the virtual space image creating means 12.

Returning to FIG.
In the camera operation information detecting means 11, an example of specific means will be shown.
The camera operation information can be converted into a signal by providing the camera system unit 2 with various sensors. In the present invention, for example, by providing the photographing camera body with a GPS function device, the current position of the camera body can be known by radio wave communication with the GPS communication satellite.

  Alternatively, by providing an IC tag with a position sensor function (hereinafter referred to as an IC tag sensor) in the camera body for photographing, it is possible to know the position from a radio wave transmitted from the IC tag sensor and a response from the sensor arranged in the studio. it can.

  Furthermore, other known techniques may be used as the position sensor, and for example, a magnetic sensor, a laser light sensor, or the like may be used as the position measuring means.

(CG system department)
Next, the CG system unit 3 will be described.
The CG system unit 3 includes a three-dimensional CG program 20, a first computer 21 for operating the three-dimensional CG program 20, and an operation terminal device as main device configurations, and creates a CG image in real time by the three-dimensional CG program 20. Has function.

  The virtual space image creating means 12 inputs necessary information to the first computer 21 and freely creates a virtual space image that does not actually exist by inputting background data such as a studio set. Can do.

  However, such a virtual space image is uncomfortable for the viewer when the movement of the subject and the state of the virtual space are shifted in the combined image with the real space image. Therefore, in order to prevent such a problem from occurring, the shooting position of the camera in the virtual space image and the state of the lens operation such as zooming coincide with the image shot by the real space image shooting means 10 in real time. It is necessary to let If it is a moving image, a new image must be created for each frame at a timing without delay.

  The camera operation information detected and output as a signal by the camera operation information detection means 11 is first input to the virtual space image creation means 12, where the actual operation information is obtained based on information such as the camera position, angle, and zoom. A virtual space image is created in accordance with the space image.

(Video composition and display)
The video synthesizing unit 13 has a function of synthesizing the video of the subject 5 photographed by the real space video photographing unit 10 and the virtual space video as the background created by the virtual space video creating unit 12.

  The video composition process is performed by the second computer 23 equipped with the image processor 22 of the camera system unit 2. In this process, first, an image obtained by cutting out only the subject 5 from the real space image, excluding the background blue background region, is created and then superimposed on the synchronized virtual space image. This synthesis process is performed for each frame and output as a video signal.

  The video display means 14 inputs the synthesized video created by the video synthesis means 13 and displays it on the video display monitor 24. The user 4 visually recognizes the composite image displayed on the monitor 24.

  In other words, by providing these means, the camera system unit 2 can perform real-time video image synthesis processing and composite video display while shooting a real space video image. You can check the image and operate the camera while watching this to produce video.

  Although the composite video created here is not particularly shown, it can be connected to an external video recording device, an external monitor, or a broadcasting facility to record or save the video, It can also be possible to confirm.

(System processing procedure)
FIG. 2 is an example of an operation procedure in the system of the present invention, and is a flowchart showing a processing procedure of the system in FIG. Here, steps S1 to S9 are procedures in which a user of the system of the present invention produces a video.

  First, in step S1, the user 4 operates the camera to photograph the subject 5. At this time, the user can freely operate the camera according to the user's own intention, and after predicting the synthesized video of the subject to be photographed and the virtual space video to be synthesized, It is possible to determine the position and orientation of the camera, lens operations such as zooming, and the like.

  In addition, the camera can be moved during zooming or zooming operation can be performed at any time according to the user's intention. That is, such settings related to the operation of the camera are not one way and may change at any time during shooting.

Next, in step S2, camera operation information such as the position and orientation of the camera and the lens operation when the user 4 is operating the camera is detected.
Here, in order to know the operation state of the camera, the camera operation information detecting means 11 detects the camera operation information using various sensors such as the GPS function, the IC tag sensor, and the gyro sensor described above, and this is detected by the CG system unit 3. This is transmitted to the virtual space image creating means 12. If the captured video is a moving image, the camera operation information is updated and transmitted in real time for each moving image frame.

In the next step S3, a viewpoint for rendering a virtual space image in the three-dimensional CG program 20 based on the transmitted camera operation information, that is, information on the camera operation such as the camera position, shooting direction, and lens aperture. Updates drawing information related to position, angle and video creation range.
However, since the camera operation information does not change when the user 4 stops the camera operation, the drawing information is not updated and the previous information can be maintained.

  In step S4, the three-dimensional CG program 20 creates a virtual space image based on the drawing information updated in step S3 and outputs the data. This video data is created for each video frame, but when the drawing information is not updated, there is no change in the video, so the previous data can be applied as it is, and when the drawing information is updated, the video is immediately updated. As data is created, video data following the change in the camera operation information is output in real time.

  In step S5, real space video is captured by the camera in parallel with step S4, and the video data is input to the second computer 23 of the camera system unit 2.

  In step S6, the virtual space video data created in step S4 is transferred to the second computer 23 of the camera system unit 2 via the communication interface.

  In step S7, the second computer 23 combines the real space image created in step S5 with the virtual space image data transferred in step S6. At this time, the real space video uses the above-described blue back screen as the background of the subject, and the subject is cut out from the video by chroma key processing, and the virtual space video is synthesized as the background.

  This composition processing is performed in real time for each moving image frame. Since such processing requires high speed, it is preferable to perform image composition calculation processing using the image processor 22 of the second computer 23. .

  In step S8, the synthesized video is displayed on the screen of the video confirmation monitor 24 provided in the camera system unit 2. The composite video displayed here is a video in which the subject photographed in the real space and the background of the virtual space created by the three-dimensional CG are synthesized as described above, and the camera operation and the virtual space are visually recognized. This is an image that is linked to the direction.

  Step S9 is a step in which the user 4 sees and confirms the composite video displayed on the monitor 24. Here, the user 4 can check whether there is a problem in the synthesized video of the subject 5 and the virtual space. If the monitor 24 is in the finder portion of the camera, the finder can be viewed while photographing, and the user 4 can efficiently check.

  By performing the above steps S1 to S9, creation of a virtual reality space image and confirmation of the image can be performed in real time. Thereafter, even if the process returns to step S1 and the camera operation is changed, the composite video is created and confirmed by performing each step in the same manner.

  In the above description, an example and an implementation procedure based on the basic configuration of the system are shown. However, the system of the present invention is not limited to the configuration shown here, and components may be added or changed as necessary. Can be implemented.

  According to the virtual reality space video production system of the present invention, it is possible to create and display in real time a virtual video that is linked to the camera operation by the user, which is effective for the viewer without forming an expensive set in the studio. It is possible to produce images that can be appealing at a low cost.

DESCRIPTION OF SYMBOLS 1 Virtual reality video production system 2 Camera system part 3 CG system part 4 User 5 Subject 10 Real space video imaging means 11 Camera operation information detection means 12 Virtual space video creation means 13 Video composition means 14 Video display means 20 3D CG program 21 First computer 22 Image processor 23 Second computer 24 Monitor

Claims (7)

A virtual reality space video production system that mixes virtual space video and real space video using a camera, creating virtual space video using a computer; and
Real space video photographing means for photographing real space video using a camera;
Video composition display means for synthesizing the virtual space image and the real space image and displaying on the screen;
At least camera operation information detecting means for detecting camera operation information based on user operation,
The camera operation information is input to the virtual space video creation means, and a virtual space video reflecting the camera operation information is repeatedly created for each video frame,
Virtual reality space video production system. The camera operation information detection means includes
A camera position acquisition step of acquiring position information of the camera in real space based on a common reference point in virtual space and real space;
A shooting direction acquisition step of acquiring shooting direction information of the camera;
A camera setting information acquisition step for acquiring camera setting information including a focal length and a lens aperture value of the camera;
Furthermore, it includes at least a camera operation information output step of outputting the camera operation information acquired in these steps and inputting the information to the virtual space image creation means via wireless communication or a wired cable.
The virtual reality space video production system according to claim 1. The camera operation information detection means includes a module with a GPS function,
3. The virtual reality space video production system according to claim 1 or 2, wherein the camera position acquisition step acquires camera position information using a GPS function. The camera operation information detection means includes an IC tag equipped with a position sensor function,
And the said camera position acquisition process acquires camera position information using the position sensor function of the said IC tag,
The virtual reality space video production system according to claim 1 or 2. The camera operation information detection means includes a gyro sensor (angular velocity sensor),
In addition, the shooting direction acquisition step detects angular velocity when the camera is moved by a gyro sensor and acquires shooting direction information of the camera.
The virtual reality space video production system according to any one of claims 1 to 4. The video composition display means is
A subject of the real space image is a person or an object, and a blue back screen is used for the background, and the virtual space image is combined with the background as a background,
The virtual reality space video production system according to any one of claims 1 to 5. The camera comprises an image processor and a display monitor;
The video composition display means performs the composition processing for each video frame using the image processor, and displays the composite video on a display monitor.
The virtual reality space video production system according to any one of claims 1 to 6.

Images