WO2019142283A1 - Image processing device, image processing device control method, and program - Google Patents

Abstract

[Problem] To provide an image processing device and a program which provide a smooth video for video content such as a game that uses a head-tracking type HMD. [Solution] An image processing device configured from a generation device and a synthesis device. The generation device is characterized by comprising: an objective camera position identification means for repeatedly identifying an objective virtual camera position, which is the position and orientation of a virtual camera; an object position identification means for repeatedly identifying an object position, which is the position and orientation of an object in virtual space; a positioning means for moving and thereby positioning a three-dimensional model in the virtual space; a virtual space image acquisition means for generating one or a plurality of virtual space images; and a transmission means for transmitting the one or plurality of virtual space images generated by the virtual space image acquisition means to the synthesis device. The synthesis device is characterized by comprising: a reception means for receiving a virtual space image; a real space image acquisition means for acquiring a real space image; and a mixed reality image acquisition means for generating a mixed reality image.

Description

Image processing apparatus, control method for image processing apparatus, and program

The present invention relates to an image processing apparatus, a control method of the image processing apparatus, and a program. In particular, the present invention relates to an image processing apparatus that provides smooth video in video content such as a game using a head tracking HMD, and a control method and program for the image processing apparatus.

In recent years, various head mounted displays (hereinafter referred to as "HMD") have appeared, and the possibility of expressing virtual reality (hereinafter referred to as "VR") has expanded. The HMD is a device that displays a stereoscopic image by displaying parallax images on the left and right eyes of a player wearing it on the head. The HMD can also impart immersive feeling and reality to the VR by tracking the movement of the player's head and reflecting the result in the image.

In addition, an image of a virtual world (hereinafter, referred to as “virtual space”) generated by a computer is superimposed on an image of the real world (hereinafter, referred to as “real space”) captured by a camera or the like. Techniques exist for displaying on a display device. Such techniques include Mixed Reality (hereinafter referred to as "MR") and Augmented Reality (hereinafter referred to as "AR"). In this MR, in order to enhance immersiveness and reality, the position and orientation of the player wearing the HMD are acquired in real time, and images of the virtual space are generated following the change of the image obtained by imaging the real space, Display to the player in real time through the HMD.

Therefore, the MR sets the position and orientation of the player measured by a sensor or the like as the virtual position and orientation in the virtual space, draws an image of the virtual space by CG based on the setting, and combines it with the image of the real space doing. Thereby, the player wearing the HMD can observe an image as if a virtual object exists in the real space. According to the development of such technology, there is a demand that a player wearing the HMD may image a situation in which the mixed reality is experienced by MR, or a third party may want to view the imaged image.

However, even if the player wearing the HMD is imaged in the real space, it can not be imaged including the object of the three-dimensional model existing in the virtual space only by imaging the player and the real space. Therefore, techniques using MR and chroma key composition are being studied. That is, the room experiencing the mixed reality is made homogeneous color (for example, green), the player wearing the HMD is imaged, the homogeneous color in the imaged image or video is made transparent, and virtual Overlap with the image of space. By doing this, it is possible to capture an image including the same virtual space as the player who is experiencing MR, or a third party can view the captured image.

However, in such a method, a player experiencing MR may not be able to properly display an image in a virtual space. For example, in a game using an HMD, a background object serving as a player's background and an item object serving as an item possessed by the player are used. At this time, the background object is displayed behind the player, and the item object has to be displayed before or after depending on the position and orientation of the player. However, since the player can roam freely, if the position of the player in the virtual space is not taken into consideration, the item object will be an unnatural image such that the item object is always displayed behind the player. As described above, various proposals have been made for video contents such as games using an HMD, but the related art will be described based on patent documents.

For example, there is known a technology of a video compositing apparatus that enables an actor to perform and a CG operator to perform an operation while checking a compositing result at a shooting site (see Patent Document 1). Further, there is known an image processing apparatus capable of providing a highly realistic augmented reality while reducing the burden on the image processing apparatus, a control method of the image apparatus, and a program technology (see Patent Document 2). Further, there is known an information processing apparatus that displays an image generated by imaging a player in a real space at an appropriate position on the virtual space, and a control method and program technology thereof (see Patent Document 3).

Japanese Patent Laid-Open No. 2000-230374 JP, 2013-8297, A JP, 2015-170232, A

The invention described in Patent Document 1 acquires three-dimensional information indicating the depth of an actual space when combining a photographed image and CG data, and based on the three-dimensional information, before and after the photographed image and CG data The composition is performed taking into consideration the relationship. In the invention described in Patent Document 1, the size of the object as viewed from the viewpoint set in the virtual space and the size of the object in the real space can be obtained by merely arranging the physical space image including the object in the virtual space. There is a problem that the size of the object when viewed from the imaging device is different.

In the invention described in Patent Document 2, a subject image of a non-moving object is mapped to a subject object and arranged in a virtual space. The invention described in Patent Document 2 has a problem that the current position of the object moving in the real space is specified in real time, and the real space image obtained by imaging the object can not be arranged in the virtual space. According to the invention described in Patent Document 3, the size of the object viewed from the objective camera and the size of the object included in the real space image viewed from the viewpoint set in the virtual space are substantially the same. The size of the space image is changed. In the invention described in Patent Document 3, after the physical space image is arranged in the virtual space, the generation of the image and the synthesis of the image are performed by the same information processing apparatus, so processing omission frequently occurs and a smooth image is generated. There was a problem that it could not provide.

The present invention is created to solve the above-mentioned conventional problems, and achieves the following object. The object of the present invention relates to an image processing apparatus that provides smooth video, a control method of the image processing apparatus, and a program.

The present invention takes the following means to achieve the above object.
An image processing apparatus according to the first aspect of the present invention is an image processing apparatus including a generating device for generating a virtual space in which a three-dimensional model is arranged, and a synthesizing device communicably connected to the imaging device and processing the captured image. An objective camera position specifying unit that repeatedly acquires the position and the orientation of the imaging device in the real space and repeatedly specifies the objective virtual camera position that is the position and the orientation of the virtual camera in the virtual space; Target position specifying means for repeatedly obtaining the position and the direction of the target in the target position repeatedly specifying the target position and the position of the target in the virtual space, and the objective virtual camera position and / or the target position according to While arranging a three-dimensional model in the virtual space, the three-dimensional model is moved according to the movement of the imaging device and / or the object Placement means for placing in the virtual space, and the three-dimensional model placed in the virtual space as one or more virtual space images with the objective virtual camera position specified by the objective camera position specifying means as a viewpoint The apparatus includes: virtual space image acquisition means for generating; and transmission means for transmitting the virtual space image generated by the virtual space image acquisition means to the synthesizing device, wherein the synthesizing device transmits the virtual space image acquired from the generating device. Receiving means for receiving a virtual space image, Real space image acquiring means for acquiring a real space image generated by imaging a physical space including the object by the imaging device, The virtual space image and the real space image And a mixed reality image acquisition unit that generates a mixed reality image by overlapping the

An image processing apparatus of the second invention is the first invention, wherein
The virtual space image acquisition means is configured to obtain the objective background image and the objective foreground according to the objective virtual camera position specified by the objective camera position specifying means and / or the target position specified by the target position specifying means. Generating an image.

An image processing apparatus according to the third aspect of the present invention is the second aspect, wherein the objective background image includes all of the three-dimensional model disposed in the virtual space, and the objective foreground image includes the objective camera. A partial image which is a part of the three-dimensional model arranged in the virtual space according to the objective virtual camera position specified by the position specifying means and / or the target position specified by the target position specifying means Are arranged.

The image processing device of the fourth aspect of the present invention is the third aspect of the present invention, wherein all of the three-dimensional model disposed in the virtual space is disposed in the objective background image, and the objective camera is disposed in the objective foreground image. A partial image which is a part of the three-dimensional model arranged in the virtual space according to the objective virtual camera position specified by the position specifying means and / or the target position specified by the target position specifying means Are arranged.

An image processing apparatus according to a fifth aspect of the present invention is the fourth aspect, wherein the virtual space image acquiring unit extracts the partial image from the objective background image.

An image processing apparatus according to the sixth aspect of the present invention is the fifth aspect, wherein the virtual space image acquiring unit extracts the partial image by transmitting a specific color from the objective background image. .

An image processing apparatus according to a seventh aspect of the present invention is the image processing apparatus according to any one of the first to sixth aspects, wherein the physical space image acquiring unit extracts a specific image including the object from the physical space image captured by the imaging device. It is characterized by

An image processing apparatus according to an eighth aspect of the present invention is the seventh aspect, wherein the physical space image acquiring unit extracts the specific image by transmitting a specific color from the physical space image captured by the imaging device. And a transparent processing unit.

An image processing apparatus according to a ninth aspect of the present invention is the image processing apparatus according to any one of the first to eighth aspects, wherein a head mount display is connected to the generation device, and the generation device determines the position and orientation of the head mount display in real space. Subjective camera position specifying means for repeatedly specifying the position of the subjective virtual camera in the virtual space and the position of the subjective virtual camera in the virtual space repeatedly; Subjective virtual space image acquiring means for generating a subjective virtual space image with the subjective virtual camera position specified by the means as a viewpoint, and HMD display control means for controlling to display the subjective virtual space image It features.

An image processing apparatus according to a tenth aspect of the present invention is the ninth aspect, wherein the subjective virtual space image acquiring unit controls to display a camera object at the objective virtual camera position identified by the objective camera position identifying unit. It is characterized by

An image processing apparatus according to an eleventh aspect of the present invention is the image processing apparatus according to any one of the first to tenth aspects, wherein the combining apparatus is a video mixer, a video switcher or other video switching apparatus for processing one or more images. Do.

A control method of an image processing apparatus according to a twelfth aspect of the present invention is an image processing apparatus including: a generation device generating a virtual space in which a three-dimensional model is arranged; and a synthesis device communicably connected to an imaging device The objective camera position specifying means of the generation device repeatedly acquires the position and the orientation of the imaging device in the real space, and the objective virtual camera position which is the position and the orientation of the virtual camera in the virtual space is The objective camera position specifying step of repeatedly specifying, and the target position specifying means of the generating device repeatedly acquire the position and the direction of the target in the real space repeatedly, and repeatedly specify the target position which is the position and the direction of the target in the virtual space Target position specifying step, and placement means of the generation device, the third order according to the objective virtual camera position and / or the target position Arranging the model in the virtual space, moving the three-dimensional model to arrange the virtual space according to the movement of the imaging device and / or the object, and acquiring a virtual space image of the generating device Virtual space image acquiring step of generating one or a plurality of virtual space images by using the three-dimensional model arranged in the virtual space as a viewpoint with the objective virtual camera position specified by the objective camera position specifying means Transmitting means of the generating device transmits the virtual space image generated by the virtual space image acquiring means to the combining device, and the real space image acquiring means of the combining device generates the virtual space image A receiving step of receiving the virtual space image transmitted from the device, and a real space image acquiring unit of the combining device imaging the physical space including the object by the imaging device. Space acquisition step of acquiring a real space image generated by the method, and a mixed reality image acquisition means of the combining device generating a mixed reality image by superimposing the virtual space image and the real space image And an image acquisition step.

A program according to a thirteenth aspect of the present invention executes a control method of an image processing apparatus including a generating device generating a virtual space in which a three-dimensional model is arranged, and a synthesizing device communicably connected to the imaging device and processing the captured image An objective camera position that repeatedly acquires the position and orientation of the imaging device in the real space, and repeatedly specifies the objective virtual camera position that is the position and orientation of the virtual camera in the virtual space; Specifying means, target position specifying means for repeatedly obtaining the position and direction of the target in the real space, and repeatedly specifying the target position which is the position and the direction of the target in the virtual space, the objective virtual camera position and / or the According to the target position, the three-dimensional model is arranged in the virtual space, and the imaging device and / or the transfer of the target are performed. According to the arrangement means for moving the three-dimensional model to arrange in the virtual space, and the objective virtual camera position specified by the objective camera position specifying means for the three-dimensional model arranged in the virtual space Virtual space image acquiring means for generating one or a plurality of virtual space images as a viewpoint, and transmitting means for transmitting the virtual space image generated by the virtual space image acquiring means to the synthesizing device, the synthesizing device A receiving unit for receiving the virtual space image transmitted from the generating device; a physical space image acquiring unit for acquiring a physical space image generated by imaging the physical space including the object by the imaging device; It is characterized in that the virtual space image and the real space image are superimposed to generate a mixed reality image, thereby functioning as mixed reality image acquisition means.

FIG. 1 is an external view showing an image processing apparatus 1 to which a first embodiment of the present invention is applied and a state in which it is used. FIG. 2 is a diagram showing a hardware configuration of the image processing apparatus 1. FIG. 3 is a diagram showing a functional configuration of the image processing apparatus 1. FIG. 4 is a flowchart showing the flow of processing of the image processing apparatus 1. FIG. 5 is a diagram showing a table storing the position and orientation of each device.

FIG. 6 is a diagram showing the state of image composition. FIG. 7 is an external view showing an image processing apparatus 101 to which the second embodiment of the present invention is applied and a state in which the image processing apparatus 101 is used. FIG. 8 is a diagram showing a functional configuration of the image processing apparatus 101. As shown in FIG. FIG. 9 is a diagram showing the state of image composition. FIG. 10 is a diagram showing the state of image composition. FIG. 11 is a diagram showing the state of image composition.

First Embodiment of the Present Invention
Hereinafter, an image processing apparatus 1 of the present invention will be described in detail as a first embodiment of the present invention based on the drawings. FIG. 1 is an external view showing an image processing apparatus 1 to which a first embodiment of the present invention is applied and a state in which it is used. FIG. 2 is a diagram showing a hardware configuration of the image processing apparatus 1. FIG. 3 is a diagram showing a functional configuration of the image processing apparatus 1. FIG. 4 is a flowchart showing the flow of processing of the image processing apparatus 1. FIG. 5 is a diagram showing a table storing the position and orientation of each device. FIG. 6 is a diagram showing the state of image composition.

In addition, as shown in FIG. 1, the arrow U direction is taken as an upward direction, and the arrow D direction which is the reverse direction is demonstrated as a downward direction. The following description will be given assuming that the arrow L direction which is one direction orthogonal to the vertical direction in the horizontal plane is the left direction, and the arrow R direction which is the other direction orthogonal to the right direction. As shown in FIG. 1, the arrow F direction, which is the upper direction orthogonal to the vertical direction and the left-right direction, will be referred to as the forward direction, and the arrow B direction as the lower direction will be described as the backward direction. Top side, Bottom side, Front side, Back side, Left side and Right side are plane, bottom, front, back, left, right Explain as.

[Image processing apparatus 1]
The image processing apparatus 1 is a so-called MR system that provides a virtual game based on virtual space images to the player P and provides a mixed reality image to the visitor V. As shown in FIG. 1, the image processing apparatus 1 includes a generating device 10, a combining device 20, an HMD 30, a first controller 40, a second controller 50, an objective camera 60, a chroma key curtain 70, a display 80 and a base station 90. It is done. In addition, regarding the technology of virtual reality, detailed description will be omitted because the prior art is used.

[Generation device 10]
The generation device 10 is a general-purpose device equipped with an operating system. The generation device 10 is a device that generates an objective background image 601 and an objective foreground image 602 to be combined by the combining device 20 based on the information on the position and orientation of each device. In addition, the generation device 10 is also a device that separately transmits the objective background image 601 and the objective foreground image 602 to the combining device 20 as a plurality of video sources. As shown in FIG. 3, the synthesizing device 20, the HMD 30, the first controller 40 and the second controller 50 are connected to the generating device 10. The generation device 10 is communicably connected to the synthesizer 20, the HMD 30, the first controller 40, and the second controller 50 in a wired or wireless manner. The generation device 10 generates an image of a virtual space (hereinafter, virtual space image) in accordance with the position and orientation of the HMD 30, the first controller 40, and the second controller 50, and transmits the image to the HMD 30.

[Combining apparatus 20]
The combining device 20 is a video mixer, a video switcher, and other video switching devices, and is a device that processes the real space image 603 and combines the mixed reality image 604. The synthesizing device 20 is also a device that separately receives the objective background image 601 and the objective foreground image 602 separately transmitted from the generating device 10 as a plurality of video sources. As shown in FIG. 3, the synthesizing device 20 is connected to the generating device 10, the objective camera 60, and the display 80. The synthesizing device 20 is connected to the generating device 10, the objective camera 60, and the display 80 in a data communication manner in a wired or wireless manner. The synthesizing device 20 receives the real space image 603 captured by the objective camera 60, transmits a predetermined color (such as blue or green) of the real space image 603, and performs keying to the objective transparent reality space image 603 ′. Generate The synthesizing device 20 synthesizes an objective transparent reality space image 603 ′ which is a plurality of video sources, an objective background image 601 generated by the generating device 10, and an objective foreground image 602, and transmits the synthesized image to the display 80.

[HMD30]
The HMD 30 is a so-called head mounted display. As shown in FIG. 1, the HMD 30 is a display mounted on the head of the player P, and includes a right-eye video camera, a left-eye video camera, a right-eye display, and a left-eye display. The HMD 30 has a frame rate of 90 fps (frames per second). The HMD 30 can transmit the real space image captured by the right-eye video camera and the left-eye video camera to the generation device 10. The HMD 30 receives the virtual space image transmitted from the generation device 10 and displays it on the right-eye display and the left-eye display.

The HMD 30 is displayed on the right-eye display and the left-eye display at a position immediately in front of the eyes of the player P, with the images slightly shifted between the two eyes. For this reason, although the images displayed on the right-eye display and the left-eye display are flat, they can be seen three-dimensionally. The real space image captured by the HMD 30 and the virtual space image displayed may be a moving image or a still image captured at a predetermined interval. An HMD sensor 31 for detecting the position and the orientation of the HMD 30 is attached to the HMD 30. The HMD sensor 31 can pick up the infrared rays blown by the base station 90.

[First controller 40, second controller 50]
The first controller 40 and the second controller 50 are devices for operating all or part of an object to be a three-dimensional model disposed in a virtual space. As shown in FIG. 1, a first controller sensor 41 for detecting the position and orientation of the first controller 40 is attached to the first controller 40. The first controller sensor 41 can pick up the infrared rays blown by the base station 90. A second controller sensor 51 for detecting the position and the orientation of the second controller 50 is attached to the second controller 50. The second controller sensor 51 can pick up the infrared rays blown from the base station 90.

[Objective camera 60]
The objective camera 60 is an imaging device that captures a real space by moving manually or automatically. As shown in FIG. 1, the objective camera 60 is disposed at a predetermined position so as to capture an image of the player P who plays a game inside the chroma key curtain 70. The physical space image captured by the objective camera 60 is transmitted to the combining device 20. In the present embodiment, the physical space image captured by the objective camera 60 is a moving image, but may be a still image captured at a predetermined interval. An objective camera sensor 61 for detecting the position and the direction of the objective camera 60 is attached to the objective camera 60.

[Chroma key curtain 70]
The chroma key curtain 70 is a curtain for covering a room where a virtual game is to be experienced. As shown in FIG. 1, the chroma key curtain 70 is arranged to cover the player P who experiences the mixed reality. In this example, in order to be able to image the whole body of the player P with the objective camera 60, the objective camera 60 is disposed so as to cover a plane other than the rear surface, the bottom, the front, the left side, and the right side. In this example, the chroma key curtain 70 uses a green color which is complementary to the color of human skin in order to cover the person who is the player P. A plurality of colors may be used for the chroma key curtain 70, and the color of the object to be transmitted may be changed depending on the color and the density.

[Display 80]
The display 80 is a display device capable of viewing the mixed reality image 704 including the same virtual space as the player P who is experiencing the virtual game. As shown in FIG. 1, the display 80 is a thin and large liquid crystal television. The display 80 has a frame rate of 60 fps (frames per second). In this example, a thin and large liquid crystal television is used so that many visitors V can view the mixed reality image 704 including the same virtual space as the player P who is experiencing the virtual game. Other display devices such as a small television, a smartphone, and a tablet may be used.

[Base station 90]
The base station 90 is a device for detecting the position and orientation of the HMD sensor 31, the first controller sensor 41, the second controller sensor 51, and the objective camera sensor 61. The base station 90 irradiates infrared rays in a predetermined pattern in the range of about 120 degrees in the vertical and horizontal directions, and the HMD sensor 31, the first controller sensor 41, the second controller sensor 51, and the objective camera sensor 61 pick up the infrared rays. These positions and orientations are detected.

As shown in FIG. 1, two base stations 90 are disposed inside the chroma key curtain 70. However, two or more base stations 90 may be arranged so that the HMD sensor 31, the first controller sensor 41, the second controller sensor 51, and the objective camera sensor 61 can be detected under any circumstances. In addition, the base station 90 may be disposed outside the chroma key curtain 70 so as to be inconspicuous.

[Hardware Configuration of Generation Device 10]
The hardware configuration of the generation device 10 will be described based on FIG. The CPU 10 a is a central processing unit that controls devices and controllers connected to the system bus 11. The ROM 10 b or the external memory 10 j is a memory in which is stored a BIOS (Basic Input / Output System) which is a control program of the CPU 10 a, an operating system, various programs to be described later to realize functions executed by various devices, It is an apparatus. The RAM 10 c is a device that functions as a main memory, a work area, and the like of the CPU 10 a. The CPU 10a reads various programs and the like necessary for execution of processing from the RAM 10c, and realizes various operations by executing the same.

The input controller 10 f is a device that controls inputs from the first controller 40, the second controller 50, the keyboard, and the mouse or other input device 10 i. The video controller 10 h is a device that controls display on other display devices such as the right eye display and the left eye display included in the HMD 30. The right-eye display and the left-eye display are output using, for example, a high-definition multimedia interface.

The memory controller 10g is a card type connected via an adapter to a hard disk, flexible disk, or PCMCIA card slot that stores a boot program, browser software, various applications, font data, player P files, editing files, various data, etc. It is an apparatus for controlling access to an external memory 10 j such as a memory. The communication I / F controller 10 e is for connecting / communicating with an external device via a network, and is a device that executes communication control processing.

The general-purpose bus 10d is a device for capturing an image from the right-eye video camera and the left-eye video camera of the HMD 30. The general-purpose bus 10d is connected as an external input terminal to the right-eye video camera and left-eye video camera by, for example, a universal serial bus. Programs and the like used to execute various processes by the generation device 10 are recorded in the external memory 10 j, and are executed by the CPU 10 a by being loaded into the RAM 10 c as necessary. The definition file and various information tables used by the program executed by the generation device 10 are stored in the external memory 10 j.

[Hardware Configuration of Synthesizer 20]
The hardware configuration of the synthesizing device 20 will be described based on FIG. The hardware configuration of the combining device 20 is substantially the same as the hardware configuration of the generating device 10, and thus the detailed description of the same hardware configuration is omitted. Unlike the generating device 10, the right-eye display and the left-eye display are not connected to the video controller 20h of the synthesizing device 20. A display 20 k is connected to the video controller 20 h of the synthesizing device 20. The base station 90 is not connected to the synthesizer 20. The right-eye video camera and the left-eye video camera are not connected to the general-purpose bus 20 d of the synthesizing device 20. An objective camera 60 is connected to the general-purpose bus 20 d of the synthesizing device 20.

[Functional Configuration Diagram of Generating Device 10]
As illustrated in FIG. 3, the generation device 10 includes, as functional units, a communication control unit 301, an objective camera position specifying unit 302, an object position specifying unit 303, an HMD real space image acquisition unit 304, an arrangement unit 305, and a subjective virtual space image acquisition. Means 306, HMD display control means 307, virtual space image acquisition means 308, objective background image transmission means 309, and objective foreground image transmission means 310 are provided.

[Communication control means 301]
The communication control unit 301 has a function of transmitting / receiving various information to / from the HMD 30 capable of communicating with the generating device 10, the base station 90, and the combining device 20. The communication control unit 301 transmits and receives information to and from these devices through the video controller 10h, the communication I / F controller 10e, the general-purpose bus 10d, and the like.

[Objective camera position identification means 302]
The objective camera position specifying means 302 has a function of acquiring information indicating the position and orientation of the objective camera 60 in the real space. The objective camera position specifying means 302 repeatedly acquires the position and the orientation of the objective camera 60 in the real space, and repeatedly specifies the objective virtual camera position which is the position and the orientation of the virtual camera in the virtual space. The position and the orientation of the objective camera 60 are detected at the timing when the objective camera sensor 61 picks up the infrared rays blown from the base station 90.

[Target position specifying means 303]
The target position specifying means 303 has a function of acquiring information indicating the position and orientation of the HMD 30, the first controller 40 and the second controller 50 in the real space. The target position specifying means 303 repeatedly acquires the position and the direction of the target in the real space, and repeatedly specifies the target position which is the position and the direction of the target in the virtual space. The position and the orientation of the HMD 30 are detected at the timing when the HMD sensor 31 picks up the infrared rays blown off from the base station 90. The position and the orientation of the first controller 40 are detected at the timing when the first controller sensor 41 picks up the infrared rays blown off from the base station 90. The position and the orientation of the second controller 50 are detected at the timing when the second controller sensor 51 picks up the infrared rays blown off from the base station 90.

[HMD real space image acquisition means 304]
The HMD real space image acquisition unit 304 has a function of repeatedly acquiring real space images captured by the right-eye video camera and the left-eye video camera of the HMD 30.

[Placement means 305]
The arranging unit 305 has a function of arranging an object image (for example, a background image, a weapon image, a shield image) which is a three-dimensional model in a virtual space. The arranging unit 305 arranges the object image based on the position and the orientation acquired by the objective camera position specifying unit 302, the target position specifying unit 303, and the like. The object image is stored in the external memory 10 j or the like of the generation device 10, and these are read and acquired as appropriate. Determine the position and orientation in the virtual space based on the position and orientation of the HMD 30 and the objective camera 60, generate an object image (background image) when viewed from this position and orientation, and place this in the virtual space .

Based on the position and orientation of the first controller 40, the position and orientation in the virtual space are determined, and an object image (weapon image) when viewed from the position and orientation is generated and arranged in the virtual space. Based on the position and orientation of the second controller 50, the position and orientation in the virtual space are determined, and an object image (shield image) when viewed from this position and orientation is generated and acquired.

[Subjective virtual space image acquisition means 306]
The subjective virtual space image acquisition unit 306 has a function of generating a subjective virtual space image from the virtual space generated by the arrangement unit 305.

[HMD display control means 307]
The HMD display control means 307 has a function of performing display control of the right-eye display and the left-eye display of the HMD 30 connected to the generation device 10. The HMD display control means 307 displays the subjective virtual space image generated by the subjective virtual space image acquisition means 306 on the right eye display and the left eye display.

[Virtual space image acquisition means 308]
The virtual space image acquisition unit 308 has a function of generating a virtual space image by the generation device 10. The virtual space image acquisition unit 308 sets an objective background image 601 and an objective foreground image 602, which are virtual space images, with the object image arranged in the virtual space as the viewpoint and the objective virtual camera position specified by the objective camera position specifying unit 302. Generate As shown in FIG. 6, in the objective background image 601, all object images (for example, background image, weapon image, shield image) arranged in the virtual space are arranged.

As shown in FIG. 6, the objective foreground image 602 is arranged in the virtual space according to the objective virtual camera position specified by the objective camera position specifying means 302 and / or the target position specified by the target position specifying means 303. A part of the object image (eg, a weapon image, a shield image) is placed. The virtual space image acquisition unit 308 determines whether to arrange an object image (for example, a weapon image) in the objective foreground image 602 based on the positions and orientations of the HMD 30, the objective camera 60, and the first controller 40. The virtual space image acquisition unit 308 determines whether to arrange an object image (for example, a shield image) in the objective foreground image 602 based on the positions and orientations of the HMD 30, the objective camera 60, and the second controller 50. The objective foreground image 602 is an objective foreground image 602 'which is keyed and transmitted except for a part of the arranged object image.

[Objective background image transmission means 309]
The objective background image transmission unit 309 has a function of repeatedly transmitting the objective background image 601 generated by the virtual space image acquisition unit 308 to the combining device 20. The generation device 10 transmits the objective background image 601 to the combining device 20 via a dedicated interface.

[Objective foreground image transmission means 310]
The objective foreground image transmitting means 310 has a function of returning to the synthesizing device 20 the objective foreground image 602 generated by the virtual space image acquiring means 308 or the transmitted objective foreground image 602 ′ to the synthesizing device 20. The generator 10 transmits the objective foreground image 602 'to the synthesizer 20 via a dedicated interface.

[Functional Configuration Diagram of Combining Device 20]
In addition, as a functional unit, the synthesizing device 20 includes a communication control unit 351, a real space image acquisition unit 352, a transmission processing unit 353, an objective background image reception unit 354, an objective foreground image reception unit 355, a mixed reality image generation unit 356 and a display display. A control means 357 is provided.

[Communication control means 351]
The communication control unit 351 is a functional unit that transmits and receives various types of information between the generating device 10 capable of communicating with the combining device 20 and the objective camera 60. The communication control unit 301 transmits and receives information to and from these devices through the communication I / F controller 10 e, the general-purpose bus 10 d, and the like.

[Real Space Image Acquisition Means 352]
The physical space image acquisition unit 352 has a function of repeatedly acquiring the physical space image 603 captured by the objective camera 60.

[Transparency processing means 353]
The transmission processing unit 353 has a function of transmitting a predetermined color included in the physical space image 603 acquired by the physical space image acquisition unit 352 to generate an objective transmission reality space image 603 ′. Since the color of the chroma key curtain 70 is green, the transmission processing unit 353 sets the opacity of the green pixel to “0” among the pixels included in the acquired real space image 603. The opacity does not necessarily have to be “0”, but the effect of the present invention can be further improved by completely transmitting.

[Objective background image receiving means 354]
The objective background image receiving unit 354 has a function of repeatedly receiving the objective background image 601 transmitted from the objective background image transmitting unit 309. The combining device 20 receives the objective background image 601 from the generating device 10 via a dedicated interface.

[Objective foreground image receiving means 355]
The objective foreground image receiving unit 355 has a function of repeatedly receiving the objective foreground image 602 transmitted from the objective foreground image transmitting unit 310 or the transmitted objective foreground image 602 ′. The synthesizer 20 receives the objective foreground image 602 'from the generator 10 via a dedicated interface.

[Mixed Reality Image Generating Means 356]
The mixed reality image generation means 356 has a function of combining the mixed reality image 604 by superposing the objective background image 601, the objective transmission reality space image 603 'and the transmitted objective foreground image 602'. As shown in FIG. 6, the mixed reality image 604 is superimposed from the bottom in the order of the objective background image 601, the objective transmission reality space image 603 'and the transmitted objective foreground image 602'.

[Display Display Control Means 357]
The display display control means 357 has a function of performing display control of the display 80 connected to the combining device 20. The display display control means 357 displays the mixed reality image 604 synthesized by the mixed reality image generating means 356 on the display 80.

[Flow of processing]
Next, a series of processes performed by the image processing apparatus 1 according to the embodiment of the present invention will be described using the flowchart shown in FIG.

[Flow of processing of generation device 10]
A series of processes performed by the generation device 10 of the image processing apparatus 1 includes an objective camera position identification step 401, an object position identification step 402, an HMD real space image acquisition step 403, an arrangement step 404, a subjective virtual space image acquisition step 405, and an HMD. It comprises a display control step 406, a virtual space image acquisition step 407, an objective background image transmission step 408, an objective foreground image transmission step 409 and an end step 410.

[Objective camera position identification process S401]
The CPU 10a acquires information indicating the position and orientation of the objective camera 60 in the real space, and stores the information in the RAM 10c or the like. The CPU 10a uses the objective camera position specifying means 302 to acquire information indicating the position and the orientation of the objective camera 60 in the real space at the timing of picking up the infrared rays blown from the base station 90. As shown in FIG. 5, the position (coordinates) and direction (vector) of the objective camera 60 are represented by XYZ coordinates with a predetermined place in the real space as the origin and a vector using the XYZ coordinates, and an objective camera table It is stored in 560.

[Target position identification process S402]
The CPU 10a acquires information indicating the position and orientation of the HMD 30 in the real space, and stores the information in the RAM 10c or the like. The CPU 10a acquires the information indicating the position and the orientation of the HMD 30 in the real space at the timing of picking up the infrared rays skipped from the base station 90 using the target position specifying means 303. As shown in FIG. 5, the position (coordinates) and the direction (vector) of the HMD 30 are represented by XYZ coordinates with a predetermined place in the real space as the origin and a vector using the XYZ coordinates, and are stored in the HMD table 530 Be done. Similarly, the CPU 10 a acquires the position (coordinates) and the orientation (vector) of the HMD 30 of the first controller 40 and the second controller 50 using the base station 90, and the first controller table 540 and the second controller table 550. Store in

[HMD real space image acquisition step S403]
The CPU 10a repeatedly acquires the physical space image transmitted from the right-eye video camera and the left-eye video camera of the HMD 30 using the HMD physical space image acquiring unit 304, and stores the same in the RAM 10c or the like. As the right-eye video camera and the left-eye video camera, a video camera corresponding to the right eye of the player P and a video camera corresponding to the left eye are prepared, and from these, real space images for the right eye and the left eye are repeatedly acquired.

[Placement process S404]
The CPU 10a arranges an object image, which is a three-dimensional model, in the virtual space using the arranging means 305, and stores it in the RAM 10c or the like. The CPU 10a reads the position (coordinates) and direction (vector) of the objective camera 60 stored in the objective camera table 560, acquired in the objective camera position specifying step S401, from the RAM 10c or the like, and corresponds to the object image (for example, , Background image) in the virtual space and stored in the RAM 10 c or the like.

The CPU 10a reads the position (coordinates) and direction (vector) of the HMD 30 acquired in the target position specifying step S402 and stored in the HMD table 530 from the RAM 10c or the like, and an object image (for example, background image) corresponding thereto. Are arranged in the virtual space and stored in the RAM 10 c or the like. The CPU 10a reads the position (coordinates) and the orientation (vector) of the first controller acquired in the target position specifying step S402 and stored in the first controller table 540 from the RAM 10c or the like, and corresponds to the object image (for example, , Weapon images) are arranged in the virtual space and stored in the RAM 10 c or the like.

The CPU 10a reads the position (coordinates) and the orientation (vector) of the second controller acquired in the target position specifying step S402 and stored in the second controller table 550 from the RAM 10c or the like, and corresponds to the object image (for example, , Shield image) in the virtual space and stored in the RAM 10 c or the like.

[Subjective virtual space image acquisition process 405]
The CPU 10a generates a subjective virtual space image using the subjective virtual space image acquisition means 306, and stores it in the RAM 10c or the like. The CPU 10a reads the position (coordinates) and the direction (vector) of the HMD 30 acquired in the target position specifying step S402 and stored in the HMD table 530 from the RAM 10c or the like, and generates a subjective virtual space image corresponding thereto. , RAM 10c and the like. The CPU 10a generates a subjective virtual space image for the right eye and a subjective virtual space image for the left eye to display the subjective virtual space on the right eye display and the left eye display of the HMD 30.

[HMD display control step S406]
The CPU 10a reads the subjective virtual space image generated in the subjective virtual space image acquisition step S405 from the RAM 10c or the like, and displays it on the right eye display and the left eye display of the HMD 30 through the video controller 10h. Subjective virtual space images stored in the RAM 10c etc. are for the right eye and for the left eye. Therefore, the HMD display control means 307 is used to control to display the right-eye subjective virtual space image on the right-eye display, and to control to display the left-eye subjective virtual space real image on the left-eye display.

[Virtual Space Image Acquisition Step S407]
The CPU 10a generates a virtual space image using the virtual space image acquisition unit 308, and stores the virtual space image in the RAM 10c or the like. The CPU 10a reads the position (coordinates) and direction (vector) of the objective camera 60 stored in the objective camera table 560, acquired in the objective camera position specifying step S401, from the RAM 10c or the like, and a virtual space image corresponding to this. It is generated and stored in the RAM 10c or the like. The CPU 10a generates an objective background image and an objective foreground image as virtual space images, and stores them in the RAM 10c or the like.

The CPU 10a generates and stores the objective background image 601 in a state where all of the object images arranged in the virtual space (background image, weapon image, shield image) are arranged. The CPU 10a selects a part of the object image arranged in the virtual space according to the objective virtual camera position specified by the objective camera position specifying means 302 and / or the target position specified by the target position specifying means 303 (a weapon image, The objective foreground image 602 is generated and stored in a state where a partial image which is a shield image) is arranged.

The CPU 10a uses the virtual space image acquisition means 308 to determine whether or not to arrange an object image (weapon image) in the objective foreground image 602 based on the position and orientation of the HMD 30, the objective camera 60 and the first controller 40. To generate an objective foreground image 602. The CPU 10a uses the virtual space image acquisition means 308 to determine whether to arrange an object image (shield image) in the objective foreground image 602 based on the position and orientation of the HMD 30, the objective camera 60 and the second controller 50. To generate an objective foreground image 602. The CPU 10a transmits and keys a portion other than a part of the object image to be arranged, and stores it as a transmitted objective foreground image 602 '.

[Objective background image transmission step S408]
The CPU 10a reads the generated objective background image from the RAM 10c or the like using the objective background image transmission means 309, and transmits it to the synthesizing device 20. The CPU 10 a transmits the objective background image 601 from the generating device 10 to the combining device 20 via a dedicated interface for transmitting the objective background image 601.

[Objective foreground image transmission step S409]
The CPU 10 a reads the generated objective foreground image 602 or the transmitted objective foreground image 602 ′ from the RAM 10 c or the like using the objective foreground image transmission unit 310, and transmits the read object foreground image 602 or the transmitted objective foreground image 602 ′ to the combining device 20. The CPU 10a transmits the objective foreground image 602 'from the generating device 10 to the synthesizing device 20 via a dedicated interface for transmitting the objective foreground image 602'.

[End step S413]
The CPU 10a determines whether or not an instruction to end the process of displaying the virtual game is given to the player P wearing the HMD 30. If the CPU 10a determines that an end instruction has been issued, the series of processing ends. If there is no end instruction, the CPU 10a returns the process to the objective camera position specifying step S401, and repeats a series of processes until the end instruction is given.

[Flow of Processing of Synthesizer 20]
A series of processes performed by the combining device 20 of the image processing device 1 includes a real space image acquisition step 451, a transmission processing step 452, an objective background image reception step 453, an objective foreground image reception step 454, a mixed reality image generation step 455, a display It consists of a display control step 456 and an end step 457.

[Real Space Image Acquisition Step S451]
The CPU 20a acquires the physical space image 603 transmitted from the objective camera 60 using the physical space image acquisition means 352, and stores it in the RAM 20c or the like.

[Permeable treatment step S452]
The CPU 20a transmits a predetermined color included in the physical space image 603 stored in the RAM 20c to generate an objective transmission reality space image 603 ′ and stores the objective transparent reality space image 603 ′ in the RAM 20c or the like. The CPU 20a reads the physical space image 603 from the RAM 10c or the like, transmits a predetermined color of the physical space image 603 using the transmission processing means 353, generates an objective transparent reality space image 603 ', and stores it in the RAM 20c or the like. . In the real space image 603, since the portion other than the player P wearing the HMD 30 is green of the chroma key curtain 70, the real space image 603 is transmitted by changing the alpha value of this color. As shown in FIG. 6, the objective transparent reality space image 603 ′ is an image in which only the player P is extracted, and a portion covered by the chroma key curtain 70 other than the player P is transmitted.

[Objective background image reception process S453]
The CPU 20a uses the objective background image transmission means 309 to receive information indicating the objective background image 601 transmitted from the generation device 10, and stores the information in the RAM 20c or the like.

[Objective foreground image receiving step S454]
The CPU 20a uses the objective foreground image transmission means 310 to receive the objective foreground image 602 transmitted from the generation device 10 or the information indicating the transmitted objective foreground image 602 ', and stores the information in the RAM 20c or the like.

[Mixed reality image generation process S455]
The CPU 20a reads out the objective background image 601 stored in the RAM 20c etc., the objective transparent reality space image 603 'and the transmitted objective foreground image 602'. Then, the mixed reality image 604 is generated by superimposing the objective background image 601, the objective transmission reality space image 603 ′ and the transmitted objective foreground image 602 ′ using the mixed reality image generation means 356, and the mixed reality image 604 is generated. Remember. The mixed reality image is superimposed from the bottom in the order of the objective background image 601, the objective transmission reality space image 603 'and the transmitted objective foreground image 602', and is stored in the RAM 20c or the like.

[Display control step S456]
The CPU 20a reads the mixed reality image 604 generated in the mixed reality image generation step S455 from the RAM 20c or the like, and displays it on the display 80 through the video controller 20h using the display display control means 357.

[End step S457]
The CPU 20a determines whether or not an instruction to end the process of displaying the virtual game is given to the player P wearing the HMD 30. If the CPU 20a determines that an end instruction has been issued, the series of processing ends. If there is no end instruction, the CPU 20a returns the process to the physical space image acquisition step S451, and repeats a series of processes until the end instruction is given.

[First virtual game]
Next, a first virtual game using the image processing device 1 will be described based on FIGS. 3 and 7. FIG. 7 shows a process of generating a mixed reality image. In the first virtual game, the player P wears the HMD 30 on his head, holds the first controller 40 on his right hand, holds the second controller 50 on his left hand, in the space surrounded by the chroma key curtain 70, Is a game to defeat monsters. In this first virtual game, the visitor V can grasp the state of the player P by performing chroma key composition while photographing the player P with one objective camera 60 that moves manually or automatically. In this example, as shown in FIG. 7, the moment when the player P faces the objective camera 60 will be described as an example.

In this state, the objective camera position specifying unit 302 acquires the position and the direction of the objective camera 60, and the target position specifying unit 303 acquires the positions and the directions of the HMD 30, the first controller 40, and the second controller 50 S401, S402). The arranging unit 305 arranges the object image in the virtual space based on the acquired position and orientation of the objective camera 60, the HMD 30, the first controller 40, and the second controller 50 (S404). The subjective virtual space image acquiring unit 306 generates a subjective virtual space image (S405), and displays the subjective virtual space image on the HMD 30 (S406).

The virtual space image acquisition means 308 generates an objective background image 601 and an objective foreground image 602 as a virtual space image, as shown in FIG. 6, based on the position and orientation of the objective camera 60 (S407). In this example, the second controller 50 held by the player P is between the HMD 30 and the objective camera 60. Therefore, only the shield image is displayed on the objective foreground image 602. The virtual space image acquisition means 308 transmits the objective foreground image 602, generates keying and generates the transmitted objective foreground image (S407). The objective background image transmission unit 309 transmits the objective background image 601 from the generation device 10 to the combining device 20 via the dedicated interface (S408). The objective foreground image transmission means 310 transmits the transmitted objective foreground image 602 'from the generating device 10 to the synthesizing device 20 via the dedicated interface (S409).

The physical space image acquisition means 352 acquires the physical space image 603 transmitted from the objective camera 60 (S 451), and the transmission processing means 353 transmits a predetermined color of the physical space image to generate an objective transmission real space image 603 ′ (S452). The objective background image receiving unit 354 receives the objective background image 601 from the generation device 10 to the combining device 20 via the dedicated interface (S 453). The objective foreground image receiving means 355 receives the transmitted objective foreground image 602 'from the generating device 10 to the synthesizing device via the dedicated interface (S454). As shown in FIG. 6, the mixed reality image generation unit 356 generates a mixed reality image 604 by superimposing the objective background image 601, the objective transmission reality space image 603 ′ and the transmitted objective foreground image 602 ′ in this order from below. (S456).

In this example, since the second controller 50 held by the player P is between the HMD 30 and the objective camera 60, only the shield image is displayed on the objective foreground image 602, and the player P's The shield image is arranged to overlap the image. As shown in FIG. 6, the display display control means 357 displays the mixed reality image 604 on the display 80.

When the first controller 40 possessed by the player P is between the HMD 30 and the objective camera 60, a weapon image is displayed on the objective foreground image 602, so the image of the player P of the objective transmission reality space image 603 ' Is placed so that the weapon image overlaps the. In addition, when the first controller 40 and the second controller possessed by the player P are behind the HMD 30 and the objective camera 60, a weapon image and a shield image are displayed only on the objective background image 601, so an objective transmission reality space is obtained. The weapon image and the shield image do not overlap in front of the image of the player P of the image 603 ′, and are placed behind.

As described above, the depth which can not be expressed by ordinary chroma key composition can be expressed by the objective background image 601, the objective transparent reality space image 603 'and the transmitted objective foreground image 602'. That is, a predetermined image can be arranged in front of the player P extracted by chroma key combination. In addition, since the generation device 10 mainly performs processing of constructing a virtual space, and the synthesizing device 20 mainly performs composition and display of the generated image, the smooth mixed reality image 704 without a processing omission is a visitor V. Can see.

Second Embodiment of the Present Invention
Hereinafter, an image processing apparatus 101 of the present invention will be described in detail as a second embodiment of the present invention based on the drawings. The first virtual game of the first embodiment uses only one objective camera 60 that moves manually or automatically, but the second virtual game of the second embodiment uses three cameras that move manually or automatically (The objective camera 60, the objective camera 160 and the objective camera 260) are used. The second virtual game will not be described in detail for the same part as the first virtual game, and different parts will be described.

FIG. 7 is an external view showing an image processing apparatus 101 to which the second embodiment of the present invention is applied and a state in which the image processing apparatus 101 is used. FIG. 8 is a diagram showing a functional configuration of the image processing apparatus 101. As shown in FIG. FIG. 9 is a diagram showing the state of image composition. FIG. 10 is a diagram showing the state of image composition. FIG. 11 is a diagram showing the state of image composition. As shown in FIG. 7, the second virtual game is different from the first virtual game, and in addition to the objective camera 60, an objective camera 160 and an objective camera 260 are prepared. As shown in FIG. 8, the second virtual game differs from the first virtual game in that the generating device 10 has an objective camera selection control means 301 'and the combining device 20 has an objective camera identification control means 351'.

[Objective camera selection control means 301 ']
The objective camera selection control means 301 ′ has a function of controlling which one of the objective camera 60, the objective camera 160 and the objective camera 260 is to be selected. The objective camera selection control means 301 ′ selects one of the objective camera 60, the objective camera 160, and the objective camera 260 by the program incorporated in the generation device 10 and / or the input operation of the operator using the generation device 10, or the like. . The objective camera selection control unit 301 ′ transmits information of the selected camera to the combining device 20 after storing the information in the generating device 10.

[Objective camera identification control means 351 ']
The objective camera identification control means 351 ′ has a function of identifying the camera selected by the generation device 10. Based on the camera specified by the objective camera specification control means 351 ′, the real space image acquisition means 352 acquires a real space image. For example, when the objective camera 60 is selected by the objective camera selection control means 301 ′, the generation device 10 generates an objective background image 701 and a transmitted objective foreground image 702 ′ as shown in FIG. Then, an objective transmission reality space image 703 'is generated, and a mixed reality image 704 is generated.

For example, when the objective camera 160 is selected by the objective camera selection control means 301 ′, as shown in FIG. 10, the generation device 10 generates an objective background image 711 and a transmitted objective foreground image 712 ′. Then, an objective transmission reality space image 713 'is generated, and a mixed reality image 714 is generated. For example, when the objective camera 260 is selected by the objective camera selection control means 301 ′, as shown in FIG. 10, the generation device 10 generates an objective background image 721 and a transmitted objective foreground image 722 ′. Then, an objective transparent reality space image 723 'is generated, and a mixed reality image 724 is generated. As described above, in the second virtual game, the mixed reality image is displayed switchably from a plurality of cameras, so that visual effects can be further enhanced.

Although various embodiments of the present invention have been described above, the present invention is not limited to this embodiment. It is needless to say that changes can be made without departing from the object and the purpose of the present invention. The synthesizing device 20 is not limited to the video switching device, and may be a general-purpose device equipped with an operating system. The HMD 30 includes the right-eye video camera and the left-eye video camera, but these may not be present or may be one video camera. Although only the mixed reality image 604 is displayed on the display 80, the image which the player P is looking at may be inserted as a wipe at a small corner or the like of the screen. In this case, as shown in FIG. 3 and FIG. 8, the subjective virtual space image may be transmitted to the synthesizing device 20. At this time, the generating device 10 may transmit to the synthesizing device 20 via a dedicated interface for transmitting a subjective virtual space image.

Although the base station 90 has been described as emitting infrared light, the base station 90 may emit infrared light while detecting the position and orientation of each device, or each device emits infrared light. However, those positions and orientations may be detected. Further, the base station 90 irradiates infrared rays, and the HMD sensor 31, the first controller sensor 41, the second controller sensor 51, and the objective camera sensor 61 pick up these to detect their positions and directions. However, if these positions and directions are to be detected, for example, magnetic detection may be performed, or images captured by the HMD 30 or the objective camera 60 may be analyzed to specify the positions and directions.

Although the object of the three-dimensional model existing in the virtual space has been described using a weapon image, a shield image, and a background image, it may be a monster image or a camera image which is a camera object indicating the position of the objective camera 60. The monster image is also disposed at an appropriate position before or after the image of the player P in the mixed reality image 704 viewed by the visitor V, similarly to the weapon image and the shield image. The viewpoints of the subjective virtual space image viewed by the player P with the HMD 30 and the mixed reality image 704 viewed by the visitor V with the display 80 differ according to the position of the HMD 30 and the objective camera 60.

For this reason, the visitor V may see from the display 80 the monster image which can not be seen from the viewpoint of the player P. At this time, when the visitor V calls the player P, the visitor image may be noticed. In this way, the player P and the visitor V can cooperate to advance the game, which increases the interest. The present invention can also be implemented, for example, as a system, an apparatus, a method, a control method, a program or a storage medium. Specifically, the present invention may be applied to a system configured of a plurality of devices, or may be applied to an apparatus configured of a single device.

1: Image processing device 10: Generation device 10a: CPU
10b: ROM
10c: RAM
10d: General-purpose bus 10e: Communication I / F controller 10f: Input controller 10g: Memory controller 10h: Video controller 10i: Input device 10j: External memory 11: System bus 20: Synthesizer 20a: CPU
20c: RAM
20d: general purpose bus 20h: video controller 20k: display 30: HMD
31: HMD sensor 40: first controller 41: first controller sensor 50: second controller 51: second controller sensor 60: objective camera 61: objective camera sensor 70: chroma key curtain 80: display 90: base station

Claims (13)

1. An image processing apparatus comprising: a generating device (10) for generating a virtual space in which a three-dimensional model is arranged; and a synthesizing device (20) communicably connected to the imaging device and processing the captured image,
   The generator (10) is
   Objective camera position specifying means for repeatedly acquiring the position and orientation of the imaging device (60) in the physical space and repeatedly specifying the objective virtual camera position which is the position and orientation of the virtual camera in the virtual space;
   Object position identification means for repeatedly acquiring the position and orientation of the object in the real space and repeatedly specifying the object position which is the position and orientation of the object in the virtual space;
   The three-dimensional model is arranged in the virtual space according to the objective virtual camera position and / or the target position, and the three-dimensional model is moved according to the movement of the imaging device and / or the target to perform the virtual Arrangement means for arranging in space;
   Virtual space image acquisition means for generating one or more virtual space images, with the objective virtual camera position specified by the objective camera position specifying means as a viewpoint, with the three-dimensional model arranged in the virtual space;
   And transmission means for transmitting the virtual space image generated by the virtual space image acquisition means to the combining device,
   The synthesizer (20) is
   Receiving means for receiving the virtual space image transmitted from the generating device;
   A physical space image acquisition unit that acquires a physical space image generated by imaging the physical space including the object by the imaging device;
   An image processing apparatus comprising: mixed reality image acquisition means for generating a mixed reality image by superimposing the virtual space image and the real space image.
2. The image processing apparatus according to claim 1, wherein
   The virtual space image acquisition means
   An image processing apparatus characterized by generating, as the virtual space image, an objective background image arranged after the real space image and an objective foreground image arranged before the real space image.
3. The image processing apparatus according to claim 2, wherein
   The virtual space image acquisition means
   The objective background image and the objective foreground image are generated according to the objective virtual camera position specified by the objective camera position specifying means and / or the target position specified by the target position specifying means. Image processing device.
4. The image processing apparatus according to claim 3, wherein
   In the objective background image, all of the three-dimensional model arranged in the virtual space is arranged,
   The objective foreground image may be placed in the virtual space according to the objective virtual camera position identified by the objective camera position identifying means and / or the object position identified by the object position identifying means. An image processing apparatus characterized in that a partial image which is a part of a two-dimensional model is arranged.
5. The image processing apparatus according to claim 4, wherein
   The virtual space image acquisition means
   An image processing apparatus characterized by extracting the partial image from the objective background image.
6. The image processing apparatus according to claim 5,
   The virtual space image acquisition means
   An image processing apparatus characterized in that the partial image is extracted by transmitting a specific color from the objective background image.
7. The image processing apparatus according to any one of claims 1 to 6, wherein
   The real space image acquisition means
   An image processing apparatus, which extracts a specific image including the object from the physical space image captured by the imaging device.
8. The image processing apparatus according to claim 7, wherein
   The real space image acquisition means
   An image processing apparatus comprising: transmission processing means for extracting the specific image by transmitting a specific color from the physical space image captured by the imaging device.
9. The image processing apparatus according to any one of claims 1 to 8, wherein
   A head mounted display is connected to the generator,
   The generator is (10),
   Subjective camera position specifying means for repeatedly acquiring the position and orientation of the head mounted display (30) in the real space, and repeatedly specifying the position of the subjective virtual camera which is the position and orientation of the subjective virtual camera in the virtual space;
   Subjective virtual space image acquisition means for generating a subjective virtual space image, with the three-dimensional model disposed in the virtual space as a viewpoint, with the subjective virtual camera position specified by the subjective camera position specifying means as a viewpoint;
   And an HMD display control unit configured to control to display the subjective virtual space image.
10. The image processing apparatus according to claim 9,
    The image processing apparatus, wherein the subjective virtual space image acquisition means controls to display a camera object at the objective virtual camera position identified by the objective camera position identification means.
11. The image processing apparatus according to any one of claims 1 to 10, wherein
    The image processing device is characterized in that the combining device is a video mixer, a video switcher, or other video switching device that processes one or more images.
12. A control method of an image processing apparatus comprising: a generation device (10) for generating a virtual space in which a three-dimensional model is arranged; and a combining device (20) communicably connected to an imaging device and processing a captured image ,
    Objective camera position specifying means of the generating device repeatedly acquires the position and orientation of the imaging device (60) in the real space, and repeatedly specifies the objective virtual camera position which is the position and orientation of the virtual camera in the virtual space Camera position identification process,
    Target position specifying step of the target position specifying means of the generation device repeatedly acquiring the position and the direction of the target in the real space repeatedly and repeatedly specifying the target position which is the position and the direction of the target in the virtual space;
    The arrangement unit of the generation device arranges the three-dimensional model in the virtual space according to the objective virtual camera position and / or the object position, and the 3 according to the movement of the imaging device and / or the object. Placing a two-dimensional model and placing it in the virtual space;
    The virtual space image acquisition means of the generation device uses the three-dimensional model arranged in the virtual space as the viewpoint of the objective virtual camera position specified by the objective camera position specifying means as one or more virtual space images Virtual space image acquisition process to be generated;
    And transmitting the virtual space image generated by the virtual space image acquisition means to the synthesizing device.
    A receiving step of receiving the virtual space image transmitted from the generation device by a real space image acquisition unit of the combining device;
    A physical space image acquiring step of acquiring a physical space image generated by imaging the physical space including the object by the imaging device;
    A control method of an image processing apparatus, comprising: a mixed reality image obtaining step of combining the virtual space image and the real space image to generate a mixed reality image; .
13. A method of controlling an image processing apparatus can be implemented which includes a generating device (10) for generating a virtual space in which a three-dimensional model is arranged, and a synthesizing device (20) communicably connected to the imaging device and processing the captured image Program, and
    The generator,
    Objective camera position specifying means for repeatedly acquiring the position and orientation of the imaging device (60) in the physical space and repeatedly specifying the objective virtual camera position which is the position and orientation of the virtual camera in the virtual space;
    Object position identification means for repeatedly acquiring the position and orientation of the object in the real space and repeatedly specifying the object position which is the position and orientation of the object in the virtual space;
    The three-dimensional model is arranged in the virtual space according to the objective virtual camera position and / or the target position, and the three-dimensional model is moved according to the movement of the imaging device and / or the target to perform the virtual Arrangement means for arranging in space;
    Virtual space image acquisition means for generating one or more virtual space images, with the objective virtual camera position specified by the objective camera position specifying means as a viewpoint, with the three-dimensional model arranged in the virtual space;
    Allowing the virtual space image generated by the virtual space image acquisition means to function as a transmission means for transmitting the image to the combining device,
    The synthesizer is
    Receiving means for receiving the virtual space image transmitted from the generating device;
    A physical space image acquisition unit that acquires a physical space image generated by imaging the physical space including the object by the imaging device;
    An image processing program characterized by causing the virtual space image and the real space image to overlap each other to generate a mixed reality image.

Images