JP6011567B2 - Information processing apparatus, control method thereof, and program - Google Patents

Description

The present invention is a mechanism for arranging a real space image including an object in a real space imaged by an imaging device in a virtual space, and even if the object moves in the real space, the size of the object viewed from the imaging device is large. And a mechanism capable of changing the size of the real space image so that the size of the object included in the real space image viewed from the viewpoint set in the virtual space is substantially the same. The present invention relates to an information processing apparatus, a control method thereof, and a program.

  Conventionally, a technique for changing the background of a subject in an image has been considered. One of these techniques is a technique called chroma key composition. In this case, a subject is imaged in front of a background of a homogeneous color (for example, blue or green), and the homogeneous color is transmitted in an image obtained by the imaging. Then, the background of the subject is changed by superimposing the transmitted image on another image desired to be used as the background.

  The following Patent Document 1 discloses a technique related to the chroma key composition. In the chroma key composition, since two images are superimposed, there is a problem that a part of the lower image is hidden. Therefore, in Patent Document 1 below, a portion of a subject that is not transmitted is made translucent so that the lower image is not hidden by the subject.

JP 7-222191 A

  By the way, an image of a virtual world (hereinafter referred to as virtual space) generated by a computer is superimposed on an image of a real world (hereinafter referred to as real space) captured by a conventional head-mounted display (HMD) or a smartphone, and the HMD or There are technologies that display on the screen of a smartphone. For example, there are mixed reality (hereinafter referred to as MR) technology and augmented reality technology.

  In this MR technology, in order to enhance mixed reality, the position and orientation of the user are acquired in real time, an image of the virtual space is generated following the change of the image captured in the real space, and the user is notified through the HMD. It is necessary to display in real time. Therefore, in MR technology, the position and orientation of a user measured by a sensor or the like are set as a virtual position and orientation in a virtual space, and based on this setting, an image in the virtual space is drawn by CG, Synthesizing. With this technology, the user can observe an image as if a virtual object exists in the real space.

  There is a case where it is desired to take a picture of experiencing mixed reality with this MR technology. However, even if the user wearing the HMD is imaged in the real space, the user and the real space are only imaged, and the image including the 3D model object and the like existing in the virtual space cannot be captured.

  Therefore, a method using the above-described chroma key composition is conceivable. That is, the following method is used. The room where the mixed reality is experienced is made a uniform color such as blue or green, and the user wearing the HMD is imaged. Then, in the captured image (video), the homogeneous color is transmitted, and is superimposed on the image viewed from the position of the virtual space corresponding to the position where the real space is imaged. For example, it is a form as shown in FIG. 1201 in FIG. 12 is a real space image obtained by imaging a real space, and 1202 is a virtual space image obtained by imaging a virtual space. By superimposing these, it is possible to generate an image including the same virtual space as the user who is experiencing the mixed reality as shown by 1203.

  However, there is a problem that this method cannot appropriately represent the position of the user in the virtual space. Although the user 1205 in FIG. 12 is displayed in front of the object 1204, the user must be displayed in the rear of the object as indicated by 900 in FIG. Since the user can freely walk around the virtual space, an unnatural image as shown at 1203 in FIG. 12 is generated unless the position of the user in the virtual space is taken into consideration.

The present invention is a mechanism for arranging a real space image including an object in a real space imaged by an imaging device in a virtual space, and even if the object moves in the real space, the size of the object viewed from the imaging device is large. And a mechanism capable of changing the size of the real space image so that the size of the object included in the real space image viewed from the viewpoint set in the virtual space is substantially the same. The purpose is to provide a mechanism.

In order to achieve the above object, an information processing apparatus according to the present invention is an information processing apparatus that is communicably connected to an imaging apparatus and generates a virtual space in which a three-dimensional model is arranged. An information processing apparatus connected to generate a virtual space in which a three-dimensional model is arranged, and storing means for storing the position and orientation in the virtual space corresponding to the position and orientation of the imaging apparatus in real space; Position specifying means for repeatedly acquiring a current position of a target object in the real space and moving in the real space, and repeatedly specifying the current position of the target object in the virtual space based on the acquired current position A real space image acquisition means for acquiring a real space image generated by imaging a real space including the object with the imaging device, and a virtual space. With arranging the three-dimensional model to a predetermined position that, according to the current position of the object specified by said position specifying means, the physical space image acquired in the real space image acquisition means disposed in the virtual space, Further, in accordance with the movement of the object, an arrangement unit that moves and arranges the arranged real space image to the current position of the object specified by the position specifying unit, and a real space image arranged by the arrangement unit The size changing means for changing the size of the image to a size substantially the same as the range at the current position where the real space image is arranged, in the field of view when the position and orientation stored in the storage means are used as a viewpoint And the virtual space including the real space image resized by the size changing means and the three-dimensional model, with the position and orientation stored in the storage means as a viewpoint. Characterized in that it comprises a display control means for controlling to display.

According to the present invention, in a mechanism for arranging a real space image including an object in the real space imaged by the imaging device in the virtual space, the object viewed from the imaging device even if the object moves in the real space . Providing a mechanism capable of changing the size of the real space image so that the size of the object and the size of the object included in the real space image viewed from the viewpoint set in the virtual space are substantially the same. It becomes possible.

It is a figure which shows an example of the system configuration | structure of MR system in embodiment of this invention. 2 is a diagram illustrating an example of a hardware configuration of a first information processing apparatus 101. FIG. 2 is a diagram illustrating an example of a hardware configuration of a second information processing apparatus 105. FIG. 2 is a diagram illustrating an example of functional configurations of a first information processing apparatus 101 and a second information processing apparatus 105. FIG. It is a flowchart which shows an example of the flow of a series of processes in embodiment of this invention. It is a figure which shows a mode that a transparent color is set to the area | region different from the user extracted from the real space image. It is a figure which shows HMD position and attitude | position 700, and objective camera position and attitude | position 710. FIG. It is a figure which shows the virtual space where the texture was arrange | positioned according to the position and attitude | position of the objective camera 106, and the user's position. It is a figure which shows the case where the virtual space shown in FIG. 8 is browsed from the virtual camera 804. It is a figure which shows the virtual space when a user moves to the position different from FIG. It is a figure which shows the case where the virtual space shown in FIG. 10 is browsed from the virtual camera 804. It is a figure which shows an example in which image data are produced | generated by the incorrect positional relationship.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a system configuration of an MR system according to an embodiment of the present invention. The MR system is a system that provides a user with mixed reality (hereinafter referred to as MR). The MR system includes a first information processing apparatus 101, an HMD 102, an optical sensor 104, a second information processing apparatus 105, and an objective camera 106 (imaging apparatus). The HMD 102, the optical sensor 104, and the second information processing apparatus 105 are connected to the first information processing apparatus 101, and the first information processing apparatus 101 is connected to each other by wire or wireless so that data communication is possible. Has been. In addition, an objective camera 106 is connected to the second information processing apparatus 105, and the second information processing apparatus 105 is connected to the second information processing apparatus 105 in a wired or wireless manner so as to be capable of data communication with each other. The configuration of various terminals connected on the system of FIG. 1 is an example, and there are various configuration examples depending on the application and purpose.

  The first information processing apparatus 101 is a general-purpose apparatus equipped with an operating system. The first information processing apparatus 101 generates an image (hereinafter, mixed reality image) in which a real space image (hereinafter, real space image) and a virtual space image (hereinafter, virtual space image) are superimposed, and the HMD 102. Send to. Since the conventional technique is used for the MR technique, a detailed description is omitted.

  The HMD 102 is a so-called head mounted display. The HMD 102 is a display device worn on the user's head, and includes a right-eye and left-eye video camera, and a right-eye and left-eye display. The HMD 102 transmits a real space image captured by the video camera of the HMD 102 to the first information processing apparatus 101. Then, the mixed reality image transmitted from the first information processing apparatus 101 is received and displayed on the display. Further, since the right-eye and left-eye video cameras and the display are provided, a stereoscopic effect can be obtained by parallax. The physical space image captured by the HMD 102 and the mixed reality image to be displayed are preferably moving images, but may be still images captured at predetermined intervals.

  The optical sensor 104 is an apparatus that images a real space and detects the position and orientation of the optical marker 103. In the present embodiment, the optical sensor 104 detects the optical marker 103 provided in the HMD 102 and specifies the position and orientation of the HMD 102. It is desirable to install a plurality of optical sensors 104 in the room so that the optical marker 103 can be detected under any circumstances. In this embodiment, information indicating the position and orientation of the HMD 102 is acquired using the optical sensor 104 and the optical marker 103 provided in the HMD 102. However, any form may be used as long as the position and orientation of the HMD 102 can be detected. . For example, a magnetic sensor may be used, or the position and orientation may be specified by analyzing an image captured by the HMD 102.

  The second information processing device 105 is a general-purpose device equipped with an operating system. The second information processing apparatus 105 manages the same virtual space as the virtual space managed by the first information processing apparatus 101. Further, the real space image captured by the objective camera 106 is arranged in the virtual space according to the information indicating the position of the HMD 102 acquired from the first information processing apparatus 101.

  The objective camera 106 is an apparatus that captures a real space. The real space image captured by the objective camera 106 is transmitted to the second information processing apparatus 105. In the present embodiment, the real space image is preferably a moving image, but it may be a still image captured at a predetermined interval. In addition, the objective camera 106 in the present embodiment is a monocular camera and is further fixed at a predetermined position.

  FIG. 2 is a diagram illustrating a hardware configuration of the first information processing apparatus 101. Note that the hardware configuration of the first information processing apparatus 101 in FIG. 2 is an example, and there are various configuration examples depending on the application and purpose.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  The ROM 202 or the external memory 211 stores a BIOS (Basic Input / Output System) that is a control program of the CPU 201, an operating system, and various programs described later that are necessary for realizing the functions executed by various devices. Yes. The RAM 203 functions as a main memory, work area, and the like for the CPU 201.

  The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 203 and executing the program.

  An input controller (input C) 205 controls input from a pointing device (input device 210) such as a keyboard and a mouse.

  A video controller (VC) 206 controls display on a display device such as the right eye / left eye display 222 provided in the HMD 102. For example, an external output terminal (for example, Digital Visual Interface) is used for output to the right eye / left eye display 222. The right-eye / left-eye display 222 includes a right-eye display and a left-eye display.

  The memory controller (MC) 207 is an adapter to a hard disk (HD), flexible disk (FD) or PCMCIA card slot for storing boot programs, browser software, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a card-type memory connected via the.

  A communication I / F controller (communication I / FC) 208 is connected to and communicates with an external device via a network, and executes communication control processing in the network. For example, Internet communication using TCP / IP is possible. The communication I / F controller 208 also controls communication with the optical sensor 104 through Gigabit Ethernet (registered trademark) or the like.

  The general-purpose bus 209 is used for capturing video from the right-eye / left-eye video camera 221 of the HMD 102. Input from the right-eye / left-eye video camera 221 is performed using an external input terminal (for example, an IEEE 1394 terminal). The right-eye / left-eye video camera 221 includes a right-eye video camera and a left-eye video camera.

  Note that the CPU 201 enables display on a display by executing outline font rasterization processing on a display information area in the RAM 203, for example. In addition, the CPU 201 allows a user instruction with a mouse cursor (not shown) on the display.

  Various programs and the like used by the first information processing apparatus 101 of the present invention to execute various processes to be described later are recorded in the external memory 211 and executed by the CPU 201 by being loaded into the RAM 203 as necessary. Is. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 211.

  FIG. 3 is a diagram illustrating a hardware configuration of the second information processing apparatus 105. Note that the hardware configuration of the second information processing apparatus 101 in FIG. 3 is merely an example, and there are various configuration examples depending on applications and purposes.

  The hardware configuration of the second information processing apparatus 105 is the same as the hardware configuration of the first information processing apparatus 101 described above with reference to FIG. Unlike the first information processing apparatus 101, the right eye / left eye display 222 is not connected to the video controller 206 of the second information processing apparatus 105. Further, the optical sensor 104 is not connected to the communication I / F controller 208 of the second information processing apparatus 105. Furthermore, the right-eye / left-eye video camera 221 is not connected to the general-purpose bus 209 of the second information processing apparatus 105, and the objective camera 106 is connected.

  FIG. 4 is a functional configuration diagram illustrating functional configurations of the first information processing apparatus 101 and the second information processing apparatus 105. Note that the functional configurations of the first information processing apparatus 101 and the second information processing apparatus 105 in FIG. 4 are merely examples, and there are various configuration examples depending on applications and purposes.

  The first information processing apparatus 101 functions as a communication control unit 401, an HMD position / attitude acquisition unit 402, an HMD real space image acquisition unit 403, a virtual space image acquisition unit 404, a mixed reality image generation unit 405, and a virtual space management. 406 and a display control unit 407.

  The communication control unit 401 is a functional unit that transmits and receives various types of information between the HMD 102, the optical sensor 104, and the second information processing device 105 that can communicate with the first information processing device 101. The communication control unit 401 transmits / receives information to / from these apparatuses through the video controller 206, the communication I / F controller 208, the general-purpose bus 209, and the like.

  The HMD position / posture acquisition unit 402 is a functional unit that acquires information indicating the position and posture of the HMD 102 in the real space from the optical sensor 104. The HMD real space image acquisition unit 403 is a functional unit that acquires a real space image from the right-eye / left-eye video camera 221 of the HMD 102.

  The virtual space image acquisition unit 404 is a functional unit that acquires an image of the virtual space generated by the first information processing apparatus 101. When the virtual space image acquisition unit 404 acquires an image of the virtual space, a viewpoint on the virtual space is determined based on the position and orientation of the HMD 102 acquired by the HMD position / orientation acquisition unit 402 and viewed from the viewpoint. A virtual space image of the case is generated and acquired.

  The mixed reality image generation unit 405 is a functional unit that generates a mixed reality image by superimposing the virtual space image acquired by the virtual space image acquisition unit 404 on the real space image acquired by the HMD real space image acquisition unit 403. .

  The virtual space management unit 406 is a functional unit that generates a virtual space in the first information processing apparatus 101 and arranges an object including a three-dimensional model in the virtual space. Each object used in the virtual space management unit 406 is stored in the external memory 211 or the like of the first information processing apparatus 101, and these are appropriately read to generate a virtual space.

  The display control unit 407 is a functional unit that performs display control of various types of information on the right-eye / left-eye display 222 of the HMD 102 connected to the first information processing apparatus 101 and the display 212 connected to the first information processing apparatus 101. .

  The second information processing apparatus 105 includes, as functional units, a communication control unit 451, an objective camera position / posture acquisition unit 452, an objective camera real space image acquisition unit 453, a transmission processing unit 454, a texture generation unit 455, and an HMD position acquisition. A unit 456, an object placement unit 457, a texture control unit 458, a display control unit 459, and a virtual space management unit 460.

  The communication control unit 451 is a functional unit that transmits and receives various types of information between the first information processing apparatus 101 and the objective camera 106 that can communicate with the second information processing apparatus 105. The communication control unit 401 transmits / receives information to / from these apparatuses through the communication I / F controller 208, the general-purpose bus 209, and the like described above.

  The objective camera position / posture acquisition unit 452 is a functional unit that acquires information indicating the position and posture in the real space where the objective camera 106 is installed. In this embodiment, since the objective camera 106 is fixed at a predetermined location, information indicating the position and orientation of the objective camera 106 is stored in advance in the external memory 211 or the like. Therefore, the objective camera position / posture acquisition unit 452 acquires the stored information. Alternatively, based on information included in the real space image acquired by the objective camera real space image acquisition unit 453, the position and posture where the objective camera 106 is installed may be specified and acquired. In this case, the two-dimensional marker 107 as shown in FIG. 1 is associated with predetermined coordinates in the virtual space, and the two-dimensional marker 107 is installed on the floor or wall of the imaging range of the objective camera 106. Then, based on the size and shape of the two-dimensional marker 107 included in the real space image acquired from the objective camera 106, the coordinates associated with the two-dimensional marker 107, the positional relationship with the two-dimensional marker 107 is calculated, and the position of the objective camera 106 and Identify posture.

  The objective camera real space image acquisition unit 453 is a functional unit that acquires a real space image from the objective camera 106.

  The transmission processing unit 454 is a functional unit that transmits a predetermined color included in the real space image acquired by the objective camera real space image acquisition unit 453. For example, when the background color of the real space is blue, the transparency (alpha value) of the blue pixel among the pixels included in the real space image is set to “0”. The transmittance is not necessarily “0”, but the effect of the present invention can be further improved by transmitting completely. Further, there may be a plurality of colors to be transmitted, and the target color to be transmitted may be given a width depending on the color and density.

  The texture generation unit 455 is a functional unit that generates a texture that can be pasted on a planar three-dimensional model (object) arranged in the virtual space. The texture generation unit 455 generates a physical space image in which the specified color is subjected to the transmission processing by the transmission processing unit 454 as a texture.

  The HMD position acquisition unit 456 is a functional unit that acquires information indicating the position of the HMD 102 in the real space from the first information processing apparatus 101. In the present embodiment, the description is given in the form acquired from the first information processing apparatus 101. However, the optical sensor 104 and the communication I / F controller 208 are connected to indicate the position of the HMD 102 from the optical sensor 104. Information may be obtained directly.

  The object placement unit 457 is a functional unit that determines the position of the object to which the texture generated by the texture generation unit 455 is pasted, and places a planar object at the determined position. The object placement unit 457 uses the information indicating the position and posture of the objective camera 106 acquired by the objective camera position / posture acquisition unit 452 and the information indicating the position of the HMD 102 acquired by the HMD position acquisition unit 456 on the virtual space. A plane object perpendicular to the optical axis of the objective camera 106 and passing through the position of the HMD 102 is arranged.

  The texture control unit 458 is a functional unit that pastes the texture generated by the texture generation unit on the object arranged by the object arrangement unit 457. When pasting, the center point of the texture is pasted at a position where the optical axis of the objective camera 106 passes in the virtual space. Further, the texture control unit 460 controls the viewpoint in the virtual space according to the position / orientation of the objective camera 106 and satisfies the viewing range when the virtual space is browsed from the controlled viewpoint. 458 changes the size of the texture.

  The display control unit 459 is a functional unit that controls the viewpoint on the virtual space according to the position and orientation of the objective camera 106 by the virtual space management unit 460 and controls display on the display 212 of the virtual space viewed from the controlled viewpoint. It is.

  The virtual space management unit 460 generates a virtual space similar to that of the first information processing apparatus 101 in the second information processing apparatus 105 and includes the same three-dimensional model as that of the first information processing apparatus 101 in the virtual space. It is a functional unit that arranges objects. Each object used in the virtual space management unit 406 is stored in the external memory 211 or the like of the second information processing apparatus 105, and these are appropriately read to generate a virtual space. Therefore, it is desirable to copy various information for generating a virtual space from the first information processing apparatus 101 to the external memory 211 in advance.

  Next, a series of processing performed by the first information processing apparatus 101 and the second information processing apparatus 105 in the embodiment of the present invention will be described with reference to the flowchart shown in FIG.

  First, the flow of processing in the first information processing apparatus 101 will be described. The first information processing apparatus 101 presents mixed reality to the user wearing the HMD 102. In particular, in the present embodiment, the following description will be given on the assumption that the user wears the HMD 102 in a room composed of walls, floors, ceilings, and other installations of a predetermined color (blue, green, etc.) to experience mixed reality. I do.

  In step S <b> 501, the CPU 201 of the first information processing apparatus 101 acquires information indicating the position and orientation of the HMD 102 from the optical sensor 104 using the HMD position / orientation acquisition unit 402 and stores the information in the RAM 203 or the like. That is, in step S501, information indicating the position and orientation of the user wearing the HMD 102 in the real space is acquired. Information indicating the position and orientation of the HMD 102 is, for example, as shown in the HMD position / orientation 700 of FIG. XYZ coordinates when a predetermined place in the real space is the origin, and a vector using the XYZ coordinates. Let these two be the position and orientation of the HMD 102. In addition, the acquisition method of the information which shows the position and attitude | position of HMD102 is not restricted to this. The position and orientation of the HMD 102 may be specified using a magnetic sensor or a two-dimensional marker. Since a method for acquiring information indicating the position and orientation of an object in the real space is a conventional technique, description thereof is omitted.

  In step S <b> 502, the CPU 201 of the first information processing apparatus 101 uses the communication control unit 401 to obtain information indicating the position of the HMD 102 out of the HMD position / posture 700 acquired in step S <b> 501. Send to.

  In step S503, the CPU 201 of the first information processing apparatus 101 acquires the real space image transmitted from the right-eye / left-eye video camera 221 of the HMD 102 using the HMD real-space image acquisition unit 403 and stores it in the RAM 203 or the like. . Since there are two right-eye / left-eye video cameras 221, that is, a video camera corresponding to the user's right eye and a video camera corresponding to the left eye, the real space images for the right eye and the left eye are acquired from these.

  In step S504, the CPU 201 of the first information processing apparatus 101 acquires a virtual space image using the virtual space image acquisition unit 404 and stores it in the RAM 203 or the like. More specifically, first, the HMD position / orientation 700 acquired in step S501 is read from the RAM 203 or the like, and the position and orientation in the virtual space corresponding to the HMD position / orientation 700 are specified. Then, a virtual camera (viewpoint) is installed on the virtual space with the specified position and orientation on the virtual space, and the virtual space is imaged by the camera. Thereby, a virtual space image is generated. Similar to the above-described real space image, two images of the right eye virtual space image and the left eye virtual space image are acquired for display on the right eye and left eye display 222 of the HMD 102, respectively.

  In step S505, the CPU 201 of the first information processing apparatus 101 reads the real space image acquired in step S503 and the virtual space image acquired in step S504 from the RAM 203 or the like. Then, the virtual space image is superimposed on the real space image using the mixed reality image generation unit 405 to generate a mixed reality image. The generated mixed reality image is stored in the RAM 203 or the like. As described above, since the real space image and the virtual space image are stored in the RAM 203 or the like for each of the right eye and the left eye, the virtual space image for the right eye is superimposed on the real space image for the right eye. The virtual space image for the left eye is superimposed on the real space image for the left eye. In addition, as described above, in the present embodiment, the wall of the room in the real space has a predetermined color (blue, green, etc.) for the convenience of executing step S510 described later. Therefore, it is desirable to superimpose a virtual space image in which the predetermined color cannot be seen. That is, the immersive feeling is higher when the real space is not visible like the virtual reality.

  In step S <b> 506, the CPU 201 of the first information processing apparatus 101 reads the mixed reality image generated in step S <b> 505 from the RAM 203 or the like and displays it on the right eye / left eye display 222 of the HMD 102 through the video controller 206. The mixed reality image stored in the RAM 203 or the like has two sheets for the right eye and the left eye. Therefore, the display controller 407 is used to control the right-eye mixed reality image to be displayed on the right-eye display of the right-eye / left-eye display 222, and the left-eye mixed reality image is displayed on the left-eye display of the right-eye / left-eye display 222. Control to do.

  In step S507, the CPU 201 of the first information processing apparatus 101 determines whether or not there is an instruction to end the process of presenting mixed reality to the user wearing the HMD 102 (steps S501 to S506 described above). For example, it is determined whether or not there has been an instruction to stop or end the application of the first information processing apparatus 101 that executes the processes of steps S501 to S506 described above. If it is determined that an end instruction has been issued, the series of processes ends. If it is not determined that there is an end instruction, that is, if there is no end instruction, the process returns to step S501, and steps S501 to S507 are repeated until there is an end instruction.

  Next, processing of the second information processing apparatus 105 that operates asynchronously with the first information processing apparatus 101 will be described.

  In step S508, the CPU 201 of the second information processing apparatus 105 receives information (position information) indicating the position of the HMD 102 transmitted from the first information processing apparatus 101 in step S502 using the communication control unit 451. And stored in the RAM 203 or the like (storage means). In the present embodiment, information indicating the position of the HMD 102 is acquired by receiving the information from the first information processing apparatus. However, when the objective camera 106 is a compound eye camera, the distance from the objective camera 106 to the HMD 102 is determined by triangulation. And the position of the HMD 102 may be calculated using information indicating the position and orientation of the objective camera 106.

  In step S <b> 509, the CPU 201 of the second information processing apparatus 105 converts the real space image generated by imaging the real space with the objective camera 106 connected to the second information processing apparatus 105 to the objective camera real space image. It acquires using the acquisition part 453, and memorize | stores it in RAM203 etc. (real space image acquisition means). As described above, since the objective camera 106 is a monocular camera in the present embodiment, unlike a compound eye camera such as the HMD 102, one image (or video) is output.

  In step S510, the CPU 201 of the second information processing apparatus 105 reads the real space image acquired in step S509 from the RAM 203 or the like, and uses the transmission processing unit 454 to determine a predetermined color (blue, green, or the like) of the real space image. (Transmission processing means). Since the portion other than the user wearing the HMD 102 has a predetermined color in the real space image, the predetermined color of the real space image is transmitted by changing the alpha value of the predetermined color. In this way, only the user can be extracted from the real space image. FIG. 6 shows an overview of step S510. 6 in FIG. 6 is the real space image acquired in step S509. The real space image includes the user 601 wearing the HMD 102, and the floor, wall, ceiling, and the like are the predetermined colors that are subjected to the transmission processing in step S510. When the process of step S510 is executed on this real space image, a predetermined color is transmitted, so that a real space image as shown at 610 is obtained. That is, the image is transmitted through the floor, wall, ceiling, etc. other than the user.

  In step S511, the CPU 201 of the second information processing apparatus 105 generates a texture that can be displayed in the virtual space using the real space image transmitted in step S510 (texture generation unit). A texture is a pattern to be pasted on the surface of an object composed of a three-dimensional model. The texture generation unit 455 is used to generate this texture from the real space image.

  In step S <b> 512, the CPU 201 of the second information processing apparatus 105 acquires information (position and orientation information) indicating the position and orientation of the objective camera 106 using the objective camera position and orientation acquisition unit 452. As described above, in the present embodiment, information indicating the position and orientation of the objective camera 106 is stored in the external memory 211 or the like (storage means), and this is acquired. As described above, the method for acquiring the position and orientation of the objective camera 106 is not limited to this. That is, the objective camera 106 may be identified by imaging the two-dimensional marker 107 attached to the wall or the like of the real space with the objective camera 106, or may be identified by other methods. Also good. Information indicating the position and orientation of the objective camera 106 is, for example, as shown in the objective camera position / orientation 710 of FIG. The objective camera position / posture 710 is the same information as the HMD position / posture 700, and the description thereof is omitted.

  In step S 513, the CPU 201 of the second information processing apparatus 105 specifies the optical axis of the objective camera 106 in the virtual space using the virtual space management unit 460. That is, the position and posture in the virtual space corresponding to the objective camera position / posture 710 acquired in step S512 are specified, and the vector of the optical axis when the virtual space is imaged with the position and posture is specified.

  In step S514, the CPU 201 of the second information processing apparatus 105 reads information indicating the position of the HMD 102 received in step S508 from the RAM 203 or the like.

  In step S515, the CPU 201 of the second information processing apparatus 105 uses the object placement unit 457 to be perpendicular to the optical axis in the virtual space of the objective camera 106 specified in step S513 and read the HMD 102 read in step S514. A plane object passing through the position is generated and placed (object placement means).

  In step S516, the CPU 201 of the second information processing apparatus 105 uses the texture control unit 458 to paste the texture generated in step S511 on the object placed in step S515 (texture pasting unit). When pasting, the optical axis in the virtual space of the objective camera 106 is pasted so that it passes through the center point of the texture.

  In step S517, the CPU 201 of the second information processing apparatus 105 uses the texture control unit 458 to change the size of the texture pasted in step S516 according to the range of the virtual space displayed on the display 212 (texture size). Change means). That is, the texture is enlarged or reduced to the same size as the imaging range of the virtual camera that images the virtual space. The closer the texture position is to the virtual camera (the viewpoint in the virtual space corresponding to the position and orientation of the objective camera 106 in the real space), the smaller the size of the texture, and the farther the texture position is from the virtual camera. The farther away, the larger the texture size.

  In the real space image captured by the objective camera 106, the perspective of the user who is already the subject is expressed. However, since there is a perspective in the virtual space, the perspective is simply doubled unless the size of the texture is changed. That is, in order not to destroy the perspective captured by the objective camera 106, if the texture position is close to the virtual camera, the texture is reduced to cancel the perspective in the virtual space, and if the texture position is far from the virtual camera. The texture is magnified to offset the perspective in the virtual space.

  In step S518, the CPU 201 of the second information processing apparatus 105 reads out the objective camera position / posture 710 acquired in step S512 from the RAM 203 and the like, and in accordance with the objective camera position / posture 710, a virtual camera ( (Viewpoint) is set, and the virtual space viewed from the camera is imaged. Thereby, a virtual space image is generated and stored in the RAM 203 or the like.

  In step S519, the CPU 201 of the second information processing apparatus 105 acquires the virtual space image captured in step S518 from the RAM 203 or the like, and displays the virtual space image on the display 212 via the video controller 206 (display unit). .

  In step S520, the CPU 201 of the second information processing apparatus 105 determines whether or not there has been an instruction to end the process (steps S508 to S518 described above) for displaying the state in which the user wearing the HMD 102 moves in the virtual space. judge. For example, it is determined whether or not there has been an instruction to stop or end the application of the second information processing apparatus 105 that executes the processes of steps S508 to S518 described above. If it is determined that an end instruction has been issued, the series of processes ends. If it is not determined that there is an end instruction, that is, if there is no end instruction, the process returns to step S508, and steps S508 to S519 are repeated until there is an end instruction.

  An example of processing in steps S508 to S519 will be described with reference to FIGS.

  A perspective image 800 in FIG. 8 is a diagram illustrating a virtual space as a result of executing the processing in steps S508 to S517. In this virtual space, an object 801, an object 802, and an object 803 made of a three-dimensional model are arranged. A virtual camera 804 is installed in the virtual space according to the position and orientation of the objective camera 106. An image viewed from the viewpoint of the virtual camera 804 is displayed on the display 212.

  In step S513, the optical axis of the objective camera 106 is specified on the virtual space. That is, the optical axis 805 is the optical axis of the objective camera 106. A plane object perpendicular to the optical axis and passing through the position of the HMD 102 is an object 806. As shown in the top image 810, a plane perpendicular to the optical axis 805 and passing through the position 811 of the HMD 102 is an object 806.

  Then, the texture of the transmitted real space image indicated by reference numeral 610 in FIG. 6 is pasted on the object 806 (step S516), and the real space image pasted so as to become the imaging range 807 of the virtual camera 804 is pasted. Is changed (step S517). An inner space including the four line segments indicated by the imaging range 807 illustrated in FIG. 8 is a range in which the virtual camera 804 can capture images. The size of the real space image is adjusted to satisfy this range. By doing so, the texture-attached object 806 is arranged between the object 802 and the object 803, and therefore when the image is taken with the virtual camera 804, the virtual space image 900 of FIG. An image is displayed on the display 212. In the virtual space image 900, unlike the image 1203 in FIG. 12 described above, the user 601 who wears the HMD 102 can be placed at an appropriate depth position.

  If the user 601 wearing the HMD 102 moves, the position of the object 806 also changes. A perspective image 1000 illustrated in FIG. 10 illustrates a case where the user 601 moves between the object 801 and the object 802. Since the objective camera 106 is fixed, the position and posture of the virtual camera 804 are not changed. Therefore, the optical axis 805 and the imaging range 807 are the same as those in FIG. When the user 601 wearing the HMD 102 moves to the position indicated by 1011 of the upper surface image 1010, the object 806 moves to the position indicated by 1001. Therefore, when this virtual space is imaged by the virtual camera 804, an image as shown by the virtual space image 1100 in FIG. 11 is displayed on the display 212. It can be confirmed that the user 601 shown in the virtual space image 1100 in FIG. 11 is located between the object 801 and the object 802.

  As described above, the depth that cannot be expressed by normal chroma key composition can be expressed by using the texture of the real space image and the virtual space.

  As described above, according to the present embodiment, there is an effect that an image generated by imaging a user in the real space can be displayed at an appropriate position in the virtual space.

  The present invention can be implemented as a system, apparatus, method, program, storage medium, or the like, and can be applied to a system including a plurality of devices. You may apply to the apparatus which consists of one apparatus.

  Note that the present invention includes a software program that implements the functions of the above-described embodiments directly or remotely from a system or apparatus. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

101 First information processing apparatus 102 HMD
103 optical marker 104 optical sensor 105 second information processing apparatus 106 objective camera 107 two-dimensional marker 201 CPU
202 ROM
203 RAM
204 System bus 205 Input controller 206 Video controller 207 Memory controller 208 Communication I / F controller 209 General-purpose bus 210 Input device 211 External memory 212 Display 221 Right-eye / left-eye video camera 222 Right-eye / left-eye display

Claims (8)

An information processing apparatus that is communicably connected to an imaging apparatus and generates a virtual space in which a three-dimensional model is arranged,
Storage means for storing the position and orientation in the virtual space corresponding to the position and orientation of the imaging device in real space;
Position specifying means for repeatedly acquiring a current position of a target object in the real space and moving in the real space, and repeatedly specifying the current position of the target object in the virtual space based on the acquired current position When,
Real space image acquisition means for acquiring a real space image generated by imaging the real space including the object with the imaging device;
The three-dimensional model is arranged at a predetermined position in the virtual space, and the real space image acquired by the real space image acquiring unit is placed in the virtual space according to the current position of the object specified by the position specifying unit. Arranging means for moving and arranging the arranged real space image to the current position of the object specified by the position specifying means according to the movement of the object ;
The size of the real space image arranged by the arrangement means is the range at the current position where the real space image is arranged, out of the range of field of view when the position and orientation stored in the storage means are taken as viewpoints Resizing means for changing to substantially the same size;
Display control means for controlling the virtual space including the real space image resized by the size changing means and the three-dimensional model so as to display the position and orientation stored in the storage means as a viewpoint. An information processing apparatus characterized by that.   The arrangement means arranges the real space image acquired by the real space image acquisition means in the virtual space so that the context with the three-dimensional model arranged in the virtual space is expressed. The information processing apparatus according to claim 1. The placement means is a position perpendicular to the line-of-sight direction with the position and orientation stored in the storage means as a viewpoint, and the real space image acquisition means at the current position of the object specified by the position specifying means. The information processing apparatus according to claim 1, wherein the acquired real space image is arranged in the virtual space.   The real space image acquisition unit acquires a real space image obtained by extracting the object from an image generated by imaging the real space including the object by the imaging device. 4. The information processing apparatus according to any one of 3.   The real space image acquisition means generates a real space image in which the object is extracted by transmitting a predetermined color of the image generated by imaging the real space including the object with the imaging device. The information processing apparatus according to claim 4. The arrangement unit arranges an object in the virtual space according to a current position of the target object specified by the position specifying unit, and pastes the real space image on the object, whereby the real space image is attached to the virtual space. 6. The information processing apparatus according to claim 1, wherein the information processing apparatus is disposed in a space. Storage means for storing the position and orientation in the virtual space corresponding to the position and orientation of the imaging device in the real space;
A control method for an information processing apparatus that is communicably connected to an imaging apparatus and generates a virtual space in which a three-dimensional model is arranged,
The position specifying unit of the information processing apparatus is an object in the real space , repeatedly acquires the current position of the object moving in the real space, and based on the acquired current position, the object in the virtual space A position identifying step for repeatedly identifying the current position of
A real space image acquisition step of acquiring a real space image generated by imaging a real space including the object by the imaging device;
The arrangement unit of the information processing apparatus arranges the three-dimensional model at a predetermined position in the virtual space, and acquires in the real space image acquisition step according to the current position of the object specified in the position specifying step. An arrangement step of arranging the arranged real space image in the virtual space and further moving and arranging the arranged real space image to the current position of the object specified in the position specifying step according to the movement of the object. When,
The real space image in the range of the field of view when the size changing unit of the information processing apparatus uses the size and size of the real space image arranged in the arrangement step as the viewpoint stored in the storage unit. A resizing step for changing to a size substantially the same as the range at the current position where
The display control means of the information processing apparatus displays the virtual space including the real space image and the three-dimensional model whose size has been changed in the size changing step, with the position and orientation stored in the storage means as a viewpoint. And a display control step for controlling the information processing apparatus. Storage means for storing the position and orientation in the virtual space corresponding to the position and orientation of the imaging device in the real space;
A program capable of executing a control method of an information processing device that is communicably connected to an imaging device and generates a virtual space in which a three-dimensional model is arranged,
The information processing apparatus;
Position specifying means for repeatedly acquiring a current position of a target object in the real space and moving in the real space, and repeatedly specifying the current position of the target object in the virtual space based on the acquired current position When,
Real space image acquisition means for acquiring a real space image generated by imaging the real space including the object with the imaging device;
The three-dimensional model is arranged at a predetermined position in the virtual space, and the real space image acquired by the real space image acquiring unit is placed in the virtual space according to the current position of the object specified by the position specifying unit. Arranging means for moving and arranging the arranged real space image to the current position of the object specified by the position specifying means according to the movement of the object ;
The size of the real space image arranged by the arrangement means is the range at the current position where the real space image is arranged, out of the range of field of view when the position and orientation stored in the storage means are taken as viewpoints Resizing means for changing to substantially the same size;
Causing the virtual space including the real space image resized by the size changing means and the three-dimensional model to function as display control means for controlling the position and orientation stored in the storage means to be displayed as a viewpoint. A program characterized by that.

Images