JP2020204856A - Image generation system and program - Google Patents

Abstract

To provide an image generation system and a program in which a simple system can combine an image of a subject captured by a camera and an image of a virtual space with high dignity.SOLUTION: The image generation system includes an acquisition unit for acquiring a first image of a background and a subject captured by a camera positioned in a real space and a second image of a background captured by the camera, an image generation unit for generating a virtual space image visible from a shooting virtual camera positioned at a position in a virtual space corresponding to the position of the camera, and an image composition unit for extracting an image of the subject by obtaining a difference image between the first and second images to generate a composite image in which the image of the subject is combined with the virtual space image.SELECTED DRAWING: Figure 9

Description

The present invention relates to an image generation system, a program, and the like.

Conventionally, an image generation system that generates a virtual space image that can be seen from a virtual camera in a virtual space has been known. As a conventional technique of a system that realizes virtual reality (VR) by displaying an image seen from a virtual camera on an HMD (head-mounted display device), for example, there is a technique disclosed in Patent Document 1. Further, as a conventional technique of a system for synthesizing a live-action image and a virtual space image by blue background, for example, there is a technique disclosed in Patent Document 2.

Japanese Unexamined Patent Publication No. 11-309269 Japanese Unexamined Patent Publication No. 2011-35638

Chroma key composition using a blue background or a green background has a problem that a large-scale shooting equipment for a blue background or a green background must be installed at the installation location of the image generation system. Further, if the quality of the image composition of the virtual space image and the real space image is low, the virtual space image and the real space image cannot be consistent with each other, and the desired result cannot be obtained.

According to some aspects of the present invention, it is possible to provide an image generation system and a program capable of combining an image of a subject taken by a camera and a virtual space image with a simple system with high quality.

One aspect of the present disclosure is an acquisition unit for acquiring a first image in which a background and a subject are photographed by a camera arranged in a real space and a second image in which the background is photographed by the camera, and a position of the camera. An image of the subject is obtained by obtaining an image generation unit that generates a virtual space image that can be seen from a virtual camera for shooting arranged at a position in the virtual space corresponding to the above, and a difference image between the first image and the second image. The present invention relates to an image generation system, which comprises an image synthesizing unit that extracts and generates a composite image in which an image of the subject is synthesized with the virtual space image. Further, one aspect of the present disclosure relates to a program that causes a computer to function as each of the above parts, or a computer-readable information storage medium that stores the program.

According to one aspect of the present disclosure, the first image in which the background and the subject are photographed and the second image in which the background is photographed are acquired, and the positions of the cameras in which the first image and the second image are photographed correspond to each other. A virtual camera for shooting is arranged at the position of the virtual space to be shot, and a virtual space image that can be seen from the virtual camera for shooting is generated. Then, the image of the subject is extracted by obtaining the difference image between the first image and the second image, and a composite image in which the image of the subject is combined with the virtual space image is generated. In this way, it becomes possible to generate a composite image in which the image of the subject is combined with the virtual space image without the need for large-scale shooting equipment, and the image of the subject and the virtual space image taken by the camera. It will be possible to provide an image generation system or the like that can synthesize high-quality images with a simple system.

Further, in one aspect of the present disclosure, the image synthesizing unit extracts an image of a housing on which the subject is mounted in the real space, and the image of the subject and the image of the housing are combined with the virtual space image. The composite image may be generated.

By doing so, when the subject is on the housing in the real space, the real space image in the state where the subject is on the housing can generate an image synthesized with the virtual space image. Become.

Further, in one aspect of the present disclosure, the image synthesizing unit may extract the image of the housing by using the housing mask image that specifies the extraction range of the image of the housing.

By using such a housing mask image, it becomes possible to extract an image of the housing that is not originally extracted as a background in the same manner as the image of the subject.

Further, in one aspect of the present disclosure, the image synthesizing unit sets an extraction range of the image of the subject based on tracking information from at least one tracking device worn by the subject, and extracts the image of the subject. You may.

By doing so, even if the posture of the subject changes or the subject performs various actions, the image extraction process of the subject within an appropriate extraction range can be realized. For example, an object that is not planned as a subject. Can be prevented from being extracted as a subject.

Further, in one aspect of the present disclosure, the image compositing unit is based on the position of the tracking device and the position of an auxiliary point set to a position shifted by a predetermined distance from the position of the tracking device. The extraction range of the image of the above may be set.

In this way, even in a situation where an appropriate extraction range cannot be set only by the position of the tracking device, it is possible to set an extraction range that includes the subject by using the auxiliary points, and the image of the subject can be set. Appropriate extraction processing can be realized.

Further, in one aspect of the present disclosure, the image generation unit is set at a position corresponding to the position of the virtual camera for shooting in the virtual space as a virtual space image for the player displayed on the player who is the subject. You may generate a virtual space image in which at least one of the image of the virtual camera for shooting and the image of the photographer character is displayed.

In this way, it becomes possible to make the player aware that he / she is being photographed by the virtual camera for photography, and it is possible to realize an effect such as inducing the movement or pose of the player.

Further, in one aspect of the present disclosure, a head-mounted display device worn by the player who is the subject and displaying a virtual space image for the player seen from the virtual camera for the player in the virtual space, and the composite image are A gallery display device, which is displayed as a gallery image, may be included.

In this way, it is possible to let the gallery observe the behavior of the player who wears the head-mounted display device and plays in the virtual space.

Further, in one aspect of the present disclosure, the acquisition unit acquires a depth image of the background and the subject taken by the camera, and the image synthesis unit acquires the depth image of the subject based on the difference image and the depth image. The image may be extracted.

By extracting the image of the subject using not only the difference image but also the depth image in this way, it becomes possible to realize a higher quality image composition of the virtual space image and the image of the subject.

Further, in one aspect of the present disclosure, the image synthesizing unit generates a difference mask image based on the difference image, and identifies a pixel whose depth value is in a given depth range based on the depth image. An image is generated, a subject mask image that identifies the subject is generated based on the difference mask image and the depth mask image, and an image of the subject is extracted based on the subject mask image and the first image. You may.

By using such a difference mask image and a depth mask image, it is possible to extract a high-quality image of the subject from the first image in which the background and the subject are reflected.

Further, in one aspect of the present disclosure, the image synthesizing unit may perform correction processing of the depth mask image and generate the subject mask image based on the depth mask image and the difference mask image after the correction processing. Good.

By performing the correction processing on the depth mask image in this way, it is possible to prevent flicker of the edge portion, remove fine noise, and the like, and it is possible to generate a high-quality composite image.

Further, in one aspect of the present disclosure, the image synthesizing unit generates a difference depth mask image between the first depth image of the background and the subject taken by the camera and the second depth image of the background taken by the camera. Then, the depth mask image after the correction process may be generated based on the difference depth mask image.

By using such a difference depth mask image, it is possible to prevent a situation in which the background area is erroneously recognized as the subject area.

Further, in one aspect of the present disclosure, the image synthesizing unit may generate the depth mask image after the correction process by performing at least one of the morphology filter process and the time series filter process.

By performing such morphology filtering and time-series filtering, it becomes possible to remove fine-sized noise and suppress the occurrence of fine noise flicker.

Further, in one aspect of the present disclosure, the image synthesizing unit performs a process of setting the pixel value of the pixel for which the depth value could not be acquired in the depth image based on the difference image, thereby performing the correction process. The depth mask image may be generated.

By doing so, it is possible to prevent the problem that the image is chipped due to the failure to acquire the depth value, and it becomes possible to generate an appropriate extracted image of the subject.

Further, in one aspect of the present disclosure, the image synthesizing unit obtains a region size of a pixel group in which the depth value is in the depth range, and performs a filter process according to the region size to perform the correction process to obtain the depth mask. An image may be generated.

By doing so, it is possible to eliminate a pixel group having a small area size caused by noise or the like and extract a pixel group corresponding to the subject, and it is possible to generate a high-quality composite image.

Further, in one aspect of the present disclosure, the image synthesizing unit sets a second depth range based on a depth value in a subject area determined to be the area of the subject, and the depth value becomes the second depth range. An image that identifies the pixels may be generated as the depth mask image.

By doing so, it becomes possible to generate a depth mask image that reflects the movement of the subject, and it becomes possible to generate a high-quality composite image.

The block diagram which shows the configuration example of the image generation system of this embodiment. 2 (A) and 2 (B) are explanatory views of an example of tracking processing. The perspective view which shows the structural example of the housing. 4 (A) and 4 (B) are examples of virtual space images displayed on the player. 5 (A) and 5 (B) are examples of a composite image of a virtual space image and a real space image. 6 (A) and 6 (B) are explanatory views of synthesizing the image of the housing using the housing mask image. The explanatory view of the process which sets the extraction range of the subject image and extracts the subject image. 8 (A) and 8 (B) are explanatory views of a method of extracting an image of a subject by setting an extraction range using auxiliary points. The flowchart explaining the detailed processing example of this embodiment. 10 (A) to 10 (C) are explanatory views of the image composition process of the present embodiment. 11 (A) to 11 (C) are explanatory views of the image composition process of the present embodiment. 12 (A) and 12 (B) are explanatory views of a method of displaying a composite image of a virtual space image and an image of a subject on a display device for a gallery. 13 (A) and 13 (B) are explanatory views of an image composition process using a depth image. 14 (A) to 14 (D) are explanatory views of the correction process using the difference depth mask image. 15 (A) and 15 (B) are explanatory views of a correction process for setting the pixel values of the pixels for which the depth image could not be acquired based on the difference image. 16 (A) to 16 (C) are explanatory views of a correction process in which the pixel values of the pixels for which the depth image could not be acquired are set based on the difference image. 17 (A) and 17 (B) are explanatory views of correction processing by filter processing according to the area size. 18 (A) and 18 (B) are explanatory views of a process of setting a second depth range based on a depth value in a subject area and generating a depth mask image. 19 (A) to 19 (C) are explanatory views of a method using a housing mask image. 20 (A) to 20 (D) are explanatory views of a method using a housing mask image. 21 (A) to 21 (D) are explanatory views of a method that does not use a housing mask image. Explanatory drawing of the method of setting the extraction range.

Hereinafter, this embodiment will be described. It should be noted that the present embodiment described below does not unreasonably limit the description of the scope of claims. Moreover, not all of the configurations described in the present embodiment are essential configuration requirements.

1. 1. Image generation system FIG. 1 is a block diagram showing a configuration example of the image generation system of the present embodiment. The image generation system of the present embodiment realizes, for example, a simulation system that simulates virtual reality (VR). The image generation system of the present embodiment can be applied to various systems such as a game system that provides game content, a real-time simulation system such as a driving simulator or a sports competition simulator, or a content providing system that provides content such as video. .. The image generation system of the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting a part of its constituent elements (each part) or adding other constituent elements are possible. ..

The housing 30 is, for example, a ride housing on which a player can board. Specifically, the housing 30 is, for example, a movable housing that changes the play position of the player. The housing 30 is called, for example, an arcade housing, and is an outer shell of a device of a simulation system realized by an image generation system, and does not have to be box-shaped. The housing 30 may be a cockpit housing (experience housing) in a car game, a robot game, an airplane game, or the like, or may be a housing of another form. The housing 30 is the main body of the simulation system, and is provided with various devices and structures for realizing the simulation system. At least a play position is set in the housing 30. A configuration example of the housing 30 will be described in detail with reference to FIG. 3 described later.

The camera 150 is a device that captures an image such as a color image or a depth image. For example, the camera 150 includes a color camera 152 and a depth camera 154. The color camera 152 is a camera that captures a color image such as RGB, and can be realized by an image sensor such as a CMOS sensor or a CCD and an optical system such as a lens. The depth camera 154 is a camera capable of detecting the positional relationship of an object in the field of view in the depth direction, and a depth image can be acquired by using the depth camera 154. The depth image is, for example, an image in which a depth value, which is a Z value, is set as the pixel value of each pixel. For example, the depth camera 154 can be realized by the first and second depth sensors constituting the stereo camera and an IR projector that projects infrared rays. The first and second depth sensors can be realized by, for example, an infrared camera. For example, the accuracy of the measured depth value can be improved by measuring the depth value by the first and second depth sensors and projecting the IR pattern by the IR projector. Specifically, the points in the left eye image acquired by the first depth sensor are associated with the points in the right eye image acquired by the second depth sensor, and the depth is determined by the amount of shift between these points. Calculate the depth value, which is the value. The depth value measurement method is not limited to the above-mentioned method, and various modifications can be performed. Further, the color camera 152 and the depth camera 154 may be realized by one camera housing, or each of the color camera 152 and the depth camera 154 may be realized by a separate camera housing.

The operation unit 160 is for the player (user) to input various operation information (input information). The operation unit 160 can be realized by various operation devices such as a steering wheel, an accelerator pedal, a brake pedal, a lever, an operation button, a direction instruction key, a game controller, a gun type controller, a touch panel, or a voice input device.

The storage unit 170 stores various types of information. The storage unit 170 functions as a work area such as the processing unit 100 and the communication unit 196. A program for image generation processing and game processing, and data necessary for executing the program are stored in the storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), an HDD (hard disk drive), an SSD, an optical disk device, or the like. The storage unit 170 includes a virtual space information storage unit 172 and a drawing buffer 178.

The information storage medium 180, which is a medium that can be read by a computer, stores programs, data, and the like, and its function can be realized by an optical disk (DVD, BD, CD), an HDD, a semiconductor memory (ROM), or the like. .. The processing unit 100 performs various processes of the present embodiment based on the program (data) stored in the information storage medium 180. That is, the information storage medium 180 is a program for causing a computer (a device including an input device, a processing unit, a storage unit, and an output unit) to function as each part of the present embodiment (a program for causing the computer to execute processing of each part). Is memorized.

The HMD200 (head-mounted display device) is a device that is attached to the player's head and displays an image in front of the player's eyes. The HMD 200 is preferably a non-transparent type, but may be a transparent type. Further, the HMD 200 may be a so-called eyeglass type HMD. The HMD 200 can include a tracking device 206 for realizing a tracking process such as head tracking. The position and direction of the HMD 200 are specified by the tracking process using the tracking device 206. By specifying the position and direction of the HMD 200, the viewpoint position and line-of-sight direction of the player can be specified. Various methods can be adopted as the tracking method. In the first tracking method, which is an example of the tracking method, a plurality of light receiving elements (photodiodes, etc.) are provided as the tracking device 206. Then, by receiving the light (laser, etc.) from the light emitting element (LED, etc.) provided outside by these plurality of light receiving elements, the position of the HMD200 (player's head) in the three-dimensional space in the real world, Specify the direction. In the second tracking method, a plurality of light emitting elements (LEDs) are provided in the HMD 200 as the tracking device 206. Then, the position and direction of the HMD 200 are specified by capturing the light from these plurality of light emitting elements with an imaging unit provided outside. In the third tracking method, a motion sensor is provided as the tracking device 206, and the position and direction of the HMD 200 are specified using the motion sensor. The motion sensor can be realized by, for example, an acceleration sensor or a gyro sensor. For example, by using a 6-axis motion sensor using a 3-axis acceleration sensor and a 3-axis gyro sensor, the position and direction of the HMD200 in a three-dimensional space in the real world can be specified. The position and direction of the HMD 200 may be specified by a combination of the first tracking method and the second tracking method, or a combination of the first tracking method and the third tracking method. Further, instead of specifying the player's viewpoint position and line-of-sight direction by specifying the position and direction of the HMD 200, a tracking process that directly specifies the player's viewpoint position and line-of-sight direction may be adopted. For example, various viewpoint tracking methods such as eye tracking, face tracking or head tracking may be used. Further, the environment recognition camera may be used to perform recognition processing of the real space around the player, and the position and direction of the player may be specified based on the result of the recognition processing. For example, the position and direction of the player may be specified from the relative positional relationship with the recognized real space object.

The display unit 208 of the HMD 200 can be realized by, for example, an organic EL display (OEL) or a liquid crystal display (LCD). For example, the display unit 208 of the HMD 200 includes a first display or a first display area set in front of the left eye of the player and a second display or a second display area set in front of the right eye. It is provided so that a stereoscopic display is possible. In the case of stereoscopic display, for example, an image for the left eye and an image for the right eye having different parallax are generated, the image for the left eye is displayed on the first display, and the image for the right eye is displayed on the second display. To do. Alternatively, the image for the left eye is displayed in the first display area of one display, and the image for the right eye is displayed in the second display area. Further, the HMD 200 is provided with two eyepieces (fisheye lens), one for the left eye and the other for the right eye, thereby expressing a VR space that extends over the entire circumference of the player's field of view. Then, a correction process for correcting the distortion generated in the optical system such as the eyepiece is performed on the left eye image and the right eye image.

The gallery display device 210 is a device for displaying a gallery image, and can be realized by, for example, an LCD, an organic EL display, or a CRT. For example, the composite image generated by the present embodiment is displayed on the gallery display device 210, and the gallery, which is a spectator, can watch the player playing. The gallery display device 210 is installed in a facility of a simulation system realized by, for example, an image generation system.

The sound output unit 192 outputs the sound generated by the present embodiment, and can be realized by, for example, a speaker or headphones.

The I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function can be realized by an ASIC or the like for I / F processing. The portable information storage medium 195 is a storage device for the player to store various types of information, and holds the storage of such information even when the power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

The communication unit 196 communicates with an external device (another device) via a wired or wireless network, and its functions include hardware such as a communication ASIC or a communication processor, and communication firmware. Can be realized by.

The program (data) for operating the computer as each part of the present embodiment is distributed from the information storage medium of the server (host device) to the information storage medium 180 (or storage unit 170) via the network and the communication unit 196. You may. The use of information storage media by such a server (host device) can also be included within the scope of the present disclosure.

The processing unit 100 (processor) provides operation information from the operation unit 160, tracking information in the HMD 200 (information on at least one of the position and direction of the HMD, information on at least one of the viewpoint position and the line-of-sight direction), a program, and the like. Based on this, image acquisition processing, virtual space setting processing, game processing (simulation processing), virtual camera control processing, image generation processing, image composition processing, sound generation processing, and the like are performed.

Each process (each function) of the present embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the functions of each part may be realized by individual hardware, or the functions of each part may be realized by integrated hardware. For example, a processor includes hardware, which hardware can include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor may be composed of one or more circuit devices (for example, ICs, etc.) mounted on a circuit board, or one or more circuit elements (for example, resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. Further, the processor may be a hardware circuit by ASIC. Further, the processor may include an amplifier circuit, a filter circuit, and the like for processing an analog signal. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD), or an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and when the instructions are executed by the processor, the processing (function) of each part of the processing unit 100 is realized. The instructions here may be an instruction set that constitutes a program, or may be an instruction that instructs the hardware circuit of the processor to operate.

The processing unit 100 includes an acquisition unit 102, a virtual space setting unit 104, a game processing unit 106, a virtual camera control unit 110, an image generation unit 120, an image composition unit 122, and a sound generation unit 130. The game processing unit 106 includes a mobile processing unit 107 and a housing control unit 108. As described above, each process of the present embodiment executed by each of these parts can be realized by a processor (or a processor and a memory). It is possible to carry out various modifications such as omitting a part of these components (each part) or adding other components.

The acquisition unit 102 performs various information acquisition processes, and is an interface for information acquisition. For example, the acquisition unit 102 acquires an image taken by the camera 150. For example, a color image or a depth image taken by the camera 150 is acquired. Further, the acquisition unit 102 acquires player information including at least one of the player's position information (viewpoint position information), direction information (line-of-sight direction information), and posture information (motion information).

The virtual space setting unit 104 performs setting processing of the virtual space (object space) in which the object is arranged. For example, moving objects (cars, people, robots, trains, planes, ships, monsters or animals, etc.), maps (terrain), buildings, spectators' seats, courses (roads), items, trees, walls, water surfaces, etc. Performs processing to arrange and set various objects to be represented (objects composed of primitive surfaces such as polygons, free-form surfaces, and subdivision surfaces) in virtual space. That is, the position and rotation angle (synonymous with orientation and direction) of an object in the world coordinate system are determined, and the rotation angle (rotation angle around the X, Y, Z axis) is used at that position (X, Y, Z). Place the object. Specifically, in the virtual space information storage unit 172 of the storage unit 170, object information which is information such as the position, rotation angle, moving speed, or moving direction of an object (part object) in the virtual space corresponds to the object number. It will be remembered. That is, the object information is stored in the virtual space information storage unit 172 as virtual space information. The virtual space setting unit 104 updates the object information, which is the virtual space information, for each frame, for example.

The game processing unit 106 performs various game processing for the player to play the game. In other words, the game processing unit 106 (simulation processing unit) executes various simulation processes for the player to experience virtual reality (virtual reality). The game process calculates, for example, a process of starting a game when the game start condition is satisfied, a process of advancing the started game, a process of ending the game when the game end condition is satisfied, or a game result. Processing etc. The game processing unit 106 includes a mobile processing unit 107 and a housing control unit 108.

The moving body processing unit 107 performs various processing on the moving body moving in the virtual space. For example, a process of moving a moving body in a virtual space (object space, game space) or a process of operating a moving body is performed. For example, the mobile body processing unit 107 is based on the operation information input by the player by the operation unit 160, the tracking information acquired, the program (movement / motion algorithm), various data (motion data), and the like. Performs control processing to move a model object) in a virtual space or to move a moving body (motion, animation). Specifically, a simulation in which movement information (position, rotation angle, velocity, or acceleration) and motion information (position or rotation angle of a part object) of a moving body are sequentially obtained for each frame (for example, 1/60 second). Perform processing. A frame is a unit of time for performing movement / motion processing (simulation processing) and image generation processing of a moving body. The moving body is, for example, a player character (virtual player) corresponding to a player in real space, or a boarding moving body on which the player character is boarded. The boarding mobile is a mobile that imitates a vehicle such as a car, a ship, a boat, an airplane, a tank, or a robot that appears in a virtual space, for example. Then, the player character is boarded on the boarding mobile body in the virtual space corresponding to the housing 30 in the real space. The moving body processing unit 107 performs a process of moving such a boarding moving body in the virtual space or moving the player character into the virtual space.

The housing control unit 108 performs control processing for the housing 30. For example, the movable mechanism of the housing 30 is controlled to perform a control process for changing the posture and position of the housing 30 in the real space. For example, by changing the posture and position of the housing 30, the play position of the player boarding the housing 30 changes.

The virtual camera control unit 110 controls the virtual camera. For example, the virtual camera is controlled based on the player's operation information and tracking information input by the operation unit 160. Specifically, it controls a virtual camera for the player. For example, by setting a virtual camera for the player at a position corresponding to the viewpoint (first-person viewpoint) of the player character moving in the virtual space and setting the viewpoint position and line-of-sight direction of the virtual camera, the position (position) of the virtual camera is set. Controls the coordinates) and posture (rotation angle around the rotation axis). Alternatively, the virtual camera for the player is set at the position of the viewpoint (third person viewpoint) that follows the moving object such as the player character or the boarding mobile body, and the viewpoint position and the line-of-sight direction of the virtual camera are set. Control the position and posture of the camera.

For example, the virtual camera control unit 110 controls the virtual camera for the player so as to follow the change in the viewpoint of the player based on the tracking information of the viewpoint information of the player acquired by the viewpoint tracking. For example, in the present embodiment, tracking information (viewpoint tracking information) of viewpoint information that is at least one of the player's viewpoint position and line-of-sight direction is acquired. This tracking information can be obtained, for example, by performing tracking processing of HMD200. Then, the virtual camera control unit 110 changes the viewpoint position and the line-of-sight direction of the virtual camera for the player based on the acquired tracking information (information on at least one of the viewpoint position and the line-of-sight direction of the player). For example, the virtual camera control unit 110 virtually changes the viewpoint position and line-of-sight direction (position, posture) of the virtual camera in the virtual space according to changes in the viewpoint position and line-of-sight direction of the player in the real space. Set the camera. By doing so, the virtual camera can be controlled so as to follow the change in the viewpoint of the player based on the tracking information of the viewpoint information of the player.

The virtual camera control unit 110 also controls a virtual camera for capturing a virtual space image to be combined with the real space image. For example, a virtual camera for shooting is arranged at a position in the virtual space corresponding to the position of the camera 150 in the real space. Then, the virtual space image seen from the virtual camera for shooting is combined with the real space image taken by the camera 150.

The image generation unit 120 performs a virtual space image generation process. Virtual space images are game images and simulation images. For example, the image generation unit 120 performs drawing processing based on the results of various processes (game processing, simulation processing) performed by the processing unit 100, thereby generating a virtual space image. For example, a virtual space image for a player that can be seen from a virtual camera for a player is displayed on the HMD 200. The virtual space image seen from the virtual camera for shooting is combined with the real space image. Specifically, geometry processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, fluoroscopic conversion, or light source processing is performed, and drawing data (positions of vertices on the primitive surface) is performed based on the processing results. Coordinates, texture coordinates, color data, normal vector or α value, etc.) are created. Then, based on this drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is subjected to image information in pixel units such as a drawing buffer 178 (frame buffer, work buffer, etc.). Draw in a memorable buffer). As a result, an image that can be seen from a virtual camera (given viewpoint. First and second viewpoints for the left eye and the right eye) is generated in the virtual space. The drawing process performed by the image generation unit 120 can be realized by a vertex shader process, a pixel shader process, or the like.

The image synthesizing unit 122 performs a synthesizing process of the virtual space image and the real space image to generate a composite image. For example, the image synthesizing unit 122 generates a composite image in which an image of a subject in real space is combined with a virtual space image. The composite image generated by the image compositing unit 122 is displayed on the gallery display device 210 as a gallery image.

The sound generation unit 130 performs sound processing based on the results of various processes performed by the processing unit 100. Specifically, a game sound such as a musical piece (music, BGM), a sound effect, or a voice is generated, and the game sound is output to the sound output unit 192.

Then, as shown in FIG. 1, the image generation system of the present embodiment includes an acquisition unit 102, an image generation unit 120, and an image composition unit 122.

The acquisition unit 102 acquires the first image in which the background and the subject are photographed by the camera 150. In addition, a second image in which the background is photographed by the camera 150 is acquired. For example, the acquisition unit 102 acquires the first image and the second image, which are color images taken by the color camera 152. In the first image, both the background and the subject are photographed, but in the second image, the background is photographed but the subject is not photographed. The subject is a real-space object to be photographed by the camera, for example, a player. However, the subject of the present embodiment is not limited to this, and may be a subject other than the player.

The image generation unit 120 generates a virtual space image that can be seen from the virtual camera in the virtual space. For example, in a virtual space which is an object space in which a plurality of objects are arranged, a virtual space image which is an image seen from a virtual camera is generated. For example, the image generation unit 120 generates a virtual space image for the player that can be seen from the virtual camera for the player in the virtual space. By displaying the virtual space image for the player generated in this way on the HMD 200, the player can experience the world of VR. The virtual space image for the player generated by the image generation unit 120 may be displayed on a display device of a type different from that of the HMD 200. For example, it may be displayed on a display device having a dome-shaped display screen that covers the player's field of view.

Further, the image generation unit 120 generates a virtual space image that can be seen from a shooting virtual camera arranged at a position in the virtual space corresponding to the position of the camera 150. The virtual camera for shooting is a virtual camera for shooting a virtual space image to be combined with the real space image shot by the camera 150. The virtual camera for shooting is arranged at a position corresponding to the position of the camera 150 in the real space in the virtual space. As an example, the camera 150 is arranged in front of the subject in the real space so as to face the subject. That is, the camera 150 is arranged so that the subject is within the shooting range. In this case, the virtual camera for shooting is arranged in front of the object in the virtual space corresponding to the subject so as to face the object. In this case, the camera distance between the virtual camera for shooting and the object is set to a distance corresponding to the camera distance between the camera 150 and the subject. Taking the case where the subject is a player as an example, the camera 150 is arranged in front of the player in the real space so as to face the player. Then, the virtual camera for shooting is arranged in front of the player character (player moving body) corresponding to the player so as to face the player character. In this case, the distance between the virtual camera for shooting and the player character is set to a distance corresponding to the distance between the camera 150 and the player.

The image synthesizing unit 122 extracts the image of the subject by obtaining the difference image between the first image and the second image. That is, the process of extracting the image of the subject from the difference image between the first image in which the background and the subject are reflected and the second image in which the background is reflected is performed. Then, the image composition unit 122 generates a composite image in which the image of the subject is combined with the virtual space image. This composite image is displayed, for example, on the gallery display device 210.

The display device on which the composite image is displayed is not limited to the gallery display device 210. For example, a composite image may be distributed via a network and displayed on a terminal device such as a PC or a game device. As the distribution, for example, distribution using the Internet, a server, or the like can be considered. Alternatively, a composite image in which the image of the subject is combined with the virtual space image may be displayed in a part of the display area of the HMD 200.

Further, the image synthesizing unit 122 extracts an image of the housing 30 on which the subject is boarded in the real space. Then, a composite image is generated in which the image of the subject and the image of the housing 30 are combined with the virtual space image. This composite image is displayed on a display device such as a gallery display device 210. For example, a player who is a subject enjoys playing a VR game (simulation game) by boarding a housing 30 which is a riding housing. For example, in the virtual space image, a moving body such as a car, a ship, or a robot corresponding to the housing 30 is displayed. When the housing 30 is a movable housing, the housing 30 is controlled so that the posture of the housing 30 changes. As a result, the play position of the player boarding the housing 30 changes variously. The image composition unit 122 generates not only an image of a subject such as a player, but also a composite image in which such an image of the housing 30 is combined with a virtual space image. As a result, when the player is boarding the housing 30 in the real space, the real space image showing the state in which the player is boarding the housing 30 can generate a composite image synthesized with the virtual space image. Then, this composite image is displayed on a display device such as a gallery display device 210.

Further, the image synthesizing unit 122 extracts the image of the housing 30 by using the housing mask image that specifies the extraction range of the image of the housing 30. For example, the image of the housing 30 is extracted from the color image such as the first image. For example, the operator manually performs an operation such as tracing the outline of the housing 30 to specify the extraction range, so that the housing mask image for specifying the range is generated. It is also possible to perform deformation to automatically recognize the extraction range (edge) of the image of the housing 30 by image processing and generate a housing mask image. The housing mask image is a mask image for identifying the extraction range of the housing 30 from other backgrounds. By designating the extraction range by this housing mask image, the image of the housing 30 is extracted in the same manner as the subject even though it is actually an image corresponding to the background, and becomes a virtual space image. It will be synthesized. Details of the extraction process using the housing mask image will be described later.

Further, the image synthesizing unit 122 sets an extraction range of the image of the subject based on the tracking information from at least one tracking device worn by the subject, and extracts the image of the subject. For example, the position information of the tracking device is acquired based on the tracking information from the tracking device worn by the subject as a wearable device, and the extraction range of the image of the subject is set based on the acquired position information and the like. For example, when the subject is equipped with a plurality of tracking devices, the range including the positions of the plurality of tracking devices is set as the extraction range of the image of the subject, and the extraction range is set as the target range of the extraction process. Extract the image of. The tracking device may be a tracking device 206 built in the HMD 200 worn by the subject.

Further, the image synthesizing unit 122 sets the extraction range of the image of the subject based on the position of the tracking device and the position of the auxiliary point set to the position shifted by a predetermined distance from the position of the tracking device. For example, information on the placement direction of the tracking device is acquired from the tracking information. Then, a position shifted (offset) by a predetermined distance from the position of the tracking device along the arrangement direction of the tracking device is set as the position of the auxiliary point. Then, the extraction range is set to include the position of the tracking device and the position of the auxiliary point, and the image of the subject is extracted.

Further, the image generation unit 120 sets the image of the virtual camera and the image of the photographer character at a position corresponding to the position of the virtual camera for shooting in the virtual space as a virtual space image for the player displayed on the player as the subject. Generate a virtual space image that displays at least one. For example, a virtual camera object that is a three-dimensional model representing a virtual camera for shooting is arranged at a position and direction corresponding to the position and direction of the virtual camera for shooting. Alternatively, the image of the photographer character to be photographed using the virtual camera for photographing is arranged at a position corresponding to the position of the virtual camera. Then, the photographer character performs an effect process such as performing a motion of the shooting operation or uttering a line related to the shooting.

Further, the image generation system (simulation system) of the present embodiment is mounted on the player who is the subject and displays the virtual space image for the player which can be seen from the virtual camera for the player in the virtual space, and the composite image for the gallery. Includes a gallery display device 210 that displays images. By doing so, the virtual space image for the player generated by the image generation unit 120 is displayed on the HMD200, and the player displays the image seen from the virtual camera for the player in the virtual space as the VR image of the HMD200. It will be possible to see through. On the other hand, the composite image in which the image of the subject is combined with the virtual space image is displayed on the gallery display device 210, and the gallery shows what kind of situation the player is in in the virtual space and what kind of action is taken. You can see if it is by looking at the image for the gallery.

Further, the acquisition unit 102 acquires a depth image of the background and the subject taken by the camera 150. For example, the depth information of the background and the subject when viewed from the camera 150 (depth camera 154) is acquired as a depth image. Then, the image synthesizing unit 122 extracts the image of the subject based on the difference image and the depth image. That is, the image of the subject is extracted by using the difference image between the first image showing the subject and the background and the second image showing the background, and the depth image. For example, the image of the subject is extracted by using the pixels determined to be the pixels corresponding to the subject by both the difference image and the depth image as the pixels of the subject.

Further, the image synthesizing unit 122 generates a difference mask image based on the difference image. For example, a difference mask image is generated by performing a binarization process of the difference image. In this case, the image synthesizing unit 122 may generate a difference mask image based on the difference image and the depth image (depth mask image). For example, the difference mask image may be generated by ANDing (logical producting) the mask image obtained by performing the binarization process of the difference image and the depth mask image obtained from the depth image. In this case, the depth mask image to be ANDed may be a depth mask image before correction processing, which will be described later, or a depth mask image after correction processing. Further, the image synthesizing unit 122 generates a depth mask image that identifies pixels having a depth value within a given depth range (depth effective range) based on the depth image. For example, the depth range is a range in which the depth value is equal to or greater than the first depth value on the near side and equal to or less than the second depth value on the far side. The image composition unit 122 generates a depth mask image in which the pixels in the depth range have the first pixel value (for example, the white pixel value) and the pixels outside the depth range have the second pixel value (for example, the black pixel value). .. Then, the image synthesizing unit 122 generates a subject mask image for identifying the subject based on the difference mask image and the depth mask image. For example, a subject mask image is generated such that the pixels in the range of the subject have the first pixel value (for example, the white pixel value) and the pixels outside the range of the subject have the second pixel value (for example, the black pixel value). Then, the image synthesizing unit 122 extracts the image of the subject based on the subject mask image and the first image. For example, in the first image, the range specified by the subject mask image is determined to be the subject image, and the subject image is extracted.

Further, the image synthesizing unit 122 corrects the depth mask image and generates a subject mask image based on the corrected depth mask image and the difference mask image. For example, the image synthesizing unit 122 performs correction processing for removing noise or the like on a depth mask image that identifies a pixel whose depth value is in a given depth range. Then, a subject mask image is generated based on the depth mask image after the correction process and the difference mask image obtained from the difference image between the first image and the second image.

For example, the image synthesizing unit 122 creates a difference depth mask image between the first depth image of the background and the subject taken by the camera 150 (depth camera 154) and the second depth image of the background taken by the camera 150 (depth camera 154). Generate. That is, a difference depth mask image, which is a difference image between the first depth image and the second depth image, is generated. Then, the image synthesizing unit 122 generates a depth mask image after the correction process based on the difference depth mask image. That is, as a correction process for the depth mask image, a process for generating a difference depth mask image is performed.

Further, the image composition unit 122 generates a depth mask image after the correction process by performing at least one of the morphology filter process and the time series filter process. That is, as the correction process of the depth mask image, at least one of the morphology filter process and the time series filter process is performed to remove noise and the like. The morphology filtering process is an expansion / contraction filtering process. The time-series filter process is a smoothing process, which is a process of enabling as a mask when, for example, there are a predetermined number of frames or more and a continuous difference value of a predetermined value or more.

Further, the image synthesizing unit 122 generates a depth mask image after the correction process by performing a process of setting the pixel values of the pixels for which the depth value could not be acquired in the depth image based on the difference image. For example, with the stereo camera of the depth camera 154, the pixels for which the depth value could not be acquired are set as blank pixels. Then, the blank pixel is set based on the difference image between the first image and the second image, which are color images. For example, using a difference mask image based on the difference image, a process of filling the pixel values of blank pixels in the depth mask image is performed.

Further, the image synthesizing unit 122 obtains the area size of the pixel group (adjacent pixel group) whose depth value is in the depth range. For example, the area size is obtained by counting the pixels of a pixel group whose depth value is in a given depth range. Then, the pixel group of the region having the largest region size or the region having the region size of a predetermined size or more is determined as the pixel group corresponding to the subject, and the depth mask image is generated.

Further, the image composition unit 122 sets the second depth range based on the depth value in the subject area determined to be the area of the subject. For example, a second depth range narrower than the above depth range is set. Then, an image that identifies the pixel whose depth value is in the second depth range is generated as a depth mask image. That is, the pixel group whose depth value is in the second depth range is determined to be the pixel group corresponding to the subject, and the depth mask image is generated.

In the present embodiment, virtual reality simulation processing is performed as game processing of the game played by the player. The virtual reality simulation process is a simulation process for simulating an event in the real space in the virtual space, and is a process for allowing the player to experience the event virtually. For example, a moving body such as a player character corresponding to a player in a real space or a moving body on board the player is moved in a virtual space, or a process is performed to allow the player to experience changes in the environment and surroundings due to the movement.

Further, the processing of the image generation system of the present embodiment of FIG. 1 includes a processing device such as a business game device and a home game device, a processing device such as a PC installed in a facility, and a processing device worn by a player on the back or the like ( It can be realized by backpack PC) or distributed processing of these processing devices. In this case, for example, a processing device that realizes the processing of the image synthesizing unit 122 and a processing device that realizes other processing such as the image generation unit 120 and the game processing unit 106 may be realized by another hardware device. Alternatively, the processing of the image generation system of the present embodiment may be realized by the server system and the terminal device. For example, it may be realized by distributed processing of the server system and the terminal device.

2. 2. Tracking process Next, an example of tracking process will be described. FIG. 2A shows an example of HMD200. The HMD 200 is provided with a plurality of light receiving elements 201, 202, 203 (photodiodes). The light receiving elements 201 and 202 are provided on the front surface side of the HMD 200, and the light receiving elements 203 are provided on the right side surface of the HMD 200. Further, a light receiving element (not shown) is also provided on the left side surface, the upper surface, and the like of the HMD. Further, the player PL has the tracking devices 250 and 260 attached to the hand, which is a predetermined portion. Similar to the HMD 200, the tracking device 250 mounted on the right hand is provided with a plurality of light receiving elements 251 to 254. The tracking device 260 mounted on the left hand is also provided with a plurality of light receiving elements 261 to 264 (not shown). By using the tracking devices 250 and 260 provided with such a light receiving element, it becomes possible to specify the part information such as the position and direction of a predetermined part such as a hand, as in the case of the HMD200. The predetermined portion of the player PL on which the tracking device is mounted is not limited to the hand, and various regions such as the legs, head, chest, abdomen, and hips can be assumed.

As shown in FIG. 2B, base stations 280 and 284 are installed around the housing 30. The base station 280 is provided with light emitting elements 281 and 282, and the base station 284 is provided with light emitting elements 285 and 286. The light emitting elements 281, 282, 285, and 286 are realized by LEDs that emit a laser such as an infrared laser. The base stations 280 and 284 use these light emitting elements 281, 282, 285 and 286 to emit, for example, a laser radially. Then, the light receiving elements 201 to 203 and the like provided in the HMD200 of FIG. 2A receive the laser from the base stations 280 and 284, so that the tracking process of the HMD200 is realized, and the position and orientation of the head of the player PL are oriented. The direction (viewpoint position, line-of-sight direction) can be detected. For example, the position information and attitude information (direction information) of the player PL can be detected.

Further, the light receiving elements 251 to 254 and 261 to 264 provided in the tracking devices 250 and 260 receive the laser from the base stations 280 and 284 to receive at least one of the positions and directions of the hands (predetermined parts) of the player PL. It will be possible to detect. Note that the tracking devices 250 and 260 may be realized by a Leap Motion tracker.

A camera 150 having a color camera 152 and a depth camera 154 is installed in front of the housing 30. For example, the camera 150 is arranged in front of the player PL and the housing 30 so as to face the player PL and the housing 30. Then, the virtual camera VCM for shooting described with reference to FIGS. 4 (A) and 4 (B) described later is arranged at a position in the virtual space corresponding to the position of the camera 150. For example, the virtual camera VCM is arranged in front of a moving body such as a player character corresponding to the player PL or a car corresponding to the housing 30 so as to face the player character or the moving body. Further, the camera distance between the player character or the moving body and the virtual camera VCM is also set to a distance corresponding to the camera distance between the player PL or the housing 30 and the camera 150.

3. 3. Case Figure 3 shows a configuration example of the case 30. In the housing 30 of FIG. 3, a cover portion 33 is provided on the bottom portion 32, and a base portion 34 (pedestal portion) is provided on the cover portion 33. The base portion 34 is provided so as to face the bottom portion 32. The base portion 34 is provided with a ride portion 60 having a seat 62. The player PL is seated on the seat 62 of the ride unit 60. Further, the base portion 34 is provided with a moving portion 40, and the moving portion 40 blows air to an operating unit 160 such as a handle 50, an accelerator pedal 52, and a brake pedal 54 (not shown), and a player. A blower 80 is provided. The moving portion 40 is movable in the front-rear direction.

A camera 150 is arranged in front of the housing 30. By providing such a camera 150, it becomes possible to take a picture of the housing 30 and the player boarding the housing 30.

The housing 30 changes the play position of the player according to the result of the game processing (game situation). For example, in the present embodiment, a virtual reality simulation process is performed as a game process of a game played by a player. For example, a moving body (car, robot, etc.) on which a player character corresponding to a player in a real space is boarded is moved in a virtual space, and processing is performed so that the player can experience changes in the environment and surroundings due to the movement. Then, the housing 30 changes the play position based on the result of the simulation process which is the game process. For example, the play position is changed based on the result of movement processing in the virtual space of the moving body (or player character) on which the player character is boarded. For example, in a racing game, a process of changing the play position is performed as a simulation process for the player to experience acceleration or deceleration when the car (racing car) is moving or acceleration due to a change in direction. Alternatively, when an attack from an enemy hits a car, a process of changing the play position is performed as a simulation process for allowing the player to experience the impact. Here, the play position is a play position in which the player is located when playing a virtual reality (VR) simulation game. For example, the play position is the ride position of the ride unit 60 of the player.

Specifically, in FIG. 3, between the bottom portion 32 and the base portion 34, first to fourth air spring portions (expandable portions in a broad sense) (not shown) are provided at four corners thereof. These first to fourth air spring portions expand and contract in the Y-axis direction of FIG. 3 by supplying and discharging air using an air compressor or a bubble. For example, the base portion 34 can be moved upward or downward in the Y-axis direction by expanding or contracting all the first to fourth air spring portions. By moving the base portion 34 in the vertical direction, for example, it is possible to reproduce the road surface condition of the course. For example, by moving the base portion 34 in the vertical direction with a small stroke and at a high speed, unevenness (rattling road) of the road surface can be expressed.

Further, one of the front and rear air spring portions of the first to fourth air spring portions at the four corners extends and the other air spring portion contracts, so that the base portion 34 can be pitched around the X axis. it can. Further, the base portion 34 can be rolled around the Z axis by extending one of the left and right air spring portions of the first to fourth air spring portions at the four corners and contracting the other air spring portion. By performing such pitching and rolling, the player can experience the feeling of acceleration and deceleration associated with the movement of a moving body such as a car, and the inertial force at the time of cornering. As a result, the virtual reality of the player can be improved, and so-called 3D sickness can be suppressed.

4. Method of this embodiment Next, the method of this embodiment will be described. In the following, a case where the method of the present embodiment is applied to a racing game in which a player character gets into a car and competes with another car will be mainly described. However, the game to which the method of the present embodiment is applied is not limited to such a racing game. For example, the method of the present embodiment includes various games other than racing games (competition games other than cars, robot games, fighting games such as FPS (First Person shooter) and fighting games, simulation games for vehicles such as trains and airplanes, and RPGs. , Action games, virtual experience games, sports games, horror experience games, music games, etc.), and can be applied to other than games. Further, in the following description, the case where the subject of the camera 150 is a player will be mainly taken as an example.

4.1 Description of the game FIGS. 4 (A) and 4 (B) are examples of virtual space images for the player displayed to the player in the racing game realized by the present embodiment. In this embodiment, the virtual space images for the players of FIGS. 4 (A) and 4 (B) are displayed on the HMD 200 worn by the player. In this racing game, a player character in a virtual space corresponding to a player in the real space acts as a driver, rides on a moving car (racing car), and competes with an enemy car. An item area is set on the course, and by acquiring and using items in this item area, you can advance the racing game to your advantage.

In the virtual space images (game images) for the players of FIGS. 4 (A) and 4 (B), the player's car MV (moving body) and enemy car MV (enemy moving body) traveling on the course of the virtual space are displayed. Has been done. In addition, parts such as the steering wheel ST of the car MV and the hands HR and HL of the player character who operates the steering wheel ST are also displayed. Then, when the player in the real space steers the steering wheel 50 in FIG. 3 to the left, the steering wheel ST of the car MV is steered to the left by the hands HR and HL of the player character even in the virtual space as shown in FIG. 4 (A). A virtual space image is displayed. Further, when the player PL in the real space steers the steering wheel 50 to the right, a virtual space image in which the steering wheel ST is steered to the right by the player character's hands HR and HL is displayed even in the virtual space as shown in FIG. 4 (B). Will be done. That is, a virtual space image is displayed in which the hands HR and HL of the player character in the virtual space move in conjunction with the movement of the player's hand in the real space. By doing so, it is possible to give the player a virtual reality as if he / she is driving a real car. Further, in FIGS. 4 (A) and 4 (B), an image of a virtual camera VCM for taking a virtual space image to be combined with the real space image is also displayed.

5 (A) and 5 (B) are examples of a composite image of the virtual space image and the real space image generated by the present embodiment. The images of the player PL and the housing 30 in FIGS. 5A and 5B are real space images of the player PL and the housing 30 taken by the camera 150 of FIGS. 2B and 3B. Specifically, it is a real space image obtained by extracting a player PL or the like from a captured image of the camera 150 by the background subtraction method. On the other hand, the images of the enemy character CHE, the enemy vehicle MVE, the effects EF1, EF2, the course CS, and the like are virtual space images taken by the virtual camera VCM for shooting in FIGS. 4 (A) and 4 (B). By synthesizing the real space image of the player PL and the housing 30 with the virtual space image seen from the virtual camera VCM for shooting, the composite images of FIGS. 5 (A) and 5 (B) are generated. .. This composite image is displayed on, for example, a gallery display device 210 as shown in FIG. 12B described later. As a result, the gallery can grasp how the player PL enjoys playing the game in the virtual space.

In FIG. 5A, since the enemy character CHE and the enemy vehicle MVE are located behind the player PL and the housing 30 when viewed from the virtual camera VCM for shooting, the enemy character CHE and the enemy vehicle MVE The image of the person is erased. On the other hand, in FIG. 5B, since the player PL and the housing 30 are located behind the enemy character CHE and the enemy vehicle MVE when viewed from the virtual camera VCM for shooting, the player PL and the housing 30 The image on the other side is erased. As described above, in the present embodiment, the hidden surface is erased by reflecting the positional relationship in the virtual space of the player PL and the housing 30 which are real space images. Further, in the present embodiment, for example, a lighting process (shading process) for the player PL and the housing 30 is performed based on the lighting information in the virtual space. For example, when the brightness of the virtual space is dark, the images of the player PL and the housing 30 are also darkened, and when the virtual space is bright, the images of the player PL and the housing 30 are also brightened. It is said.

In the present embodiment, the first image in which the background and the subject are photographed by the camera 150 arranged in the real space and the second image in which the background is photographed by the camera 150 are acquired. The subject is a player PL or the like. Further, a virtual space image that can be seen from the virtual camera VCM for shooting arranged at the position of the virtual space corresponding to the position of the camera 150 is generated. Then, by obtaining the difference image between the first image and the second image, the image of the subject such as the player PL is extracted, and as shown in FIGS. 5 (A) and 5 (B), the subject is displayed in the virtual space image. Generates a composite image in which the images are combined. By doing so, it becomes possible to generate a composite image as if a subject in the real space appeared in the virtual space.

Further, in the present embodiment, an image of the housing 30 on which the player PL, which is the subject, is boarded in the real space is extracted. Then, as shown in FIGS. 5A and 5B, a composite image is generated in which the image of the player PL as the subject and the image of the housing 30 are combined with the virtual space image. In this way, not only the player PL, which is the subject, but also the housing 30 on which the player PL is boarded in the real space can generate a composite image as if it appeared in the virtual space.

In this case, in the present embodiment, the housing mask image MSC as shown in FIG. 6 (A) is used to extract the image of the housing 30 as shown in FIG. 6 (B). The housing mask image MSC is a mask image that specifies the extraction range of the image of the housing 30, and is created manually by an operator as an example. For example, the housing mask image MSC is created by the operator specifying the extraction range by performing an operation such as tracing the outline of the housing 30 while looking at the captured image of the housing 30. In the housing mask image MSC, for example, the pixel value in the extraction range of the image of the housing 30 is the first pixel value (white pixel value), and the pixel value outside the extraction range is the second pixel value (black pixel value). It is a mask image that becomes. By using such a housing mask image MSC, it becomes possible to extract an image of the housing 30, which is originally non-extracted as a background, in the same manner as the image of the subject. A specific example of the extraction process using the housing mask image MSC will be described in detail in FIGS. 19 (A) to 21 (D) described later.

Further, in the present embodiment, as a virtual space image for the player displayed on the player as the subject, the image of the virtual camera for shooting and the photographer character are placed at a position corresponding to the position of the virtual camera for shooting in the virtual space. Generate a virtual space image that displays at least one of the images. For example, in FIG. 4A, an image of a virtual camera VCM for shooting is displayed in front of the vehicle MV on which the player character is boarding. Further, in FIG. 4B, an image of the photographer character CHM is displayed in front of the vehicle MV. The photographer character CHM points the virtual camera VCM toward the car MV and mutters the line "Look at this".

As described with reference to FIGS. 2B and 3, the camera 150 in the real space is arranged in front of the housing 30 on which the player PL is boarded. Specifically, the camera 150 in the real space is arranged in front of the housing 30 at a position separated by a given camera distance with the camera direction facing the housing 30. Then, in the present embodiment, the virtual camera VCM for shooting is arranged at a position in the virtual space corresponding to the position of the camera 150 in the real space. That is, the virtual camera VCM for shooting is arranged in front of the car MV in the virtual space at a position separated by a distance corresponding to the above-mentioned camera distance, with the camera direction facing the car MV.

Then, in FIGS. 4A and 4B, the image of the virtual camera VCM for shooting corresponding to the camera 150 in the real space is displayed in the virtual space image in this way. Normally, the image of the virtual camera VCM is not displayed in such a virtual space image, but in FIG. 4A, the image of the virtual camera VCM is intentionally displayed. Further, in FIG. 4B, the image of the photographer character CHM possessing the virtual camera VCM is displayed in the virtual space image. For example, an image of a photographer character CHM that shoots while flying in the air with the virtual camera VCM directed toward the player character is displayed.

In this way, the player becomes aware that he / she is being photographed by the virtual camera VCM. Then, when the player performs an action such as waving to the virtual camera VCM that is shooting itself, the player PL also waves the hand in the composite images of FIGS. 5 (A) and 5 (B). The real space image obtained by the above operation is combined with the virtual space image. Then, by displaying such a composite image on the gallery display device 210 as shown in FIG. 12B, it is possible to realize an effect that the player and the gallery are united and excited.

The present embodiment is also applicable to a multiplayer game. In this case, the first to nth housings (n is an integer of 2 or more) on which the first to nth players are boarded are provided. Further, at a position such as in front of the first to nth housings, the first to nth cameras for photographing the first to nth housings and the first to nth players are installed. Then, when generating a gallery image for the i-th player (i is an integer such that 1 ≦ i ≦ n) among the first to n-th players, the i-th player corresponding to the i-th player is generated. The virtual camera VCM for shooting is moved to a position such as in front of the character, and the i-th moving body on which the i-th player character or the i-th player character is boarded is photographed. Then, a composite image is generated by synthesizing the real space image taken by the third camera that shoots the i-th housing and the i-player and the virtual space image that can be seen from the virtual camera VCM for shooting. Then, the generated composite image is displayed on the gallery display device 210. In this way, the i-th player can recognize that he / she is the shooting target by displaying the virtual camera VCM for shooting in front of himself / herself, and the virtual camera for shooting can be recognized. Actions such as waving at the VCM and the photographer character CHM will be performed. As a result, it is possible to realize a production effect in which the player and the gallery are united and excited.

FIG. 7 is an example of application of this embodiment to a VR game of high altitude experience of crossing a narrow bridge hung between buildings. In the present embodiment, the extraction range of the image of the subject is set based on the tracking information from at least one tracking device worn by the subject, and the image of the subject is extracted. For example, in FIG. 7, the player PL is equipped with tracking devices TR1, TR2, TR3, and TR4 as described in FIG. 2A on the left hand, right hand, left foot, and right foot. By using such tracking devices TR1 to TR4, the movements of the limbs of the player PL can be tracked, and the posture and movement of the player PL can be detected. Then, in the present embodiment, the extraction range AR of the image of the player PL as the subject is set based on the tracking information from these tracking devices TR1 to TR4, and the image of the player PL is extracted. More specifically, in FIG. 7, since the HMD200 also has the tracking device TR5 built-in, the extraction range AR is determined based on the tracking information from the tracking devices TR1 to TR4 mounted on the limbs and the tracking device TR5 provided on the HMD200. Set. For example, as tracking information from the tracking devices TR1 to TR5, information such as the positions of the tracking devices TR1 to TR5 is acquired, and the range including the positions of the tracking devices TR1 to TR5 is set in the extraction range AR. Then, in this extraction range AR, the image extraction process of the player PL is performed. By doing so, even when the posture of the player changes or various operations are performed in the VR game, the image extraction process of the player PL in the appropriate extraction range AR can be realized. In addition, it is possible to prevent a situation in which a person who is not scheduled as a subject is mistakenly extracted as a subject.

Further, in the present embodiment, the extraction range of the image of the subject is set based on the position of the tracking device and the position of the auxiliary point set at the position shifted (offset) by a predetermined distance from the position of the tracking device. .. For example, in FIGS. 8 (A) and 8 (B), the player PL is on the elevator of the building. In this case, if the extraction range AR is set based only on the positions of the tracking devices TR1 to TR5 as shown in FIG. 8A, for example, the back portion of the player PL will not be included in the extraction range AR, and proper extraction will be performed. Processing cannot be realized.

Therefore, in the present embodiment, as shown in FIG. 8B, an auxiliary point PAX for setting the extraction range AR is set at a position shifted from the position of the tracking device TR5 of the HMD200 mounted on the player PL, for example. For example, the direction information of the HMD200 is also acquired based on the tracking information from the tracking device TR5 of the HMD200. Using this direction information, the auxiliary point PAX is set at a position shifted backward by a given distance from the position of the HMD 200. Then, the extraction range AR of the player PL, which is the subject, is set based on the positions of the tracking devices TR1 to TR5 and the positions of the auxiliary points PAX. By doing so, as shown in FIG. 8B, it becomes possible to set the extraction range AR that includes all of the player PL, and it is possible to realize an appropriate extraction process of the image of the player PL that is the subject. become.

4.2 Image composition processing Next, a detailed example of the image composition processing of the present embodiment for synthesizing a virtual space image and a real space image will be described. FIG. 9 is a flowchart illustrating a detailed processing example of the present embodiment.

First, a second image in which the background is photographed by the camera 150 is acquired (step S1). For example, in FIGS. 2B and 3B, when the player is not on the housing 30, the camera 150 acquires a second image of the background including the housing 30 and the like. Then, it is determined whether or not the game has started (step S2), and when the game has started, it is determined whether or not the frame has been updated (step S3). For example, in the present embodiment, a virtual space image or a composite image is generated every time a frame is updated.

In the case of frame update, the virtual camera generates a virtual space image that can be seen from the virtual camera (step S4). For example, a virtual space image seen from a virtual camera for shooting and a virtual space image for a player displayed on the player's HMD200 are generated. Further, the camera 150 acquires a first image of the background and the subject (step S5). That is, the background and the first image in which the player as the subject is reflected are captured by the camera 150 and acquired.

Next, the image of the subject is extracted by obtaining the difference image between the first image and the second image (step S6). That is, as will be described later, an image of the player who is the subject is extracted using the background subtraction method. Then, a composite image in which the image of the subject is combined with the virtual space image is generated (step S7). That is, as shown in FIGS. 5 (A) and 5 (B), a real space image (live-action image) of the player PL or the like as the subject is used with respect to the virtual space image in which the enemy character CHE, the enemy vehicle MVE, the course CS, etc. are displayed. ) Generates a composite image. Then, the composite image is displayed on the gallery display device 210 (step S8). The virtual space image for the player that can be seen from the virtual camera for the player is displayed on the HMD 200. Then, it is determined whether or not the game is finished (step S9), and if the game is not finished, the process returns to step S3, and if the game is finished, the process is finished.

As described above, in the present embodiment, the first image in which the background and the subject are photographed by the camera 150 arranged in the real space and the second image in which the background is photographed by the camera 150 are acquired. In FIG. 9, the second image is acquired in step S1 and the first image is acquired in step S5. Further, a virtual space image that can be seen from a virtual camera for shooting arranged at a position in the virtual space corresponding to the position of the camera 150 is generated. That is, a virtual space image that can be seen from the virtual camera VCM for shooting as shown in FIGS. 4 (A) and 4 (B) is generated. Then, as shown in steps S6 and S7, the image of the subject is extracted by obtaining the difference image between the first image and the second image, and a composite image in which the image of the subject is combined with the virtual space image is generated. For example, FIG. 10A is an example of the first image IM1. In the first image IM1, an image of the player PL, which is the subject, and an image of the background BG such as a ceiling, a wall, and a pillar are displayed. FIG. 10B is an example of the second image IM2. In the second image IM2, the image of the background BG is displayed, but the image of the player PL, which is the subject, is not displayed. By obtaining such a difference image between the first image IM1 and the second image IM2, the image of the subject can be extracted by the background subtraction method. As a result, as shown in FIGS. 5A and 5B, it becomes possible to generate a composite image in which the image of the subject, which is a real space image, is combined with the virtual space image, which is a game image.

For example, in a VR game using HMD200 or the like, there is a problem that the fun is not transmitted to the player unless the player actually experiences it. Also, the gallery does not know what the player is doing in the VR game. Therefore, there is a problem that the player and the gallery cannot share the experience of the VR game.

In this case, if the virtual space image of the VR game and the real space image of the player who is the subject are combined and the image that the player seems to enter the virtual space and play is displayed in the gallery, the gallery will be VR. Since it is possible to grasp what kind of play the player is playing in the game, it is possible to share the experience of the VR game. However, if such image composition of virtual space image and real space image is performed by chroma key composition using blue background or green background, it is necessary to provide large-scale shooting equipment to realize blue background or green background. There is a problem that it does not become. For example, in a game facility where a game housing 30 is installed, it is not realistic to provide such a large-scale shooting equipment, and the decoration inside the store is also limited. Further, in chroma key composition, the blue or green part of the subject is regarded as the background, and there is a problem that the color is lost. In addition, in order for the player to appear as if he / she is playing in the virtual space, it is necessary that the virtual space image and the real space image are consistent. At this time, if the quality of the image composition of the virtual space image and the real space image is low, such as when an extra object such as a person or an object other than the player is included in the real space image, the virtual space image and the real space image may be displayed. It will be inconsistent and you will not get the results you want.

Therefore, in the present embodiment, the first image in which the background and the subject are photographed by the camera 150 in the real space and the second image in which the background is photographed are acquired. Further, a virtual camera for shooting is arranged at a position in the virtual space corresponding to the position of the camera 150 in the real space, and a virtual space image that can be seen from the virtual camera for shooting is generated. Then, by obtaining the difference image between the first image and the second image, the image of the subject is extracted by the background subtraction method, and a composite image in which the image of the subject is combined with the virtual space image is generated. In this way, the virtual space image and the real space image of the subject can be image-combined without providing a large-scale shooting equipment. Therefore, it is possible to provide an image generation system capable of combining the image of the subject taken by the camera 150 and the virtual space image with a simple system with high quality.

Further, in the present embodiment, the depth image of the background and the subject taken by the camera 150 is acquired, and the image of the subject is extracted based on the difference image between the first image and the second image and the depth image. By extracting the image of the subject using not only the difference image by the background subtraction method but also the depth image in this way, it becomes possible to realize a higher quality image composition of the virtual space image and the image of the subject. For example, if an image of a subject is extracted only by the background subtraction method, there is a problem that pixels having the same color as the background in the image of the subject are chipped due to color cast. In addition, the camera 150 is installed inside the booth, and in situations where general passers-by come and go through the passage behind the player when viewed from the camera 150, only the background subtraction method using color subtraction images is used. Then, there is also a problem that proper extraction processing cannot be realized. In this regard, the above problems can be solved by performing image composition using a depth image in addition to the difference image obtained by the background subtraction method.

More specifically, in the present embodiment, a difference mask image is generated based on the difference image, and a depth mask image for identifying a pixel whose depth value is in a given depth range is generated based on the depth image. Then, a subject mask image that identifies the subject is generated based on the difference mask image and the depth mask image, and the subject image is extracted based on the subject mask image and the first image. By using such a difference mask image, the background area and the subject area can be easily identified by using the difference mask image. Further, by using a depth mask image that identifies a pixel whose depth value is in a given depth range, a subject located in the depth range can be easily identified. Therefore, by using such a difference mask image and a depth mask image, it is possible to extract the image of the subject with high quality from the first image in which the background and the subject are reflected.

FIG. 10C is an example of the difference mask image MSDF. The difference mask image MSDF can be generated by performing a binarization process of the difference image between the first image IM1 of FIG. 10A and the second image IM2 of FIG. 10B. For example, the difference mask image MSDF may be generated based on the difference image and the depth image. For example, a difference mask image based on a mask image generated by performing a binarization process of a difference image and a depth mask image MSDP (depth mask image after correction processing or before correction processing) generated from the depth image. Generate MSDF. In this difference mask image MSDF, the area of the player PL (subject) is a pixel with a white pixel value, and the area of the background BG is a pixel with a black pixel value. However, as shown in FIG. 10C, the background and the portion covered with the color are lacking in color. When the pixel value range is 0 to 255, the white pixel value is the maximum pixel value "255" and the black pixel value is the minimum pixel value "0".

FIG. 11A is an example of a depth mask image MSDP. This depth mask image MSDP is, for example, as shown in FIG. 13A, a mask image that identifies pixels whose depth value is in a given depth range RA. The depth range RA is a range in which the depth value is equal to or greater than the depth value ZN on the near side and equal to or less than the depth value ZF on the far side. The subject SB located in this depth range RA is shown in FIG. 11 (A). Depth mask image White pixels in MSDP. By using the depth mask image MSDP that identifies the pixels whose depth value is in the depth range RA in this way, it becomes possible to set an object in the depth range RA as an extraction target. Then, when an object that should not be a subject exists or passes on the front side or the back side of the depth range RA, the object can be excluded from the extraction target by the depth mask image MSDP.

Then, in this embodiment, the subject mask image MSSB shown in FIG. 11 (B) is generated based on the difference mask image MSDF of FIG. 10 (C) and the depth mask image MSDP of FIG. 11 (A). Subject mask image MSSB is a mask image for identifying a subject. For example, the subject mask image MSSB is a mask image in which white pixels are formed in a subject area and black pixels are formed in a region other than the subject. As an example, the subject mask image MSSB of FIG. 11 (B) can be generated by taking the OR (logical sum) of the difference mask image MSDF of FIG. 10 (C) and the depth mask image MSDP of FIG. 11 (A). .. Specifically, the subject mask image MSSB is generated by taking an OR of the difference mask image MSDF and the depth mask image MSDP after correction processing such as morphology filter processing described later. More specifically, the difference mask is obtained by taking an AND (logical product) between the mask image generated by performing the binarization process of the difference image and the depth mask image MSDP after the correction process or before the correction process. Generate image MSDF. Then, the subject mask image MSSB is generated by taking an OR of the generated difference mask image MSDF and the depth mask image MSDP after the correction process. For example, in the subject mask image MSSB, the pixels that are white in the difference mask image MSDF or the pixels that are white in the depth mask image MSDP are set as white pixels. Further, in the subject mask image MSSB, the pixels that become black in both the difference mask image MSDF and the depth mask image MSDP are set to black pixels. Then, an image of the subject is extracted as shown in FIG. 11 (C) based on the subject mask image MSSB of FIG. 11 (B) and the first image IM1 of FIG. 10 (A). For example, an image of a subject can be extracted from the first image IM1 by cutting out a region of a pixel group in which the subject mask image MSSB is a white pixel. For example, as shown in FIG. 10C, the difference mask image MSDF has a problem that pixels of the same color as the background in the subject image are chipped due to color cast. In this regard, in the depth mask image MSDP of FIG. 11A, the black pixels lacking due to color cast in the difference mask image MSDF become white pixels. Therefore, by generating the subject mask image MSSB based on the difference mask image MSDF and the depth mask image MSDP, it is possible to solve the problem that pixels of the same color as the background are missing due to color cast, and an appropriate extraction process of the subject can be performed. It will be possible.

If there is a non-extraction target object such as an operator or a passerby who is not planned to be extracted on the front side of the subject and it overlaps with the subject, in the depth mask image MSDP, the non-extraction target object is used. The pixels in the area become black, and there is a risk that the subject behind the non-extracted object cannot be properly extracted. If the depth threshold value is set in the difference mask image MSDF as well, the pixels in the region of the non-extracted object will be black. In this way, the system detects that the subject cannot be extracted correctly due to the presence of the non-extracted object, and instead of extracting the player who is the subject, displays a character image on behalf of the player or extracts the player. , You may switch to the extraction of other players playing with you. For example, the position of the HMD is detected before the start of game play. Then, during game play, if the HMD is hidden by the presence of a non-extractable object on the front side of the HMD and the pixels in the HMD area become black, it is determined that an error has occurred and the character in the game space. If you switch to the display of the image of the player, switch to the image of another player, or if you are using multiple cameras, the cameras that shoot the player are installed at different positions such as high places and rear. Switch to another camera. Further, for example, the extraction target area where the player should be is estimated from the position information such as the HMD at all times during play. When the number of pixels in the area becomes a certain percentage or less, it is determined that the non-extracted object exists on the front side of the HMD, and the switching as described above is performed. On the contrary, if the extraction target area is too large, it is determined that false detection of the floor or wall surface has occurred, extraction / synthesis is interrupted, and the character display in the game space is switched or switched to another camera. To do so. Further, similarly to the method of setting the auxiliary points described with reference to FIG. 8 (B), in addition to the position of the HMD, some auxiliary points are set at the lower part of the housing, etc. The switching process as described above may be performed by making a judgment using the pixels of. For example, in addition to the housing mask images described with reference to FIGS. 6 (A) and 6 (B), tracking information from a tracking device such as an HMD and information on auxiliary points can be used together for more accurate extraction. The processing can be realized.

Then, in the present embodiment, the image of the subject extracted as described above is combined with the virtual space image. For example, FIG. 12A is an example of a composite image in which the image of the player PL, which is the subject, is combined with the virtual space image which is the game image. As a result, it is possible to generate a composite image that looks as if a real-space player PL has appeared in the virtual space. Then, as shown in FIG. 12B, by displaying the composite image of FIG. 12A on the gallery display device 210, the gallery can observe the behavior of the player PL playing in the virtual space. Become. For example, in a VR game facility, a composite image as shown in FIG. 12A is displayed in the waiting time of the VR game for a gallery waiting for the VR game to be played. As a result, the player PL and the gallery can share the experience of the VR game, and the fun of the VR game can be improved.

Further, in the present embodiment, the depth mask image is corrected, and the subject mask image is generated based on the corrected depth mask image and the difference mask image. That is, the depth mask image MSDP of FIG. 11 (A) is corrected, and the subject mask of FIG. 11 (B) is based on the corrected depth mask image MSDP and the difference mask image MSDF of FIG. 10 (C). Generate an image MSSB. For example, the difference mask image MSDF has a problem of color loss, and the depth mask image MSDP is used to solve this problem of color loss. On the other hand, the difference mask image MSDF has an advantage that the difference can be clearly taken even for the edge part, but the depth mask image MSDP has problems such as the edge part flickering due to noise and fine noise being superimposed. is there. In this regard, if the depth mask image MSDP is corrected, it is possible to prevent flicker at the edge portion, remove fine noise, and the like, and it is possible to generate a high-quality composite image.

For example, in the present embodiment, a difference depth mask image of a first depth image in which the background and the subject are photographed by the camera 150 and a second depth image in which the background is photographed by the camera 150 is generated, and based on the difference depth mask image, the difference depth mask image is generated. Generates a depth mask image after correction processing. For example, the first depth image IMDP1 of FIG. 13B is a depth image obtained by photographing the background BG and the subject SB by the camera 150 (depth camera 154), and the second depth image IMDP2 is a depth image obtained by photographing the background BG by the camera 150. It is a depth image. A difference depth mask image MSDPDF (background subtraction depth mask image) is generated from the difference image between the first depth image IMDP1 and the second depth image IMDP2. For example, the difference depth mask image MSDPDF can be generated by performing a binarization process of the difference image. For example, processing such as setting a pixel whose depth value difference value is equal to or greater than a predetermined value to a white pixel is performed. FIG. 14A is an example of the difference depth mask image MSDPDF. In a normal depth mask image that does not take a difference, there is a problem that white pixels are also formed on the ceiling and floor as shown in FIG. 14 (C). By using the difference depth mask image MSDPDF, such a problem occurs. It will be possible to solve the problem and generate high-quality composite images.

Further, in the present embodiment, a depth mask image after the correction process is generated by performing at least one of the morphology filter process and the time series filter process. For example, FIG. 14B is a difference depth mask image MSDPDF after the difference depth mask image MSDPDF of FIG. 14A is corrected by a morphology filter process or a time series filter process. The morphology filtering process is a filtering process that expands and contracts, and by performing such a morphology filtering process, noise of a fine size can be removed. In the time-series filtering process, for example, pixels determined to be white pixels continuously for a predetermined number of frames or more (pixels having a depth value difference value of a predetermined value or more) are determined as white pixels effective as a mask. .. By performing such a time-series filter processing, it is possible to suppress the occurrence of fine noise flicker. Then, in the present embodiment, the difference depth mask image MSDPDF subjected to such morphology filtering and time series filtering is used to generate the corrected depth mask image MSDP as shown in FIG. 14 (D). .. For example, the depth mask image MSDP shown in FIG. 14 (D) is generated by ANDing the difference depth mask image MSDPDF of FIG. 14 (B) and the normal depth mask image of FIG. 14 (C) (logical product). .. Then, using this depth mask image MSDP, the subject mask image MSSB described in FIG. 11B is generated, and the image of the subject is extracted as shown in FIG. 11C.

Further, in the present embodiment, the depth mask image after the correction process is generated by performing the process of setting the pixel value of the pixel for which the depth value could not be acquired in the depth image based on the difference image. That is, a correction process is performed in which the pixel value of the blank pixel, which is the pixel for which the depth value could not be acquired, is filled with the pixel value of the difference image. FIG. 15 (A) is an example of the extracted image of the subject when the correction process for filling the blank pixels is performed, and FIG. 15 (B) shows the subject when the correction process for filling the blank pixels is not performed. This is an example of the extracted image of. As shown in FIG. 15B, if the correction process for filling the blank pixels is not performed, there is a problem that the image of the subject is chipped, for example, near the edge of the hand.

In order to prevent the occurrence of such a problem, in the present embodiment, as shown in FIG. 16A, a blank mask image for identifying the blank pixel for which the depth value could not be acquired is generated. That is, in the acquisition of the depth value by the stereo camera, there are pixels that can be seen by one camera of the stereo camera but cannot be seen by the other camera, and an appropriate depth value cannot be acquired in such pixels. The result is that it could not be obtained. A blank mask image as shown in FIG. 16A can be generated by setting, for example, white pixels as the pixels for which the result that the depth value cannot be obtained is returned. FIG. 16B is a difference mask image generated by obtaining the difference between the first image and the second image. FIG. 16C is a mask image obtained by ANDing the blank mask image of FIG. 16A and the difference mask image of FIG. 16B. The mask image of FIG. 16 (C) is a pixel determined to be a blank pixel whose depth value cannot be obtained in the blank mask image of FIG. 16 (A), and is a subject in the difference mask image of FIG. 16 (B). The pixel determined to be is set to the white pixel. A process (filling process) of setting the pixel value of the pixel for which the depth value could not be obtained based on the difference mask image which is the difference image is performed. Then, a subject mask image is generated and the image of the subject is extracted so that the white pixels in FIG. 16C are set as the pixels of the subject. By doing so, it is possible to prevent the problem that the image is chipped due to the failure to acquire the depth value as shown in FIG. 15 (B), and the extracted image of an appropriate subject as shown in FIG. 15 (A). Will be able to generate.

Further, in the present embodiment, the area size of the pixel group whose depth value is in the depth range is obtained, and the depth mask image after the correction processing is generated by performing the filter processing according to the area size. For example, the area size of the pixel group whose depth value is the depth range RA in FIG. 13 (A) is obtained. For example, the area size of the pixel group can be obtained by counting the number of pixels of the pixel group (adjacent pixel group) whose depth value is the depth range RA. Then, a filter process is performed in which the pixel group having the largest area size or the pixel group having the area size of a predetermined size or more is set as the pixel group constituting the subject. For example, FIG. 17 (A) is an example of a depth mask image before filtering by area size, and FIG. 17 (B) is an example of a depth mask image after filtering by area size. is there. In FIG. 17A, a pixel group having a small area size remains as white pixels, but in FIG. 17B, a filter process for removing a pixel group having a small area size is performed. Then, in FIG. 17B, among the pixel groups whose depth value is the depth range RA, the pixel group having the largest area size is determined as the pixel group of the subject and set as white pixels. By doing so, it is possible to remove the pixel group having a small area size generated due to noise or the like and extract only the pixel group corresponding to the subject, so that a high-quality composite image can be generated. Become.

Further, in the present embodiment, a second depth range is set based on the depth value in the subject area determined to be the subject area, and an image that identifies pixels whose depth value is the second depth range is used as a depth mask image. Generate. For example, in the depth mask image of FIG. 18A, since the region of white pixels is determined to be the subject region, the average value ZAV of the depth values of this subject region is obtained. Then, based on this average value ZAV, the second depth range RA2 as shown in FIG. 18B is set. The second depth range RA2 is a range in which the depth value is equal to or greater than the near-side depth value ZN2 and equal to or less than the far-side depth value ZF2. The second depth range RA2 is narrower than the depth range RA of FIG. 13, and the range of the depth value of the subject SB is specified more strictly.

For example, in a free loam (open world) content in which a player freely roams and moves around the world of virtual space, it is difficult to set the depth range RA of FIG. 13 (A). For example, if a distance range (RA) that covers the entire range in which the player may move is specified, there is a risk that unnecessary objects may be erroneously extracted.

In this regard, in FIG. 18A, the second depth range RA2 is set based on the depth value (ZAV) in the subject region determined to be the subject region. Therefore, even when the player moves, the second depth range RA2 can be set so as to follow the movement of the player. For example, the depth range is gradually narrowed from the depth range to the second depth range RA2 so as not to erroneously extract an unnecessary object as described above. Therefore, it is possible to generate a strict depth mask image that reflects the movement of the player who is the subject, and it is possible to generate a high-quality composite image.

Further, the distance range from the camera may be obtained based on the tracking information such as the HMD so as to limit the player area by using the tracking information, and the depth range may be set based on the distance.

4.3 Various processing examples Next, various processing examples of the present embodiment will be described. For example, in the present embodiment, as shown in FIGS. 5A and 5B, an image of the housing 30 on which the player PL is boarded is extracted in the real space, and the image of the player PL and the housing 30 are combined with the virtual space image. Is generating a composite image of the above images. As a result, the real space image in which the player PL is mounted on the housing 30 can be combined with the virtual space image. In this case, in the present embodiment, as described with reference to FIGS. 10 (A) and 10 (B), the image of the subject is extracted by the background subtraction method for obtaining the difference image between the first image IM1 and the second image IM2. .. Since the housing 30 is reflected on both the first image IM1 and the second image IM2, the housing 30 is determined to be the background by the normal background subtraction method and disappears without being extracted. Further, when the housing 30 is a movable housing, the housing 30 moves during game play, so that the housing 30 may or may not disappear.

Therefore, in the present embodiment, as described with reference to FIGS. 6 (A) and 6 (B), the image of the housing 30 is extracted by using the housing mask image that specifies the extraction range of the image of the housing 30. ing. A method using this housing mask image will be described in detail.

FIG. 19A is an example of a housing mask image set by the operator manually specifying the range of the housing 30. The range of the housing 30 may be specified with a roughness that traces the approximate shape of the housing 30. FIG. 19B is an example of a depth mask image of the housing 30. By ANDing the housing mask image for specifying the range of FIG. 19 (A) and the depth mask image of FIG. 19 (B), the housing region as shown in FIG. 19 (C) is shown. A housing mask image is generated. Actually, the depth mask image of FIG. 19 (B) is subjected to correction processing such as filling holes and edge smoothing as described above, and the depth mask image after the correction processing and the range designation of FIG. 19 (A) are specified. The AND of the housing mask image for In FIG. 19 (A), only the approximate shape of the housing 30 was specified, but by taking an AND with the depth mask image of FIG. 19 (B), as shown in FIG. 19 (C), the housing 30 It is possible to generate a housing mask image that accurately reflects the shape. Then, by taking OR of the housing mask image of FIG. 19 (C) and the difference mask image (color background subtraction), an image as shown in FIG. 20 (A) is generated, and the housing of FIG. 19 (C) is generated. By taking OR of the mask image and the difference depth mask image (depth background subtraction), an image as shown in FIG. 20B is generated. Then, by taking the OR of the image of FIG. 20 (A) and the image of FIG. 20 (B), the subject mask image shown in FIG. 20 (C) is generated. As a result, as shown in FIG. 20 (D), the housing 30 and the player PL can be appropriately extracted. For example, FIGS. 21 (A) to 21 (D) are explanatory views of a method that does not use a housing mask image. FIG. 21 (A) is a difference mask image (color background subtraction), and FIG. 21 (B) is a difference depth mask image (depth background difference). In FIG. 21B, the difference to the extent that the player PL rides on the housing 30 is equal to or less than the difference threshold. 21 (C) is a subject mask image generated by taking an OR of the image of FIG. 21 (A) and the image of FIG. 21 (B), and is shown in FIG. 21 (D) depending on the subject mask image. As described above, only the player PL can be extracted, and the housing 30 cannot be extracted.

According to the method using the housing mask image described above, the operator only needs to roughly specify the outline shape of the housing 30 as shown in FIG. 19A, which facilitates the work of specifying the range of the operator. .. Further, by moving the housing 30, when the area of a part of the pixels in which the housing 30 is reflected becomes the background when the housing 30 is stationary, the area of the pixels that became the background can be appropriately determined as the background. It is possible to realize an appropriate extraction process of the housing 30 which is a movable housing.

FIG. 22 is an explanatory diagram of the method for setting the extraction range described with reference to FIGS. 7 to 8 (B). In FIG. 22, a player PL equipped with the HMD200 is moving on a thin plate-shaped bridge installed in a game facility. Then, in this game, in the play area within the angle of view of the camera 150, the operator OP assists the player PL who cannot see the outside by wearing the HMD200 in the immediate vicinity of the player PL. Since it is not possible to determine whether the image captured by the camera 150 is the player PL or the operator OP only by image processing, there arises a problem that an erroneous target other than the player PL is extracted.

In order to suppress the occurrence of such a problem, as described with reference to FIGS. 7 to 8 (B), the extraction process is performed by setting the extraction range only in the region where the player PL is considered to exist. That is, by using the tracking information from a tracking device such as the HMD200, the range in which the player PL exists can be obtained. Only the position of the head of the player PL can be detected only by the tracking information of the HMD200, but by using the tracking information from a plurality of tracking devices attached to the hands, feet, etc., various postures and movements of the player PL as the subject can be obtained. It will be possible to set the extraction range to an appropriate range according to the above.

Further, in the present embodiment, the case where one camera 150 is used has been described, but a plurality of cameras may be provided and the subject extraction process may be performed using these plurality of cameras. For example, a subject extraction process is performed using a plurality of color images and a plurality of depth images from a plurality of cameras.

In the case of a multiplayer game, a plurality of composite images (composite images of a real space image and a virtual space image) generated for a plurality of players playing the game can be sequentially switched and displayed, or a plurality of composite images can be displayed. Simultaneous display may be performed on the display device.

The depth value measurement method is not limited to the method using the stereo camera described in the present embodiment, and is limited to the ToF (Time of Flight) method for measuring the round-trip time of light and a predetermined pattern such as infrared rays. Can be adopted by various methods such as a structured optical method in which the image is projected and calculated from the deformation of the pattern. For example, in the case of an HMD of the type that does not have an infrared light receiving element for tracking, use the ToF method that measures the depth value by calculating the distance from the time from the projection of infrared rays to the return. Can be done. When measuring the distance by the intensity of the reflected light, an HMD that does not have an infrared light receiving portion may be used.

Further, in the present embodiment, the virtual camera for shooting is arranged at the position of the virtual space corresponding to the position of the camera 150 in the real space. Therefore, it is desired to accurately measure the camera distance between the camera 150 and the subject (player, housing). Therefore, for example, a box-shaped marker object such as an AR marker is prepared and placed near the subject. Then, by photographing this marker object with the camera 150, the camera distance is measured by the same method as in the case of the AR marker. By doing so, it is possible to realize the arrangement of the virtual camera for shooting that accurately reflects the distance relationship between the camera 150 and the subject in the real space.

Further, this embodiment can be applied to various games and events. For example, in a rhythm game or a dance game, the state of a player playing or dancing in a virtual space square may be generated as a composite image of a virtual space image and a real space image, and distributed as a replay image, for example. .. For example, in a virtual space, a large number of characters that serve as a gallery are arranged to excite the player's performance and dance. In this way, the player can later enjoy playing and dancing in a virtual space, which is a different space.

Further, in a ride attraction for kids, the state of the child riding in the ride housing may be generated as a composite image (composite video), printed on a photograph and taken home, or distributed. According to this embodiment, it is possible to create an image or video of a desired world or situation without arranging shooting equipment such as a blue back or a green background at an attraction.

Further, at the handshake event and the commemorative photo session, a composite image taken together with the character in the virtual space may be provided. For example, allow players to meet cartoon characters and game characters. In this way, it is possible to provide a composite image showing the actual contact with a character that does not exist in the real space. You may also generate a composite image that looks like you are eating with a character in a virtual cafe.

Further, in FIGS. 5A and 5B, an image of the player PL with the HMD200 attached is displayed, but an image of a headgear instead of the HMD200 may be combined. For example, the player PL's own image is combined as an image of a headgear, or an image of a headgear such as a cosplay is combined. Alternatively, the player PL may generate a composite image that appears to be wearing a weapon, item, or wearing a costume. Alternatively, the image of the housing 30 may be subjected to image composition processing so that it looks like a more realistic image.

Further, in the cooperative play game, the first player who is viewing the composite image (composite video) of the virtual space image and the real space image and the second player who performs the game operation or moves the body are divided into the second player. The first player may give hints about the operation and operation of the player to the second player while viewing the composite image.

Although the present embodiment has been described in detail as described above, those skilled in the art will easily understand that many modifications that do not substantially deviate from the novel matters and effects of the present disclosure are possible. Therefore, all such variations are included in the scope of the present disclosure. For example, a term (player, etc.) described with a different term (subject, etc.) in a broader sense or synonymously at least once in the specification or drawing may be replaced with the different term in any part of the specification or drawing. it can. In addition, image generation system configuration, housing configuration, camera configuration, image acquisition processing, virtual space image generation processing, subject and housing image extraction processing, virtual space image and real space image image composition processing, etc. However, the method, processing, and configuration equivalent to these are also included in the scope of the present disclosure without being limited to those described in the present embodiment. Moreover, this embodiment can be applied to various games. Further, the present embodiment can be applied to various image generation systems such as a commercial game device, a home game device, or a large attraction system in which a large number of players participate.

PL ... player, VCM ... virtual camera for shooting, CHM ... photographer character,
MV ... car, MVE ... enemy car, CHE ... enemy character, ST handle, HL, HR hand,
EF1, EF2 ... effect, CS ... course, MSC ... housing mask image,
TR1 to TR5 ... Tracking device, AR ... Extraction range, PAX ... Auxiliary point,
IM1 ... 1st image, IM2 ... 2nd image, MSDF ... Difference mask image,
MSDP ... Depth mask image, MSSB ... Subject mask image,
IMDP1 ... 1st depth image, IMDP2 ... 2nd depth image,
MSDPDF ... differential depth mask image, MSC ... housing mask image,
BG ... background, SB ... subject, RA ... depth range, RA2 ... second depth range,
ZN, ZN2, ZF, ZF2 ... Depth value, ZAV ... Mean value of depth value,
30 ... housing, 32 ... bottom, 33 ... cover, 40 ... moving part,
50 ... steering wheel, 52 ... accelerator pedal, 54 ... brake pedal,
60 ... Ride part, 62 ... Seat, 80 ... Blower,
100 ... Processing unit, 102 ... Acquisition unit, 104 ... Virtual space setting unit, 106 ... Game processing unit,
107 ... Mobile processing unit, 108 ... Housing control unit, 110 ... Virtual camera control unit,
120 ... image generation unit, 122 ... image composition unit, 130 ... sound generation unit,
150 ... camera, 152 ... color camera, 154 ... depth camera, 160 ... operation unit,
170 ... storage unit, 172 ... virtual space information storage unit, 178 ... drawing buffer,
180 ... Information storage medium, 192 ... Sound output section, 194 ... I / F section,
195 ... Portable information storage medium, 196 ... Communication unit, 200 ... HMD (head-mounted display device), 201-203 ... Light receiving element, 206 ... Tracking device, 208 ... Display unit,
210 ... Gallery display device, 250, 260 ... Tracking device,
251 to 254 ... light receiving element, 261 to 264 ... light receiving element,
280, 284 ... Base station, 281, 282, 285, 286 ... Light emitting element

Claims (16)

An acquisition unit that acquires a first image in which a background and a subject are photographed by a camera arranged in a real space and a second image in which the background is photographed by the camera.
An image generation unit that generates a virtual space image that can be seen from a virtual camera for shooting arranged at a position in the virtual space corresponding to the position of the camera.
An image compositing unit that extracts the image of the subject by obtaining the difference image between the first image and the second image and generates a composite image in which the image of the subject is combined with the virtual space image.
An image generation system characterized by including. In claim 1,
The image synthesizing unit
An image generation system characterized in that an image of a housing on which the subject is mounted is extracted in the real space, and the composite image in which the image of the subject and the image of the housing are combined with the virtual space image is generated. .. In claim 2,
The image synthesizing unit
An image generation system characterized in that an image of the housing is extracted by using a housing mask image that specifies an extraction range of the image of the housing. In claim 1,
The image synthesizing unit
An image generation system characterized in that an image extraction range of the subject is set based on tracking information from at least one tracking device worn by the subject, and the image of the subject is extracted. In claim 4,
The image synthesizing unit
It is characterized in that the extraction range of the image of the subject is set based on the position of the tracking device and the position of an auxiliary point set at a position shifted by a predetermined distance from the position of the tracking device. Image generation system. In any of claims 1 to 5,
The image generation unit
As a virtual space image for the player displayed on the player which is the subject, the image of the virtual camera for shooting and the image of the photographer character are displayed at positions corresponding to the positions of the virtual camera for shooting in the virtual space. An image generation system characterized in generating a virtual space image in which at least one is displayed. In any of claims 1 to 6,
A head-mounted display device that is worn by the player who is the subject and displays a virtual space image for the player that can be seen from the virtual camera for the player in the virtual space.
A gallery display device in which the composite image is displayed as a gallery image, and
An image generation system characterized by including. In any of claims 1 to 7,
The acquisition unit
A depth image of the background and the subject taken by the camera is acquired.
The image synthesizing unit
An image generation system characterized in that an image of the subject is extracted based on the difference image and the depth image. In claim 8.
The image synthesizing unit
A difference mask image is generated based on the difference image, a depth mask image for identifying a pixel whose depth value is in a given depth range is generated based on the depth image, and the difference mask image and the depth mask image are generated. An image generation system characterized in that a subject mask image for identifying the subject is generated based on the above, and an image of the subject is extracted based on the subject mask image and the first image. In claim 9.
The image synthesizing unit
An image generation system characterized by performing correction processing of the depth mask image and generating the subject mask image based on the depth mask image and the difference mask image after the correction processing. In claim 10,
The image synthesizing unit
A difference depth mask image is generated between the first depth image of the background and the subject taken by the camera and the second depth image of the background taken by the camera, and the correction process is performed based on the difference depth mask image. An image generation system characterized by generating the depth mask image later. In claim 10 or 11, the image synthesizing unit
An image generation system characterized in that the depth mask image after the correction process is generated by performing at least one of a morphology filter process and a time series filter process. In any of claims 10 to 12,
The image synthesizing unit
Image generation characterized in that the depth mask image after the correction process is generated by performing a process of setting the pixel values of the pixels for which the depth value could not be acquired in the depth image based on the difference image. system. In any of claims 10 to 13,
The image synthesizing unit
An image generation system characterized in that a depth mask image after the correction processing is generated by obtaining an area size of a pixel group in which the depth value is in the depth range and performing a filter process according to the area size. In any of claims 9 to 14,
The image synthesizing unit
A second depth range is set based on the depth value in the subject area determined to be the subject area, and an image that identifies pixels whose depth value is in the second depth range is generated as the depth mask image. An image generation system characterized by that. An acquisition unit that acquires a first image in which a background and a subject are photographed by a camera arranged in a real space and a second image in which the background is photographed by the camera.
An image generation unit that generates a virtual space image that can be seen from a virtual camera for shooting arranged at a position in the virtual space corresponding to the position of the camera.
As an image synthesizing unit, an image of the subject is extracted by obtaining a difference image between the first image and the second image, and a composite image in which the image of the subject is combined with the virtual space image is generated.
A program characterized by making a computer work.

Images