JP2018173739A - Program and image generation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program, an image generation system, etc., capable of actualizing extended reality as if a virtual space were present in a region of an opening part by simple processing.SOLUTION: An image generation system includes: an image acquisition unit which acquires a picked-up image; a recognition processing unit which performs recognition processing on an object of recognition taken in the picked-up image; a virtual space setting unit which performs processing for arranging and setting, in a virtual space, at least one three-dimensional object and a mask object arranged having predetermined relationship with the three-dimensional object; and an image generation unit which generates an image such that a virtual space image composed of an image of the three-dimensional object is displayed in an image region of an opening part set correspondingly to the place of the object of recognition and the picked-up image is displayed in an image region of a periphery of the opening part with the three-dimensional object masked with the mask object.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a program, an image generation system, and the like.

  In recent years, a technique called Augmented Reality (AR) has been spotlighted as a technique for additionally presenting information to the real world using a computer. This augmented reality (AR) is also called augmented reality, and a virtual object as additional information is synthesized and displayed on a part of the actual environment. As conventional technology of such augmented reality, for example, there are technologies disclosed in Patent Documents 1 and 2.

  In the prior art of Patent Document 1, a real-world state in which an AR marker on which a predetermined pattern is drawn is arranged is photographed by an imaging unit. Then, the position of the marker in the image captured by the imaging unit is estimated and the AR marker image is recognized, and a virtual object corresponding to the AR marker pattern is superimposed and displayed at the estimated position.

  Patent Document 2 discloses an image in which the inside of a building venue can be seen as a peephole image by shooting the inside of a building such as a live venue or an exhibition with a camera and mapping the shot video to a peephole object. A generation system is disclosed. In the prior art of Patent Document 2, an image that can be seen from a virtual peephole is generated by analyzing the size, orientation, and angle of the AR marker.

JP 2010-170316 A JP 2014-106556 A

  However, in the prior art of Patent Document 2, the peephole image mapped to the peephole object is not a virtual space image composed of three-dimensional object images, but a real space image taken by a camera. And the technique which can display a virtual space image in the area | region of recognition objects, such as AR marker, by a simple process was not proposed by the prior art.

  According to some aspects of the present invention, it is possible to provide a program, an image generation system, and the like that can realize augmented reality with simple processing as if a virtual space exists in the region of the opening.

  One aspect of the present invention is directed to an image acquisition unit that acquires a captured image, a recognition processing unit that performs a recognition process on a recognition target object reflected in the captured image, at least one three-dimensional object, and the three-dimensional object. A virtual space setting unit that performs a process of setting a mask object arranged with a predetermined relationship in a virtual space, and an image area of an opening set corresponding to the location of the recognition target object, A virtual space image composed of an image of a three-dimensional object is displayed, and an image in which the captured image is displayed by masking the three-dimensional object with the mask object is displayed in an image area around the opening. The present invention relates to an image generation system including an image generation unit for generation. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to one aspect of the present invention, a captured image is acquired, recognition processing of a recognition target object reflected in the captured image is performed, and at least one three-dimensional object and a mask object are arranged and set in the virtual space. In this case, the mask object is arranged with a predetermined relationship (positional relationship, directional relationship, etc.) with respect to the three-dimensional object. Then, a virtual space image is displayed in the image area of the opening set corresponding to the location of the recognition target object, and an image in which the captured image is displayed in the image area around the opening is generated. In this case, the captured image is displayed in the image area around the opening by masking the three-dimensional object with the mask object. In this way, an image as if the virtual space exists in the area of the opening can be generated by a simple process of arranging and setting the mask object together with the three-dimensional object in the virtual space. Accordingly, it is possible to provide an image system, a program, and the like that can realize augmented reality with simple processing as if a virtual space exists in the area of the opening.

  In one embodiment of the present invention, the virtual space image is an image that can be seen from a virtual camera in the virtual space, and the predetermined relationship is behind the predetermined surface of the mask object as viewed from the virtual camera. The relationship that the three-dimensional object is located may be used.

  In this way, it is possible to generate an augmented reality image as if a three-dimensional object exists behind the predetermined surface of the mask object as viewed from the virtual camera. An effect may be performed in which a part or all of some of the three-dimensional objects are located on the near side of the predetermined surface when viewed from the virtual camera. Further, the predetermined surface can be a surface on the near side (for example, the foremost side) as viewed from the virtual camera among the surfaces constituting the mask object, for example.

  In the aspect of the invention, the virtual space setting unit may set the mask object having the first surface including the opening in the virtual space.

  In this way, the virtual space image is displayed in the image area of the opening and the captured image is displayed in the image area around the opening by masking the three-dimensional object with the first surface including the opening. Augmented reality images can be generated.

  In the aspect of the invention, the virtual space setting unit may place and set the mask object having the first surface and at least one surface intersecting the first surface in the virtual space. Also good.

  In this way, if the mask object is constituted by at least one surface intersecting the first surface in addition to the first surface, the mask object is used to mask the three-dimensional object more reliably, and the periphery of the opening. The captured image can be displayed in the image area.

  In the aspect of the invention, the predetermined relationship may be a relationship in which the three-dimensional object is located in a region surrounded by the first surface and the at least one surface.

  In this way, it becomes possible to surround the three-dimensional object by the first surface of the mask object and at least one surface intersecting the first surface, and the three-dimensional object can be more reliably used by using the mask object. The object can be masked. An effect may be performed in which a part or all of a part of the three-dimensional objects out of a plurality of three-dimensional objects goes out of an area surrounded by the first surface and at least one surface. .

  In the aspect of the invention, the virtual space setting unit may set the three-dimensional object in the virtual space so that the three-dimensional object is positioned on the back side of the opening.

  In this way, it becomes possible to realize augmented reality that looks as if a virtual space composed of three-dimensional objects exists behind the opening.

  In the aspect of the invention, the virtual space setting unit may set the three-dimensional object in the virtual space such that the three-dimensional object straddles the opening.

  In this way, it becomes possible to realize an augmented reality that looks as if a three-dimensional object has popped out of the virtual space behind the opening.

  In the aspect of the invention, the image generation unit may generate the virtual space image by performing drawing processing according to positional relationship information between the imaging unit that captures the captured image and the recognition object. Good.

  In this way, it is possible to generate an image in which the virtual space image changes according to the positional relationship between the imaging unit and the recognition object and display it in the image area of the opening.

  In one aspect of the present invention, a virtual camera setting unit that sets a virtual camera at a position corresponding to the imaging unit in the virtual space (a computer functions as a virtual camera setting unit), and the image generation unit An image visible from a virtual camera may be generated as the virtual space image.

  In this way, an image that can be seen by the virtual camera set at a position corresponding to the imaging unit can be generated as a virtual space image and displayed in the image area of the opening.

  In one aspect of the present invention, the image generation unit performs a process of changing the visibility of the virtual space image according to positional relationship information between the imaging unit that captures the captured image and the recognition target object. Also good.

  In this way, for example, when the positional relationship between the imaging unit and the recognition target object is appropriate, the visibility of the virtual space image is increased, and otherwise the rendering process is reduced. become able to.

  Moreover, in one aspect of the present invention, a sound generation unit that performs a sound generation process that changes according to positional relationship information between an imaging unit that captures the captured image and the recognition object may be included (a computer as the sound generation unit) May work).

  In this way, a sound that changes according to the positional relationship between the imaging unit and the recognition object is generated and output to the user. For example, an effect process in which sound can be heard from within the virtual space Etc. can be realized.

  In one aspect of the present invention, information on the three-dimensional object is stored in a storage unit in association with information on the recognition target object, and the virtual space setting unit is associated with the information on the recognition target object. A three-dimensional object may be arranged and set in the virtual space.

  In this way, a virtual space image in which the three-dimensional object associated with the information on the recognition target object is arranged in the virtual space is generated, and an image in which the virtual space image is displayed in the image area of the opening is generated. become able to.

  In the aspect of the invention, the recognition processing unit may perform a process of recognizing an image displayed on the display unit of the electronic device as the recognition target object.

  According to this configuration, an image displayed on the display unit of the electronic device is recognized as a recognition target object, and a virtual image configured by a three-dimensional object image is formed in the image area of the opening corresponding to the location of the recognition target object. An image on which a spatial image is displayed can be generated.

  In the aspect of the invention, the recognition processing unit may perform a process of recognizing a toy or an article as the recognition target object.

  In this way, an image in which a toy or an article is recognized as a recognition object and a virtual space image composed of an image of a three-dimensional object is displayed in the image area of the opening corresponding to the location of the recognition object. Can be generated.

1 is a block diagram illustrating a configuration example of an image generation system according to an embodiment. Explanatory drawing about a recognition target object. Explanatory drawing of the method of this embodiment. Explanatory drawing of the method of this embodiment. Explanatory drawing of the method of this embodiment. The example of the image produced | generated by the method of this embodiment. The example of the image produced | generated by the method of this embodiment. The example of the image produced | generated by the method of this embodiment. Explanatory drawing of the method of performing a drawing process etc. according to the positional relationship information of the imaging part of a user's terminal device, and a recognition target object. Explanatory drawing of the method of performing a drawing process etc. according to the positional relationship information of the imaging part of HMD which a user wears, and a recognition target object. FIGS. 11A to 11C are explanatory diagrams of a technique for changing the visibility of the virtual space image or changing the sound in accordance with the positional relationship information between the imaging unit and the recognition target object. The figure which shows an example of recognition target object information. 13A and 13B are explanatory diagrams of a method for recognizing an image displayed on a display unit of an electronic device as a recognition target object. 14A and 14B are explanatory diagrams of a method for recognizing a toy or an article as a recognition object. Explanatory drawing of the method of the modification of this embodiment. Explanatory drawing of the method of the modification of this embodiment. The flowchart which shows the process example of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Image Generation System FIG. 1 is a block diagram illustrating a configuration example of an image generation system (an image generation device, a terminal device, and an HMD) according to the present embodiment. Note that the image generation system of the present embodiment is not limited to the configuration of FIG. 1, and various modifications such as omitting some of the components (each unit) or adding other components are possible. .

  The operation unit 160 is for a user (player) to input various operation information (input information). The operation unit 160 can be realized by an operation button, a direction instruction key, a joystick, a lever, a touch panel display, or the like.

  The imaging unit 162 (camera) captures a subject (imaging target), and is realized by an optical system including an image sensor such as a CCD or CMOS sensor, a focus lens, and the like.

  The storage unit 170 stores various types of information. The storage unit 170 functions as a work area such as the processing unit 100 or the communication unit 196. A program and data necessary for executing the program are held in the storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), HDD (Hard Disk Drive), SSD, optical disk device, or the like. The storage unit 170 includes an object information storage unit 172, a drawing buffer 174, and a recognition target object information storage unit 176.

  The information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by an optical disk (DVD, BD, CD), HDD, semiconductor memory (ROM), or the like. . The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, in the information storage medium 180, a program for causing a computer (an apparatus including an input device, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by an LCD, an organic EL display, a CRT, a touch panel display, an HMD (head mounted display), or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function can be realized by an ASIC for I / F processing or the like. The portable information storage medium 195 is for a user to save various types of information, and is a storage device that retains storage of such information even when power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

  The communication unit 196 communicates with the outside (another apparatus) via a wired or wireless network, and functions thereof are hardware such as a communication ASIC or communication processor, or communication firmware. Can be realized.

  A program (data) for causing a computer to function as each unit of this embodiment is distributed from the information storage medium of the server (host device) to the information storage medium 180 (or storage unit 170) via the network and communication unit 196. May be. Use of an information storage medium by such a server (host device) can also be included in the scope of the present invention.

  The processing unit 100 (processor) performs image acquisition processing, recognition processing, virtual space setting processing, virtual camera setting processing, image generation processing, sound generation processing, and the like based on operation information and programs from the operation unit 160. .

  Each processing (each function) of this embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the function of each unit may be realized by individual hardware, or the function of each unit may be realized by integrated hardware. For example, the processor may include hardware, and the hardware may include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor can be configured by one or a plurality of circuit devices (for example, ICs) mounted on a circuit board or one or a plurality of circuit elements (for example, resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may be an ASIC hardware circuit. The processor may include an amplifier circuit, a filter circuit, and the like that process an analog signal. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD) or an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and the processing (function) of each unit of the processing unit 100 is realized by executing the instructions by the processor. The instruction here may be an instruction set constituting a program, or an instruction for instructing an operation to the hardware circuit of the processor.

  The processing unit 100 includes an image acquisition unit 112, a recognition processing unit 114, a virtual space setting unit 116, a virtual camera setting unit 118, an image generation unit 120, and a sound generation unit 130. The image generation unit 120 includes an image composition unit 122. As described above, each process of the present embodiment executed by these units can be realized by a processor (or a processor and a memory). Various modifications such as omitting some of these components (each unit) or adding other components are possible.

  The image acquisition unit 112 (program module for image acquisition processing) acquires a captured image. The recognition processing unit 114 (recognition processing program module) performs recognition processing on a captured image.

  The virtual space setting unit 116 (virtual space setting processing program module) performs setting processing of a virtual space (object space) in which a plurality of objects are arranged. For example, various objects representing display objects such as moving objects (people, robots, cars, trains, airplanes, ships, monsters, animals, etc.), maps (terrain), buildings, auditoriums, courses (roads), trees, walls, water surfaces, etc. A process of placing and setting an object (an object composed of a primitive surface such as a polygon, a free-form surface, or a subdivision surface) in a virtual space is performed. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects. Specifically, in the object information storage unit 172 of the storage unit 170, object information that is information such as the position, rotation angle, moving speed, and moving direction of an object (part object) in the virtual space is associated with the object number. Is remembered. The virtual space setting unit 116 performs a process of updating the object information for each frame, for example.

  The virtual camera setting unit 118 (virtual camera setting processing program module) performs processing for setting a virtual camera in the virtual space. An image seen from the virtual camera is generated as a virtual space image.

  The image generation unit 120 (program module for image generation processing) performs image generation processing. For example, drawing processing is performed based on the results of various processing (virtual space setting processing, virtual camera setting processing, etc.) performed by the processing unit 100, thereby generating an image and displaying it on the display unit 190. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. Based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vector, α value, etc.) are created. Then, based on this drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is converted into image information in units of pixels such as a drawing buffer 174 (frame buffer, work buffer, etc.). Draw in a buffer that can be stored. Thereby, an image that can be seen from the virtual camera in the virtual space is generated. Note that the drawing processing performed by the image generation unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

  The sound generation unit 130 (sound generation processing program module) performs sound generation processing based on the results of various processing performed by the processing unit 100. Specifically, music (music, BGM), sound effects, sound, or the like is generated and output to the sound output unit 192.

  The image generation system according to the present embodiment includes an image acquisition unit 112, a recognition processing unit 114, a virtual space setting unit 116, and an image generation unit 120, as shown in FIG.

  The image acquisition unit 112 acquires a captured image. For example, a captured image captured by the imaging unit 162 is acquired. Note that an image received from the communication unit 196 or an image stored in the storage unit 170 may be acquired as a captured image.

  The recognition processing unit 114 performs a recognition process on the acquired captured image. For example, the recognition process of the recognition target object shown in the captured image is performed. For example, the recognition processing unit 114 performs a matching process using the acquired captured image and a registered image of the recognition object stored in the recognition object information storage unit 176. And by this matching process, it is detected whether the image of the recognition target object corresponding to the registration image of a recognition target object exists in a captured image. For example, the recognition object information storage unit 176 stores the feature point information of the recognition object as information of the registered image of the recognition object. The recognition processing unit 114 performs recognition processing of the recognition target object displayed in the captured image by performing matching processing between the feature point information of the registered image of the recognition target object and the feature point information of the recognition target object in the captured image. Based on the result of the recognition processing, for example, positional relationship information between the imaging unit 162 and the recognition target is estimated. For example, the positional relationship information between the imaging unit 162 and the recognition object is estimated from the size, shape, and the like of the recognized recognition object. That is, the position and orientation of the imaging unit 162 that captures a captured image in which the recognition target object is captured is estimated. By estimating the position and orientation as described above, it is possible to obtain positional relationship information between the imaging unit 162 (the user's terminal device or viewpoint) that captures the captured image and the recognition target object.

  Here, the recognition target is, for example, an AR marker. However, the recognition target is not limited to the AR marker, and may be a target recognized by a feature point or contour shape matching process like a markerless AR. That is, the recognition target is not limited to an AR marker such as a square marker or a special shape marker, and may be an AR target recognized by natural feature point matching or contour shape matching processing.

  The virtual space setting unit 116 performs processing for setting the arrangement of at least one three-dimensional object and a mask object in the virtual space. Specifically, a process of arranging and setting in the virtual space at least one three-dimensional object and a mask object arranged with a predetermined relationship with the three-dimensional object is performed. The predetermined relationship (given relationship) is, for example, a relative positional relationship or a directional relationship between the three-dimensional object and the mask object. For example, the 3D object and the mask object are arranged and set in the virtual space with such a relationship that the display of the 3D object can be masked by the mask object. The virtual space setting unit 116 sets and arranges a three-dimensional object such as a character object or a background object in the virtual space. A three-dimensional object is a display object composed of one or a plurality of primitive surfaces (polygons or the like). The virtual space setting unit 116 sets and arranges a mask object for masking the three-dimensional object in the virtual space. For example, a mask object (opaque object) for preventing a three-dimensional object from being displayed in a place other than the opening is arranged and set in the virtual space.

  Then, the image generation unit 120 generates an image in which the virtual space image is displayed in the image area (drawing area) of the opening and the captured image is displayed in the image area (drawing area) around the opening. . Specifically, the image generation unit 120 generates an image in which a virtual space image composed of an image of a three-dimensional object is displayed in the image area of the opening set corresponding to the location of the recognition target object. . On the other hand, in the image area around the opening (image area other than the opening), an image in which the captured image is displayed is generated by masking the three-dimensional object with the mask object.

  The image area of the opening is set based on the recognition result of the recognition object in the recognition processing unit 114, for example. For example, the size and shape image area corresponding to the size and shape of the recognition object shown in the captured image are set as the image area of the opening (a peephole). A virtual space image composed of an image of a three-dimensional object is displayed in the image area of the opening. Then, the virtual space image and the captured image are combined so that the virtual space image is displayed in the image area of the opening and the captured image is displayed in the image area around the opening. This image composition processing is performed by the image composition unit 122. Here, the image area around the opening need not be a 360-degree all-around image area centered on the opening, but may be a partial image area of the all-around image area. In addition, in an image in which a 3D object straddles an opening, an image of an outer part of the 3D object (outer part of the opening) is displayed instead of a captured image even in an image region around the opening. It may be.

  The virtual space image is an image that can be seen from the virtual camera in the virtual space, and the predetermined relationship is, for example, a relationship in which the three-dimensional object is located behind the predetermined surface of the mask object when viewed from the virtual camera. is there. In this way, it is possible to generate an augmented reality image as if a virtual space exists behind the predetermined surface of the mask object. In this case, in the predetermined relationship, a part or all of some of the three-dimensional objects (for example, character objects) of the plurality of three-dimensional objects are located on the near side of the predetermined plane when viewed from the virtual camera. Events such as movement and placement are allowed to occur. The predetermined plane is a plane on the near side as viewed from the virtual camera, for example, among the mask object planes. For example, when the mask object is composed of a plurality of surfaces, the predetermined surface is a surface on the back side when viewed from the virtual camera (for example, a surface on the deepest side) among the plurality of surfaces. For example, the predetermined surface is a surface having a short depth distance from the virtual camera (for example, a surface having the closest depth distance) among the plurality of surfaces. The predetermined surface is a surface located at a location corresponding to the opening, for example. For example, the depth distance of a predetermined surface based on the virtual camera and the depth distance of the opening portion based on the virtual camera can be set to the same distance.

  The virtual space setting unit 116 sets and arranges a mask object having a first surface including the opening in the virtual space. For example, the mask object only needs to have at least a first surface including the opening. The first surface is, for example, a surface that spreads from the surface of the opening to the periphery of the opening.

  In this case, the predetermined relationship can be a relationship in which the three-dimensional object is located behind the first surface of the mask object as viewed from the virtual camera. That is, in this case, the first surface is the above-described predetermined surface. This makes it possible to have a relationship such that a three-dimensional object exists behind the first surface (a predetermined surface in a broad sense) of the mask object as viewed from the virtual camera. It becomes possible to generate an augmented reality image as if a virtual space exists behind the first surface of the mask object. In this case, in the predetermined relationship, a part or all of a part of the three-dimensional objects among the plurality of three-dimensional objects is moved or arranged on the front side of the first surface when viewed from the virtual camera. It is permissible for an event such as

  In addition, the virtual space setting unit 116 may place and set a mask object having a first surface and at least one surface intersecting the first surface in the virtual space. That is, in this case, the mask object includes a first surface and at least one surface intersecting the first surface. For example, when there are second, third, fourth, and fifth surfaces intersecting (orthogonal) with the first surface, the mask object is constituted by the first to fifth surfaces, for example, a cube or a rectangular parallelepiped. Become an object. The cubic or rectangular parallelepiped object may have a sixth surface that intersects (orthogonally) the second to fifth surfaces. The sixth surface is a surface facing the first surface.

  In this case, the predetermined relationship is a relationship in which the three-dimensional object is located in a region surrounded by the first surface (predetermined surface) and at least one surface (at least one surface intersecting the predetermined surface). can do. In this way, by positioning the three-dimensional object in a region surrounded by the first surface (predetermined surface) and at least one surface (at least one surface intersecting the predetermined surface) of the mask object, Using a mask object, a three-dimensional object can be masked more reliably. For example, when at least one surface is the above-described second to fifth surfaces, the predetermined relationship is a relationship that a three-dimensional object is located in a region surrounded by the first to fifth surfaces. become. Further, when at least one surface is the above-described second to sixth surfaces, the predetermined relationship is a relationship that the three-dimensional object is located in a region surrounded by the first to sixth surfaces. become. It should be noted that some or all of some of the three-dimensional objects (for example, character objects) are surrounded by a first plane and at least one plane (first to fifth areas). A game effect such as movement or placement outside the area surrounded by the face or the area surrounded by the first to sixth faces may be performed.

  In addition, the virtual space setting unit 116 sets the three-dimensional object in the virtual space so that the three-dimensional object is positioned on the back side of the opening. The back side with respect to the opening is, for example, the back side in the depth direction as viewed from the imaging unit 162 (the user's terminal device or viewpoint). For example, the virtual space setting unit 116 arranges and sets the three-dimensional object in the virtual space so that the three-dimensional object is displayed on the back side of the opening that is the viewing hole. In this case, a part of the three-dimensional object may be in front of the opening.

  The virtual space setting unit 116 sets the three-dimensional object in the virtual space so that the three-dimensional object straddles the opening. Here, the three-dimensional object straddling the opening means that, for example, a part of the three-dimensional object is on the near side of the opening and the other part of the three-dimensional object is on the back side of the opening. That is, the virtual space setting unit 116 may display the 3D object such that, for example, a part of the 3D object is displayed on the front side of the opening and the other part of the 3D object is displayed on the back side of the opening. Is set in the virtual space.

  In addition, the image generation unit 120 performs a drawing process according to the positional relationship information between the imaging unit 162 that captures the captured image and the recognition target, and generates a virtual space image. For example, the positional relationship information between the imaging unit 162 and the recognition target corresponds to the positional relationship information between the user's terminal device or the user's viewpoint and the recognition target. That is, the image generation unit 120 performs a drawing process according to the positional relationship information between the user terminal device or the viewpoint and the recognition target object, and generates a virtual space image. The user terminal device is a device realized by the image generation system of FIG. 1, and is, for example, a terminal device 10 such as a tablet PC or a smartphone as shown in FIG. 2 described later. The user's viewpoint is, for example, the user's viewpoint VP when using an HMD (head-mounted display device) as shown in FIG. The positional relationship information is the position and direction of the recognition object with reference to the imaging unit 162 (terminal device or viewpoint), or the position and direction of the imaging unit 162 (terminal device or viewpoint) with reference to the recognition object.

  The drawing process according to the positional relationship information between the imaging unit 162 and the recognition target object is, for example, setting the image area of the opening according to the positional relationship information. In the present embodiment, image synthesis is performed such that a virtual space image is displayed in the opening according to the positional relationship information. In addition, the drawing process according to the positional relationship information between the imaging unit 162 and the recognition target object is to perform a three-dimensional object drawing process by setting a virtual camera at a position and a direction according to the positional relationship information.

  Specifically, the virtual camera setting unit 118 sets a virtual camera at a position corresponding to the imaging unit 162 in the virtual space. For example, in the virtual space, the virtual camera is set at a position and a direction according to the positional relationship information between the imaging unit 162 and the recognition target object. For example, in the real world, the positional relationship information between the imaging unit 162 and the recognition target object matches the positional relationship information between the virtual camera in the virtual space and the opening corresponding to the recognition target object. Set. Then, the image generation unit 120 generates an image that can be seen from the virtual camera as a virtual space image. For example, in the real world, when the recognition target is directly facing the imaging unit 162 (the user's terminal device or viewpoint), an image that can be seen from the virtual camera set in the front direction of the three-dimensional object in the virtual space is displayed. Generated. Further, when the recognition target is located in an oblique direction with respect to the imaging unit 162 (user terminal device or viewpoint), an image that can be seen from a virtual camera set in an oblique direction of a three-dimensional object in the virtual space is generated. .

  In addition, the image generation unit 120 performs a process of changing the visibility of the virtual space image in accordance with the positional relationship information between the imaging unit 162 (the user's terminal device or viewpoint) and the recognition target object. The process for changing the visibility of the virtual space image is a process for increasing or decreasing the visibility of the virtual space image. The process for changing the visibility can be realized by, for example, a process for changing the brightness, contrast, translucency or blurring degree of the virtual space image. For example, when the recognition object is directly facing the imaging unit 162, a process for increasing the visibility of the virtual space image is performed, for example. For example, processing for increasing the brightness of the virtual space image, increasing the contrast, making it more opaque, or reducing the blur level is performed. Further, when the recognition target object is located in an oblique direction with respect to the imaging unit 162, for example, a process of reducing the visibility of the virtual space image is performed as compared with the case where the recognition object is facing directly. For example, a process of darkening the brightness of the virtual space image, lowering the contrast, making it more transparent, or increasing the blurring degree is performed.

  In addition, the image generation system of the present embodiment includes a sound generation unit 130, and the sound generation unit 130 changes in accordance with positional relationship information between the imaging unit 162 (user terminal device or viewpoint) and the recognition target object. Performs sound generation processing. For example, the sound generation unit 130 increases the volume of the generated sound when the distance between the imaging unit 162 and the recognition target is short, and the distance is short when the distance between the imaging unit 162 and the recognition target is long. Compared to the case, the volume of the generated sound is reduced. That is, sound parameters such as sound volume are changed according to the relative positional relationship between the imaging unit 162 and the recognition object. Note that the pitch, timbre, and the like of the sound may be changed according to the positional relationship information between the imaging unit 162 and the recognition object. In this way, changing the sound according to the positional relationship information between the imaging unit 162 and the recognition target means changing at least one of the volume, the height, the tone color, and the like according to the positional relationship information, for example. It is. Such sound generation processing can be realized, for example, by setting the sound source of the generated sound at the position of the virtual space set on the back side of the opening.

  In the present embodiment, information about the three-dimensional object is stored in the storage unit 170 in association with the information about the recognition target. For example, the recognition target object information storage unit 176 of the storage unit 170 stores information of a three-dimensional object (character information, virtual space setting information) in association with the recognition target object. Then, the virtual space setting unit 116 sets and arranges the three-dimensional object associated with the information on the recognition target object in the virtual space. For example, when the recognition processing unit 114 recognizes that the recognition target object is reflected in the captured image, information on the three-dimensional object associated with the recognized recognition target object is stored in the storage unit 170 (recognition target information storage unit 176). ). The virtual space setting unit 116 performs processing for arranging and installing the three-dimensional object in the virtual space based on the read information of the three-dimensional object. For example, a three-dimensional object such as a character object or a background object is set in the virtual space. In the virtual space thus set, for example, a virtual camera is set at a position corresponding to the positional relationship information between the imaging unit 162 and the recognition object, and an image that can be seen from the virtual camera is generated as a virtual space image. . Then, a virtual space image is displayed at the location of the opening, and an image in which the captured image is displayed at a location other than the opening is generated.

  The recognition processing unit 114 may perform processing for recognizing an image displayed on the display unit of the electronic device as a recognition target object. For example, a process of recognizing a display object such as a character displayed on the display unit of the electronic device as a recognition target object is performed. For example, like the markerless AR described above, processing for recognizing a display object such as a character as a recognition target object is performed by matching processing of feature points and contour shapes. For example, the recognition target object information storage unit 176 stores registered feature point information or registered contour shape information of the display object as registration information of the display object. The recognition processing unit 114 performs a feature point matching process or a contour shape matching process between the registered feature point information or the registered contour shape information and the feature point information or the contour shape information of a display object such as a character shown in the captured image, It recognizes whether the display object shown in the captured image matches the registered display object. If it is determined that they match, the information of the three-dimensional object associated with the display object that is the recognition target is read from the storage unit 170, and a virtual space image is generated. Here, as the electronic device, for example, an information processing device such as a game machine, a PC or a notebook PC, a portable terminal device such as a smartphone or a mobile phone, an in-vehicle device such as a video device, an audio device, or a navigation device. Various devices such as can be assumed.

  The recognition processing unit 114 may perform processing for recognizing a toy or an article as a recognition target object. For example, like the markerless AR described above, a process of recognizing a toy or an article as a recognition object is performed by a matching process of feature points or contour shapes. For example, the recognition target object information storage unit 176 stores registered feature point information or registered contour shape information of the toy or article as the toy or article registration information. The recognition processing unit 114 performs a feature point matching process or a contour shape matching process between the registered feature point information or the registered contour shape information and the feature point information or the contour shape information of the toy or the article reflected in the captured image, and the captured image It recognizes whether the toy or article reflected in the item matches the registered toy or article. Then, if it is determined that they match, the information of the three-dimensional object associated with the toy or article that is the recognition target is read from the storage unit 170, and a virtual space image is generated. Here, the toy is a so-called toy or the like, and may be an electronic toy that operates with electricity or a toy other than an electronic toy. Examples of toys include educational toys, infant toys, models (plastic models), dolls and stuffed animals, figures, capsule trays, acoustic toys, practice toys, puzzles, playing cards, card games, board games such as baseball boards, and sports toys. Various toys such as vehicle toys or war toys can be envisaged. The article is a tangible object such as an article or an article. For example, the article is an object to be sold or traded, for example, a product or an article. Articles (goods, supplies) include, for example, containers (plastic bottles, etc.), daily necessities, office supplies, stationery, foods, gifts, music media (CDs, etc.), video media (DVDs, etc.), posters, electronic devices, Various articles such as clothes, footwear (shoes, etc.), bags, belts, furniture, etc. can be assumed.

2. Next, the method of this embodiment will be described in detail.

2.1 Display of Virtual Space Image in Opening In FIG. 2, the user has, for example, the terminal device 10 that is a tablet PC on the right hand and the CD case 20 on the left hand. For example, a recognition object MK is printed on a cover or the like in the CD case 20. The recognition target object MK (AR image) in FIG. 2 is, for example, a two-dimensional character image. The AR marker AMK may be printed on the cover or the like in the CD case 20. The AR marker AMK is a marker having a predetermined image pattern. If AR marker AMK is used, there exists an advantage that specification of positional relationship information with the imaging part 162 is easy.

  In the terminal device 10, the display unit 190 is located on the user side, and an imaging unit 162 (camera) is provided on the back side of the terminal device 10. The imaging unit 162 can shoot the back side of the terminal device 10. For example, when the user moves the CD case 20 to the back side of the terminal device 10, the CD case 20 is photographed by the imaging unit 162 on the back side of the terminal device 10. Then, the recognition processing unit 114 in FIG. 1 performs recognition processing of the recognition target object MK (or AR marker AMK) of the CD case 20, and as shown in FIGS. 6 and 7 described later, the character corresponding to the recognition target object MK. An AR image in which a virtual space image of a three-dimensional object such as CH is displayed in the image region ROP of the opening is displayed on the display unit 190 of the terminal device 10.

  3-5 is a figure for demonstrating the method of this embodiment. 3 to 5, the character CH and the background objects OB <b> 1 to OB <b> 11 are three-dimensional objects arranged and set in the virtual space. The three-dimensional object is a three-dimensional object composed of, for example, a primitive surface (polygon). In this embodiment, these three-dimensional objects (CH, OB1 to OB11) and mask objects (MS1 to MS6) are arranged and set in the virtual space. For example, a process of setting the arrangement of the three-dimensional object (CH, OB1 to OB11) and the mask object (MS1 to MS6) arranged with a predetermined relationship with the three-dimensional object in the virtual space is performed. Then, an image seen from the virtual camera VC is generated as a virtual space image. The position and direction of the virtual camera VC are set in accordance with, for example, positional relationship information between the imaging unit 162 (user terminal device or viewpoint) and the recognition target object MK in FIG.

  As shown in FIG. 3, in this embodiment, an opening OP that is a hole is set. For example, the opening OP is a virtual peephole in the virtual space image. As shown in FIGS. 6 and 7 to be described later, in this embodiment, an image in which a virtual space image is displayed in the image region ROP of the opening OP is generated and displayed on the display unit 190 of the terminal device 10. .

  The mask objects (MS1 to MS6) are objects for masking the display of the three-dimensional objects (CH, OB1 to OB11) in a region other than the opening OP, and are, for example, opaque objects. As shown in FIGS. 6 and 7 to be described later, the captured image is displayed in the image region RMS masked by the mask object. That is, in the present embodiment, the virtual space image is displayed in the image region ROP corresponding to the opening OP, and the image in which the captured image is displayed is combined with the image region RMS masked by the mask object, and the display unit 190 is displayed. Is displayed.

  For example, in FIG. 3, only the character CH and the background object OB1 (floor) located in the opening OP can be seen, but actually, as shown in FIGS. 4 and 5, there is other space in the virtual space (object space). Three-dimensional objects such as background objects OB2 to OB11 are arranged. In FIG. 3, the three-dimensional objects such as the background objects OB2 to OB11 are masked by the mask object and cannot be seen.

  Here, the mask object has a surface MS1 (first surface) that encloses the opening OP, and in the present embodiment, such a mask object is arranged and set in the virtual space. For example, the surface MS1 is a surface that extends from the location of the opening OP around the opening OP. In other words, the hole provided in the surface MS1 is the opening OP.

  In this case, the predetermined relationship described above is such that the three-dimensional objects (at least the background objects OB1 to OB11) are located behind the mask object surface MS1 (predetermined surface in a broad sense) when viewed from the virtual camera VC. It becomes the relationship. By giving such a relationship, an augmented reality can be realized as if a virtual space constituted by a three-dimensional object exists on the back side of the surface MS1. However, as shown in FIG. 5 and FIG. 7 described later, an effect process such that a part of the three-dimensional object (character CH or the like) moves to the near side of the surface MS1 (predetermined surface) may be performed. Note that the predetermined surface in the predetermined relationship of the present embodiment is not limited to such a surface MS1. For example, the predetermined surface may be a surface that does not include the opening OP.

  The mask object has a surface MS1 and at least one surface intersecting the surface MS1. In FIG. 3, the surfaces MS2, MS3, MS4, and MS5 are arranged and set in the virtual space as at least one surface intersecting (for example, orthogonal to) the surface MS1. Further, in FIG. 3, the mask object further has a surface MS6 that intersects (for example, is orthogonal to) the surfaces MS2, MS3, MS4, and MS5. For example, a cubic mask object is formed by the surfaces MS1 to MS6.

  In this case, the above-described predetermined relationship is that the three-dimensional object is in a region surrounded by the surface MS1 (predetermined surface) and at least one surface intersecting the surface MS1 (at least one surface intersecting the predetermined surface). The relationship is that (at least the background objects OB1 to OB11) are located. For example, the predetermined relationship is a relationship in which a three-dimensional object (background objects OB1 to OB11) is located in a region surrounded by the surfaces MS1 to MS5 or a region surrounded by the surfaces MS1 to MS6. However, as shown in FIG. 5 and FIG. 7 described later, there is an effect process in which some three-dimensional objects (such as characters CH) move out of the area surrounded by the planes MS1 to MS5 or the planes MS1 to MS6. It may be done.

  Note that objects of various shapes can be used as the mask object. For example, the shape is not limited to the cubic shape as shown in FIG. 3, and may be a rectangular parallelepiped shape, a columnar shape (triangular prism shape or the like), or a curved surface shape (spherical shape or the like). For example, the mask object only needs to have at least the surface MS1. For example, when the mask object is configured only by the surface MS1 including the opening OP, it is desirable that the area of the surface MS1 is as large as possible (for example, a substantially infinite area).

  6 to 8 are examples of images generated by the method of this embodiment. A virtual space image is displayed in the image area ROP of the opening (OP in FIG. 3). The image area ROP of the opening is set corresponding to the location of the recognition object MK in FIG. Specifically, the image area ROP of the opening is set based on positional relationship information (relative positional relationship) between the imaging unit 162 (terminal device) and the recognition target object MK. For example, the CD case 20 (recognition object MK) is located on the back side of the terminal device 10 in FIG. 6 in a position and direction corresponding to the shape and size of the image area ROP of the opening.

  For example, in FIG. 6, the CD case 20 of FIG. 2 is at an angle of θ = θ1 with respect to a direction (normal direction) orthogonal to a plane (plane of the display unit 190) that is the main surface of the terminal device 10. It is located on the back side. In FIG. 7, the CD case 20 is positioned on the back side of the terminal device 10 at an angle θ = θ2 with respect to the direction orthogonal to the plane (main surface) of the terminal device 10, and θ1> θ2. . That is, the angle θ formed by the planar direction of the CD case 20 with respect to the direction orthogonal to the plane of the terminal device 10 is smaller than that in FIG. 6 in FIG. Approaching an orthogonal direction.

  In FIG. 8, the CD case 20 is positioned on the back side of the terminal device 10 at an angle θ = θ3 with respect to the direction orthogonal to the plane of the terminal device 10, and θ1> θ2> θ3. That is, the angle θ formed by the planar direction of the CD case 20 with respect to the direction orthogonal to the plane of the terminal device 10 is smaller than that in FIGS. 7 and 6 in FIG. 8, and the planar direction of the CD case 20 is It is closer to the direction perpendicular to the plane. 6 and 7, a virtual three-dimensional image is displayed in the image area ROP of the opening. However, in FIG. 8, since it is difficult to stably recognize the recognition target object MK, the virtual three-dimensional image is displayed. Is not displayed. For example, if a virtual three-dimensional image is displayed in the situation of FIG. 8, the display may become unstable. However, even in the situation shown in FIG. 8, it is possible to set so that the virtual three-dimensional image is displayed without disappearing.

  In FIG. 6, three-dimensional objects such as a character CH and background objects OB1, OB2, OB4, and OB5 are displayed in the image area ROP of the opening. The character CH is a three-dimensional image character corresponding to the recognition object MK that is the character of the two-dimensional image of FIG.

  In the virtual space image of the image region ROP of the opening in FIG. 6, the virtual camera VC in FIG. 3 is disposed at a position diagonally to the right with respect to the opening OP, and looks into the opening OP like a peephole. This corresponds to an image seen from the virtual camera VC. Therefore, in FIG. 6, the images of the character CH and the background objects OB1, OB2, OB4, and OB5 that can be seen through the opening OP are displayed in the image area ROP of the opening. On the other hand, the background objects OB3 and OB6 to OB11 in FIG. 4 at the positions masked by the mask objects (MS1 to MS6) are not displayed by being masked by the mask objects.

  As shown in FIG. 6, the captured image captured by the imaging unit 162 is displayed in the image region RMS around the opening. For example, in the image region RMS, an image of the user's left hand taken by the imaging unit 162 is displayed. Specifically, a captured image is displayed in the image region RMS around the opening by masking the background objects OB3 and OB6 to OB11 in FIG. 4 with a mask object. As described above, in FIG. 6, the virtual space image is displayed in the image area ROP of the opening, and the image composition is performed such that the captured image is displayed in the image area RMS around the opening, and the combined image is the terminal device. 10 display units 190.

  In FIG. 7, the angle formed by the planar direction of the CD case 20 with respect to the direction orthogonal to the plane of the terminal device 10 is smaller than that in FIG. 6. Thereby, the virtual camera VC of FIG. 3 is arranged at a position further obliquely to the right with respect to the opening OP. For this reason, the images of the background objects OB2, OB4, and OB5 displayed in the image area ROP of the opening in FIG. 6 are not displayed in FIG. 7, but the image of the background object OB6 is displayed in FIG. 7 instead. . That is, the virtual camera VC shown in FIG. 3 is arranged at a position further obliquely to the right of the opening OP in FIG. 6 than in FIG. 7, and the image seen when looking into the opening OP from the virtual camera VC at the arrangement position. Is displayed in the image area ROP. Therefore, the images of the background objects OB2, OB4, and OB5 displayed in the image region ROP in FIG. 6 are not displayed in FIG. 7, but the image of the background object OB6 is displayed.

  In the image region RMS around the opening in FIG. 6, a method of masking a three-dimensional object with a mask object and hiding it can be realized by, for example, an AR occlusion method (occlusion shader). For example, in AR, an occlusion technique is used in which a virtual object on the back side of a real world object (an object reflected in a captured image) is hidden and erased. Specifically, an image area of an object in the real world is set as an occlusion area, and in this occlusion area, the virtual object is erased and hidden. For example, in the occlusion area, the virtual object is treated as a transparent object, and the image of the virtual object is not displayed but the captured image is displayed. As a result, in the occlusion area corresponding to the image area of the real world object, it is possible to display an image as if the virtual object has been hidden by the real world object.

  Then, this occlusion technique can be used to mask the background objects OB3 and OB6 to OB11 in FIG. 4 with the mask objects (MS1 to MS6) and hide them in FIG. That is, the mask object area is set as an occlusion area, and the background objects OB3 and OB6 to OB11 located behind the mask object as viewed from the virtual camera VC in FIG. . As a result, the images of the background objects OB3 and OB6 to OB11 are not displayed in the image region RMS around the opening in FIG. As a result, as shown in FIG. 6, an image is generated as if the opening portion was used as a peephole and the virtual space on the back side was looked into.

  As described above, in this embodiment, the three-dimensional object in the virtual space is arranged and displayed behind the position of the recognition object MK (AR marker), and the three-dimensional object in the virtual space cannot be seen from other than the opening. An image composition method is realized. Thereby, the process of displaying the virtual space image in the area of the opening corresponding to the area of the recognition object can be realized by a simple process.

  In the present embodiment, the three-dimensional object is arranged and set in the virtual space so that the three-dimensional object is located behind the opening. For example, in FIG. 6, the character CH, which is a three-dimensional object, and the background objects OB1, OB2, OB4, and OB5 are arranged and set in the virtual space so as to be located behind the opening (image region ROP). According to such a virtual space arrangement setting method, a three-dimensional object (CH, OB1, OB2, OB4, OB5) exists on the back side of the opening, and the three-dimensional object passes through the opening that is a peephole. You can generate an image that looks like you are looking into it. Therefore, an augmented reality (AR) that has never been achieved can be realized.

  In this embodiment, the 3D object is arranged and set in the virtual space so that the 3D object straddles the opening. For example, in FIG. 5, the character CH, which is a three-dimensional object, is arranged and set in the virtual space so as to straddle the opening OP. Specifically, a part (upper body side) of the character CH is positioned on the near side of the opening OP, and the other part (lower body side) of the character CH is positioned on the back side of the opening OP. A character CH is arranged. In this way, as shown in FIG. 7, it is possible to generate an image that looks as if the character CH is protruding from the opening (image region ROP). Therefore, it is possible to express augmented reality like never before.

  Further, in the present embodiment, in FIG. 9, drawing processing is performed according to the positional relationship information between the imaging unit 162 (the terminal device 10 of the user US) and the recognition target object MK (CD case 20), and FIGS. A virtual space image as shown in FIG. Specifically, a virtual camera is set at a position corresponding to the imaging unit 162 in the virtual space, and an image seen from the set virtual camera is generated as a virtual space image.

  For example, in FIG. 9, the relative positional relationship between the recognition object MK (CD case 20) located on the back side of the terminal device 10 and the imaging unit 162 of the terminal device 10 is obtained. This relative positional relationship can be obtained by analyzing the size and shape of the recognition object MK shown in the captured image of the imaging unit 162. Then, the virtual camera VC of FIG. 3 is set at a position and direction corresponding to the obtained relative positional relationship. For example, the virtual camera VC is arranged and set so that the relative positional relationship between the opening OP and the virtual camera VC in FIG. 3 matches the relative positional relationship between the recognition target object MK and the imaging unit 162 in FIG. To do. Then, a rendering process for generating a virtual space image that can be seen from the virtual camera VC set in this way is performed, and the generated virtual space image is displayed in the image area ROP of the opening as shown in FIGS. Generate an image to be processed.

  For example, in FIG. 9, when the recognition object MK (CD case 20) is located at a location directly facing the imaging unit 162 of the terminal device 10, an opening portion is located at a location directly facing the virtual camera VC in FIG. The virtual camera VC is arranged and set so that the OP is located. As a result, an image of the virtual space viewed from the virtual camera VC is generated as a virtual space image.

  In FIG. 9, when the recognition object MK (CD case 20) is located at a position oblique to the imaging unit 162 of the terminal device 10, the position oblique to the virtual camera VC is also shown in FIG. In addition, the virtual camera VC is arranged and set so that the opening OP is positioned. As a result, an image obtained by viewing the virtual space from the oblique direction with the virtual camera VC is generated as a virtual space image.

  9, when the recognition object MK (CD case 20) is located at a position away from the imaging unit 162 of the terminal device 10, the position away from the opening OP in FIG. 3 as well. The virtual camera VC is arranged and set. As a result, an image viewed from a virtual camera located far away is generated as a virtual space image.

  As described above, when a drawing process corresponding to the positional relationship information between the imaging unit 162 and the recognition target object MK is performed to generate a virtual space image, an image as if the imaging unit 162 saw the virtual space is displayed as a virtual space image. A spatial image is generated. Then, as shown in FIGS. 6 and 7, the virtual space image generated in this way is displayed in the image area ROP of the opening, so that the imaging section 162 can use the opening as a peephole. It becomes possible to generate an image that looks as if you looked into the virtual space in the back, and you can realize unprecedented augmented reality.

  In FIG. 10, the user US is wearing an HMD (head-mounted display device). For example, the HMD is worn so as to cover the field of view of the user US. For example, the HMD is realized by the image generation system of this embodiment. The HMD is provided with an imaging unit 162 at a position corresponding to the viewpoint VP of the user US. Then, the captured image (video) captured by the imaging unit 162 is displayed on the HMD. That is, an image as shown in FIGS. 6 and 7 in which the virtual space image is synthesized with the captured image of the imaging unit 162 is displayed on the HMD.

  Specifically, in FIG. 10, drawing processing is performed according to the positional relationship information between the HMD imaging unit 162 (the viewpoint VP of the user US) and the recognition target object MK (CD case 20), and FIGS. A virtual space image as shown in FIG. That is, a virtual camera is set at a position corresponding to the imaging unit 162 (viewpoint VP) in the virtual space, and an image seen from the set virtual camera is generated as a virtual space image. And the image of FIG. 6, FIG. 7 produced | generated by synthesize | combining this virtual space image and the picked-up image of the imaging part 162 is displayed on HMD.

  For example, in FIG. 9, the user US needs to hold the CD case 20 on which the recognition object MK is printed with one hand and the terminal device 10 with the imaging unit 162 provided with the other hand. There is a disadvantage that the trouble of US becomes complicated. On the other hand, in FIG. 10, the user US does not need to have a terminal device or the like, and directs the HMD provided with the imaging unit 162 on the front side toward the CD case 20 on which the recognition object MK is printed. good. That is, the user US only needs to point the CD case 20 toward the viewpoint VP. Therefore, in FIG. 10, compared with FIG. 9, there exists an advantage that the effort of the user US for experiencing augmented reality can be reduced.

  Note that the HMD in FIG. 10 includes a display unit realized by, for example, an organic EL display (OEL) or a liquid crystal display (LCD). The HMD display unit is provided with a first display or first display area set in front of the left eye of the user US, and a second display or second display area set in front of the right eye. 3D display is possible. When performing stereoscopic display, for example, a left-eye image and a right-eye image with different parallax are generated, the left-eye image is displayed on the first display, and the right-eye image is displayed on the second display. Alternatively, the left-eye image is displayed in the first display area of one display, and the right-eye image is displayed in the second display area. The HMD is provided with two eyepiece lenses (fisheye lenses) for the left eye and the right eye, thereby expressing a VR space that extends over the entire periphery of the field of view of the user US. Then, correction processing for correcting distortion generated in an optical system such as an eyepiece is performed on the left-eye image and the right-eye image. 10 is a non-transmissive type, the HMD may be a transmissive type. The HMD may be a so-called glasses-type HMD.

  Moreover, in this embodiment, the process which changes the visibility of a virtual space image is performed according to the positional relationship information of the imaging part 162 which image | photographs a captured image, and a recognition target object. Also, a sound generation process that changes in accordance with the positional relationship information between the imaging unit 162 and the recognition object is performed.

  For example, in FIG. 11A, the imaging unit 162 (camera) of the terminal device 10 faces the recognition object MK of the CD case 20 at a close distance. In this case, for example, a process for improving the visibility is performed, such as making the virtual space image a bright image or a clearer image. That is, processing for improving the visibility of the virtual space image displayed in the image region ROP of FIGS. 6 and 7 is performed. Further, the volume of the sound output from the sound output unit (speaker, headphone terminal) of the terminal device 10 is increased. For example, in the present embodiment, an effect process is performed in which music such as BGM flows from the virtual space on the back side of the opening in FIGS. As shown in FIG. 11A, when the imaging unit 162 is facing the recognition object MK at a close distance, the volume of music such as BGM flowing from the virtual space is increased.

  On the other hand, in FIG. 11B, the recognition target object MK is located at a position oblique to the imaging direction (optical axis direction) of the imaging unit 162. In this case, for example, a process for reducing the visibility is performed such as making the virtual space image a dark image or a blurred image. That is, processing for reducing the visibility of the virtual space image displayed in the image region ROP of FIGS. 6 and 7 is performed. Further, the volume of the sound output from the sound output unit of the terminal device 10 is reduced. For example, the volume of music such as BGM flowing from the virtual space behind the opening is reduced.

  In FIG. 11C, the recognition object MK is located at a location far from the imaging unit 162. Also in this case, for example, a process for reducing the visibility is performed such as making the virtual space image a dark image or a blurred image. Further, the volume of the sound output from the sound output unit of the terminal device 10 is reduced. For example, the volume of music flowing from the virtual space behind the opening is reduced.

  As shown in FIGS. 11A to 11C, it is possible to increase the real degree of augmented reality by performing rendering processing such as changing the visibility of the virtual space image or changing the sound. . For example, in the present embodiment, an effect process is performed in which music flows from a virtual space behind the opening. In this case, as shown in FIG. 11A, when the positional relationship between the imaging unit 162 and the recognition target object MK corresponding to the opening is a direct relationship or when the distance is short, When the volume increases and the relationship or distance in the oblique direction is long, it is possible to realize an effect process that reduces the volume of the music. Therefore, it is possible to give the user a virtual reality feeling as if the music is actually flowing from the virtual space behind the opening, and the realism of augmented reality can be improved.

  In the present embodiment, a sound source of music such as BGM may be set in the virtual space at the back of the opening, and sound processing may be performed so that music from this sound source is output from the sound output unit. For example, sound processing of stereophonic sound (for example, 5.1ch stereophonic sound) in which music can be heard from the virtual space behind the opening may be performed. By doing so, the real degree of augmented reality can be further improved.

  In the present embodiment, information about a three-dimensional object is stored in the storage unit 170 (recognition target information storage unit 176) in FIG. 1 in association with the recognition target information. Then, the recognition target object information is read from the storage unit 170, the three-dimensional object associated with the recognition target object information is arranged and set in the virtual space, and a virtual space image is generated.

  For example, FIG. 12 shows an example of the data structure of the recognition target object information. In FIG. 12, merchandise, character information, and virtual space information are associated with recognition object information. The product in FIG. 12 is a product targeted by the recognition target object information, such as a CD in the CD case 20 in FIG. The character information is information on the object of the character CH in FIGS. The virtual space information is setting information of the background objects OB1 to OB11 shown in FIGS.

  In this way, in FIG. 12, information of a three-dimensional object such as character information or virtual space information is associated with the recognition target object information. Therefore, for example, when the recognition processing unit 114 in FIG. 1 performs recognition processing of a recognition target and the recognition target is specified, three-dimensional information such as character information and virtual space information associated with the specified recognition target is obtained. The object information can be read from the storage unit 170 (object information storage unit 172). Thereby, the virtual space setting unit 116 performs a virtual space setting process based on the read information of the three-dimensional object, and the image generation unit 120 generates a virtual space image that is an image of the set virtual space. By synthesizing with the captured image, an image as shown in FIGS. 6 and 7 can be generated.

  And in this embodiment, as shown in FIG. 12, the recognition target object is matched with goods, and the method of this embodiment is used for sales promotion of the goods. For example, in order to promote sales of a CD, a recognition object MK (or AR marker AMK) is printed on the cover of the CD case 20 as shown in FIG. Then, the user who has purchased the CD places the CD case 20 on the back side of the terminal device 10 as shown in FIG. By doing so, the image of the three-dimensional character CH corresponding to the two-dimensional character that is the recognition target object MK in FIG. 2 and the image of the background object are converted into a virtual space image as shown in FIGS. 10 display units 190. Therefore, the display of such a virtual space image is a motivation, so that the user can purchase the CD, and the sales promotion of the CD as the product can be promoted. Note that the product to which the recognition target object is associated is not limited to such a CD, and various products (DVD, book, portable game machine, home appliance, information terminal device, etc.) can be assumed.

2.2 Modifications Next, various modifications of the present embodiment will be described. In the present embodiment, a process of recognizing an image displayed on the display unit of the electronic device as a recognition target object may be performed.

  For example, in FIG. 13A, an image of the character CHG is displayed on the display unit 32 of the portable game machine 30 that is an electronic device. In this embodiment, the process of recognizing the image of the character CHG displayed on the display unit 32 as a recognition target object is performed. Then, a virtual space image composed of, for example, a three-dimensional character image corresponding to the character CH is generated, and an AR image as shown in FIGS. 6 and 7 is displayed. In this way, the user simply points the imaging unit 162 of the terminal device 10 such as a tablet PC or a smartphone toward the portable game machine 30, the display unit 190 of the terminal device 10 is displayed in FIGS. 6 and 7. An image as shown is displayed. In this case, the image of the character displayed in the virtual space image varies depending on the type of character CHG displayed on the portable game machine 30 and the like. By displaying such an AR image, it is possible to promote sales of the portable game machine 30 and the like.

  In FIG. 13B, a plurality of posters PS1 to PS6 are pasted on the wall at a venue such as a product exhibition or an event. Then, when the user US holding the smartphone 12 that is the terminal device points the imaging unit of the smartphone 12 toward the posters PS1 to PS6, the display unit of the smartphone 12 has a display as illustrated in FIGS. The AR image is displayed. In this case, the virtual space image displayed as the AR image is different for each of the posters PS1 to PS6. By doing in this way, sales promotion of goods, such as goods advertised by posters PS1-PS6, and concerts, etc. can be aimed at now.

  Moreover, in this embodiment, the process which recognizes the toy 40 as shown to FIG. 14 (A) and the article | item 46 as shown to FIG. 14 (B) as a recognition target object is performed. The article 46 in FIG. 14B is an example of a container such as a plastic bottle, but the present embodiment is not limited to this. In this embodiment, a virtual space image composed of, for example, an image of a three-dimensional object (such as a character) corresponding to the toy 40 or the article 46 is generated, and an AR image as shown in FIGS. 6 and 7 is displayed. Is done. In this way, the user simply points the imaging unit 162 of the terminal device 10 such as a tablet PC or a smartphone toward the toy 40 or the article 46, and displays the display unit 190 of the terminal device 10 in FIGS. An image as shown is displayed. In this case, the image of the three-dimensional object displayed in the virtual space image varies depending on the type, shape, attribute, or the like of the toy 40 or the article 46. For example, in the case of the toy 40, an image of a three-dimensional object corresponding to the type or shape of the toy 40 is displayed as a virtual space image. For example, a character image imitating the shape of the toy 40 may be displayed as a virtual space image. Alternatively, an image of a three-dimensional object such as a character corresponding to an image (such as an image printed on a container label) printed on a container that is the article 46 may be displayed as a virtual space image. By displaying such an AR image, sales promotion of the toy 40 or the article 46 can be promoted.

  In FIG. 15, the character CH does not go out of the opening OP but goes out of the area of the mask object. In this way, for example, in FIGS. 6 and 7, it is possible to realize an effect such that the three-dimensional character CH suddenly appears from the image region RMS around the opening. That is, it is possible to realize an effect such that a three-dimensional character CH suddenly jumps out of a space where nothing exists.

  In FIG. 16, mask objects (MS1 to MS6) are arranged so as to surround the user US wearing the HMD. Openings OP1 to OP10 are provided for the mask object, and the user US can see an image of the virtual space set outside the mask object via the openings OP1 to OP10. In this way, it becomes possible to provide the user US with an AR image having a display mode different from those shown in FIGS.

3. Detailed Processing Next, a detailed processing example of the present embodiment will be described with reference to the flowchart of FIG.

  First, a captured image is acquired (step S1). For example, a captured image captured by the imaging unit 162 is acquired. And the recognition process of the recognition target object reflected in a captured image is performed (step S2). For example, recognition processing of a recognition target object is performed by performing matching processing of feature points and contour shapes. Then, processing for setting the arrangement of the three-dimensional object and the mask object in the virtual space is performed (step S3). For example, as described with reference to FIG. 12, the information of the three-dimensional object corresponding to the recognized recognition target is read from the storage unit 170, and the virtual space arrangement setting process is performed. Then, a drawing process is performed according to the positional relationship information between the imaging unit that captures the captured image and the recognition target, a virtual space image is displayed in the image area of the opening, and an image area around the opening is An image on which the captured image is displayed is generated by masking the three-dimensional object with the mask object (step S4). For example, a virtual camera is set at a position corresponding to the imaging unit in the virtual space, and an image seen from the set virtual camera is generated as a virtual space image, thereby generating an AR image as shown in FIGS. .

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term (such as the first surface) described at least once together with a different term having a broader meaning or the same meaning (predetermined surface, etc.) in the specification or drawing is different from that term in any part of the specification or drawing. Can be replaced. The captured image acquisition process, recognition target object recognition process, virtual space arrangement setting process, image generation and synthesis process, mask object arrangement setting process, image drawing process, and the like are also described in this embodiment. The present invention is not limited, and techniques, processes, and configurations equivalent to these are also included in the scope of the present invention.

MK recognition object, AMK AR marker, US user,
OP, OP1-OP110 opening, MS1-MS6 plane, VC virtual camera,
CH, CHG character, OB1-OB11 background object,
Image area of ROP opening, image area around RMS opening,
PS1-PS6 poster, OP1-OP10 opening, VP viewpoint,
10 terminal devices, 12 smartphones, 20 CD cases,
30 portable game machines, 32 display units, 40 toys, 46 articles,
100 processing unit, 112 image acquisition unit, 114 recognition processing unit,
116 virtual space setting unit, 118 virtual camera setting unit,
120 image generation units, 122 image synthesis units, 130 sound generation units, 160 operation units,
162 imaging unit, 170 storage unit, 172 object information storage unit,
174 drawing buffer, 176 recognition object information storage unit,
180 information storage medium, 190 display unit, 192 sound output unit,
194 I / F unit, 195 portable information storage medium, 196 communication unit

Claims (15)

An image acquisition unit for acquiring a captured image;
A recognition processing unit that performs recognition processing of a recognition object reflected in the captured image;
A virtual space setting unit that performs processing for setting the arrangement of at least one three-dimensional object and a mask object arranged with a predetermined relationship to the three-dimensional object in the virtual space;
A virtual space image composed of the image of the three-dimensional object is displayed in the image area of the opening set corresponding to the location of the recognition object, and the image area around the opening includes the As an image generation unit that generates an image in which the captured image is displayed by masking a three-dimensional object with the mask object,
A program characterized by causing a computer to function. In claim 1,
The virtual space image is an image seen from a virtual camera in the virtual space,
The program according to claim 1, wherein the predetermined relationship is a relationship in which the three-dimensional object is located on the back side of the predetermined surface of the mask object as viewed from the virtual camera. In claim 1 or 2,
The virtual space setting unit
A program for arranging and setting the mask object having the first surface including the opening in the virtual space. In claim 3,
The virtual space setting unit
A program characterized in that the mask object having the first surface and at least one surface intersecting the first surface is arranged and set in the virtual space. In claim 4,
The program according to claim 1, wherein the predetermined relationship is a relationship in which the three-dimensional object is located in a region surrounded by the first surface and the at least one surface. In any one of Claims 1 thru | or 5,
The virtual space setting unit
A program characterized in that the three-dimensional object is arranged and set in the virtual space so that the three-dimensional object is located behind the opening. In any one of Claims 1 thru | or 6.
The virtual space setting unit
A program characterized in that the three-dimensional object is arranged and set in the virtual space so that the three-dimensional object straddles the opening. In any one of Claims 1 thru | or 7,
The image generation unit
A program characterized in that the virtual space image is generated by performing drawing processing according to positional relationship information between an imaging unit that captures the captured image and the recognition target object. In claim 8,
As a virtual camera setting unit that sets a virtual camera at a position corresponding to the imaging unit in the virtual space,
Make the computer work,
The image generation unit
A program for generating an image visible from the virtual camera as the virtual space image. In any one of Claims 1 thru | or 9,
The image generation unit
The program which performs the process which changes the visibility of the said virtual space image according to the positional relationship information of the imaging part which image | photographs the said captured image, and the said recognition target object. In any one of Claims 1 thru | or 10.
As a sound generation unit that performs a sound generation process that changes according to positional relationship information between the imaging unit that captures the captured image and the recognition target object,
A program characterized by causing a computer to function. In any one of Claims 1 thru | or 11,
Information on the three-dimensional object is stored in a storage unit in association with information on the recognition target object,
The virtual space setting unit
A program for arranging and setting the three-dimensional object associated with information of the recognition target in the virtual space. In any one of Claims 1 to 12,
The recognition processing unit
The program which performs the process which recognizes the image displayed on the display part of the electronic device as the said recognition target object. In any one of Claims 1 thru | or 13.
The recognition processing unit
A program for recognizing a toy or an article as the recognition target. An image acquisition unit for acquiring a captured image;
A recognition processing unit that performs recognition processing of a recognition object reflected in the captured image;
A virtual space setting unit that performs processing for setting the arrangement of at least one three-dimensional object and a mask object arranged with a predetermined relationship to the three-dimensional object in the virtual space;
A virtual space image composed of the image of the three-dimensional object is displayed in the image area of the opening set corresponding to the location of the recognition object, and the image area around the opening includes the An image generation unit that generates an image in which the captured image is displayed by masking a three-dimensional object with the mask object;
An image generation system comprising: