JP7418956B2 - Image generation system and program - Google Patents

Description

The present invention relates to an image generation system, a program, and the like.

In recent years, a technology called photogrammetry that generates three-dimensional models from captured images has been in the spotlight. In photogrammetry, for example, an analysis process is performed based on two-dimensional photographed images obtained by photographing an object, which is an object in the real world, from a plurality of photographing directions, to determine the shape, dimensions, etc. of the object. Then, based on the determined shape, dimensions, etc., a three-dimensional model that can be placed in virtual space is generated. Photogrammetry is also called 3D scanning. As a conventional technique of 3D scanning, for example, there is a technique disclosed in Patent Document 1. Furthermore, as a conventional technology of a photographing system that photographs an object from a plurality of photographing directions, there is a technique disclosed in Patent Document 2.

JP 2017-130008 Publication Japanese Patent Application Publication No. 2013-152332

In 3D scanning, a photographed image photographed by a photographing unit such as a camera is used, but there is a limit to the resolution of the photographed image. For this reason, when an object to be photographed has a fine shape, it is difficult to generate a three-dimensional model that faithfully reproduces this fine shape. In addition, analysis processing based on photographed images of a target object requires a very heavy processing load on the computer, so for example, after photographing the target object, a three-dimensional model display corresponding to the target object can be created in a short period of time. It is difficult to make it appear in virtual space, which is a three-dimensional space.

According to the present disclosure, it is possible to provide an image generation system, a program, and the like that can make a display object corresponding to a target object appear in a virtual space while faithfully expressing the shape of the target object.

One aspect of the present invention includes an acquisition unit that acquires a group of photographed images obtained by photographing an object from a plurality of photographing directions by a photographing unit, a storage unit that stores the acquired group of photographed images, and a movement control unit. a movement processing unit that performs a process of moving a primitive in a virtual space based on the information; and a texture that is mapped to the primitive based on a photographed image selected from the group of photographed images according to the movement control information of the primitive. a texture generation unit that generates a texture, and an image generation unit that maps the texture that changes according to the movement control information onto the primitive to generate a virtual space image that can be seen from a virtual camera in the virtual space. Related to the generation system. The present invention also relates to a program that causes a computer to function as each of the above sections, or a computer-readable information storage medium that stores the program.

According to one aspect of the present invention, a group of photographed images is obtained by photographing an object from a plurality of photographing directions, and a photographed image according to movement control information of a primitive is selected from this group of photographed images. The texture generated based on the selected captured image is then mapped onto the primitive that moves based on the movement control information. As a result, the texture that changes according to the movement control information is mapped onto the primitive, and a virtual space image that can be seen from the virtual camera in the virtual space is generated. Therefore, a texture image corresponding to the movement control state of the primitive is mapped onto the primitive, and a virtual space image including an image of the display object corresponding to the target object can be generated. This makes it possible to provide an image generation system or the like that can make a display object corresponding to a target object appear in a virtual space while faithfully expressing the shape of the target object.

Further, in one aspect of the present invention, the movement processing unit performs a process of moving the primitive and the virtual camera according to the movement control information while the positional relationship between the primitive and the virtual camera is fixed. Good too.

In this way, it becomes possible to generate an image that appears to be a three-dimensional image in a pseudo manner as an image of the displayed object by mapping the texture to the primitive.

Further, in one aspect of the present invention, the photographing unit photographs the target object and the background of the target, and the texture generating unit determines that the color of each pixel of the photographed image corresponds to the color of the background. If it is determined that the pixel is a color that corresponds to the color of the background, processing is performed to make the pixel of the color that corresponds to the background color transparent. Textures may also be generated.

In this way, an appropriate virtual space image in which other objects in the virtual space are displayed in the background can be generated by simple processing with low processing load.

Further, in one aspect of the present invention, the texture generation unit selects the photographed image from the group of photographed images by performing seek control of a moving image composed of the group of photographed images based on the movement control information, The texture may be generated based on the selected captured image.

In this way, by effectively utilizing the seek control of a video composed of a group of captured images, it is possible to select an appropriate captured image corresponding to the movement control information of the primitive, generate a texture, and map it to the primitive. Become.

Further, in one aspect of the present invention, the group of photographed images may be a group of images in which a change in the photographing direction of the photographing unit with respect to the target object is within a photographing direction range of 180 degrees or more.

In this way, even if the primitive moves in various directions, a corresponding texture based on an appropriate photographed image can be generated and mapped to the primitive.

Further, in one aspect of the present invention, the movement processing unit performs movement processing of a second primitive in which a relative positional relationship with respect to the primitive changes according to the movement control information, and the image generation unit An image including an image of the mapped primitive and an image of the second primitive may be generated as the virtual space image.

In this way, an image of an object that does not exist in the real world can be displayed using the image of the second primitive.

Further, in one aspect of the present invention, the movement processing unit performs a process of moving the second primitive from an initial placement position of the second primitive set based on the analysis result of the target object, using the primitive as a reference. You may do so.

In this way, the second primitive can be placed at an appropriate initial placement position according to the analysis result of the target object, and when the primitive moves according to the movement control information, it can be placed at an appropriate position linked to the movement. , it becomes possible to display an image based on the second primitive.

Further, in one aspect of the present invention, the image generation unit may perform hidden surface removal processing on the second primitive in accordance with the movement control information.

By performing the hidden surface removal process on the second primitive in this manner, the image formed by the second primitive becomes hidden from the virtual camera, making it possible to generate an appropriate virtual space image.

Further, in one aspect of the present invention, the image of the second primitive may be an effect image.

In this way, an image with an effect that does not exist on objects in the real world can be displayed using the second primitive.

Further, in one aspect of the present invention, the image generation unit generates, as the virtual space image, an image including an image of the primitive and images of a plurality of objects set in the virtual space, and Post-effect processing may be performed on the virtual space image.

By performing such post-effect processing, it becomes possible to generate a virtual space image in which the character image generated based on the captured images of the captured image group blends seamlessly into the surrounding virtual space. .

Further, in one aspect of the present invention, the acquisition unit acquires a group of photographed images for the other display object obtained by photographing the other target object corresponding to the other display object from a plurality of photographing directions, and The texture generation unit generates a texture for another display object based on a photographed image selected from a group of photographed images for the other display object according to movement control information for the other display object, and generates a texture for the other display object. The generation unit may generate the virtual space image by mapping a texture for the other display object that changes according to movement control information for the other display object onto a primitive for the other display object. good.

In this way, in the same way as the display object corresponding to the target object, the virtual space image that looks as if the other display object corresponding to the other display object has appeared in the virtual space. will be able to generate.

Further, in one aspect of the present invention, the image generation unit may generate an image in which particles are displayed around the primitive as the virtual space image.

By displaying such particles, when the primitive moves in the virtual space, the surrounding particles become a factor, creating a three-dimensional spatial feeling.

Further, in one aspect of the present invention, rotational movement is performed for changing the photographing direction of the photographing section, the placement section where the object is placed, and the photographing direction of the photographing section with respect to the object placed in the placement section. A rotational movement mechanism may also be included.

By providing such an arrangement section and rotational movement mechanism, it becomes possible to obtain a group of photographed images in which the photographing direction of the photographing section with respect to the object falls within a photographing direction range of a predetermined angle or more.

Further, one aspect of the present invention may include a support portion that supports the object while floating in the air.

In this way, it becomes possible to make a display object that corresponds to the target object and appear floating in the air in the virtual space.

FIG. 1 is a block diagram showing a configuration example of an image generation system according to the present embodiment. FIG. 2(A) and FIG. 2(B) are explanatory diagrams of a method of generating a virtual space image based on a captured image. FIGS. 3(A) and 3(B) are explanatory diagrams of a method for acquiring photographed images of a target object taken from a plurality of photographing directions. FIG. 4(A) and FIG. 4(B) are explanatory diagrams of a method of generating texture from a photographed image. An explanatory diagram of the relationship between primitives and virtual cameras. FIG. 4 is an explanatory diagram of a texture mapping method that changes according to movement control information. An explanatory diagram of an effect image generation method. 5 is a flowchart of virtual space image generation processing based on a group of photographed images. An explanatory diagram of a method of photographing an object in a 360-degree photographing direction range. FIG. 3 is an explanatory diagram of a method for selecting a photographed image according to movement control information from a group of photographed images. FIGS. 11(A) to 11(C) are explanatory diagrams of a method of moving a primitive and a virtual camera with their positional relationship fixed. A flowchart of the process of making pixels of a color corresponding to the background color transparent. FIGS. 13(A) to 13(C) are explanatory diagrams of a method of generating an effect image using primitives for an effect image. FIG. 7 is an explanatory diagram of a method for setting the initial placement position of a primitive for an effect image. FIG. 3 is an explanatory diagram of post-effect processing for a virtual space image. FIG. 7 is an explanatory diagram of a method of generating an image of another display object corresponding to another target object. An explanatory diagram of a method for arranging particles around primitives. A configuration example of an image generation system including an imaging section, a placement section, and a rotational movement mechanism. An example of a support part that supports an object while it is suspended in the air. 5 is a flowchart of virtual space image generation processing based on the first and second photographed image groups. FIG. 21(A) and FIG. 21(B) are explanatory diagrams of a method of acquiring a second group of photographed images by photographing an object whose pose is changed. FIGS. 22(A) and 22(B) are explanatory diagrams of a method of acquiring a second group of photographed images by photographing an object to which accompanying objects have been added or an object whose shape has been deformed. FIG. 3 is an explanatory diagram of a method of acquiring a second group of captured images by processing a first group of captured images. FIG. 7 is an explanatory diagram of a method for generating a virtual space image by selecting one of the first and second photographed image groups according to operation input information of the player. 12 is a flowchart of a process of selecting one of the first and second photographed image groups according to the analysis result of the target object and the player's operation input information to generate a virtual space image. FIG. 7 is an explanatory diagram of a method of generating a texture based on a photographed image selected from one of the first and second photographed image groups according to movement control information. 27(A) and 27(B) are explanatory diagrams of a method of generating a texture by combining captured images of the first and second captured image groups. 5 is a flowchart of processing for setting game parameters and executing game processing based on the analysis results of the target object. 7 is a flowchart of processing for storing a second group of captured images different from the first group of captured images in a storage unit. FIG. 7 is an explanatory diagram of editing processing of photographed images when generating a texture from the first and second photographed image groups.

This embodiment will be described below. Note that this embodiment described below does not unduly limit the contents described in the claims. Furthermore, not all of the configurations described in this embodiment are essential configuration requirements.

1. Image Generation System FIG. 1 is a block diagram showing a configuration example of an image generation system (image generation device, game device, terminal device) of this embodiment. Note that the image generation system of this embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of its components or adding other components are possible.

The operation unit 160 is used by the player (user) to input various operation input information (input information). The operation unit 160 can be realized by, for example, an operation button, a direction key, a joystick, a lever, a touch panel display, or the like.

The photographing unit 162 photographs the object, and is realized by an optical system including an image sensor such as a CCD or a CMOS sensor, and a focus lens. As the photographing unit 162, for example, a camera such as a digital camera or a video camera can be used. The physical object is, for example, an object in the real world, and is a subject of the photographing unit 162.

The storage unit 170 stores various information. The storage unit 170 functions as a work area for the processing unit 100, the communication unit 196, and the like. Programs and data necessary for executing the programs are held in this storage unit 170. The functions of the storage unit 170 can be realized by semiconductor memory (DRAM, VRAM), HDD (hard disk drive), SSD (Solid State Drive), optical disk device, or the like. The storage unit 170 includes a photographed image information storage unit 176, a virtual space information storage unit 177, and a drawing buffer 178. The photographed image information storage section 176 stores information about photographed images photographed by the photographing section 162. For example, information about a group of photographed images is stored. The virtual space information storage unit 177 stores information about a virtual space that is a three-dimensional object space. For example, it stores information about objects placed in virtual space. The drawing buffer 178 is a buffer such as a frame buffer or a work buffer that can store image information in units of pixels.

The information storage medium 180 is a computer-readable medium that stores programs, data, and the like. The information storage medium 180 can be realized by an optical disk (DVD, BD, CD), an HDD, a semiconductor memory (ROM), or the like. The processing unit 100 performs various processes of this embodiment based on programs (data) stored in the information storage medium 180. That is, the information storage medium 180 contains a program (a program for causing the computer to execute the processing of each part) for making a computer (a device including an input device, a processing part, a storage part, and an output part) function as each part of this embodiment. is memorized.

The display unit 190 outputs the image generated according to the present embodiment, and its function can be realized by an LCD, an organic EL display, a CRT, a touch panel display, an HMD (Head Mounted Display), or the like. The sound output unit 192 outputs the sound generated according to this embodiment, and its function can be realized by a speaker, headphones, or the like.

The I/F (interface) section 194 performs interface processing with the portable information storage medium 195, and its function can be realized by an ASIC for I/F processing. The portable information storage medium 195 is used by the player to store various types of information, and is a storage device that retains the storage of this information even when power is cut off. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

The communication unit 196 communicates with the outside (other devices) via a wired or wireless network, and its function is based on hardware such as a communication ASIC or communication processor, and communication firmware. This can be achieved by

Note that the program (data) for making the computer function as each part of this embodiment is distributed from the information storage medium of the server (host device) to the information storage medium 180 (or storage unit 170) via the network and communication unit 196. You may. The use of information storage media by such servers may also be included within the scope of this embodiment.

The processing unit 100 (processor) performs game processing, editing processing, acquisition processing, analysis processing, movement processing, texture generation processing, image generation processing, or sound generation processing based on operation input information and programs from the operation unit 160. etc.

Each process of this embodiment performed by each part of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of this embodiment can be realized by a processor that operates based on information such as a program, and a memory that stores information such as a program. For example, the functions of each part of the processor may be realized by separate hardware, or the functions of each part may be realized by integrated hardware. For example, a processor includes hardware, and the hardware can include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, a processor can be configured with one or more circuit devices (eg, an IC) mounted on a circuit board, or one or more circuit elements (eg, a resistor, a capacitor, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to a CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. Further, the processor may be a hardware circuit based on an ASIC. Further, the processor may include an amplifier circuit, a filter circuit, etc. that process analog signals. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk device. For example, the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processing of each section of the processing section 100 is realized. The instructions here may be an instruction set that constitutes a program, or instructions that instruct a hardware circuit of a processor to operate.

The processing section 100 includes a game processing section 106 , an editing processing section 108 , an acquisition section 110 , an analysis section 111 , a movement processing section 112 , a texture generation section 114 , an image generation section 120 , and a sound generation section 130 . As described above, each process of this embodiment executed by each of these units can be realized by a processor (or a processor and a memory). Note that various modifications such as omitting some of these components (each part) or adding other components are possible.

The game processing unit 106 performs various game processing for the player to play the game. The game processing includes, for example, processing to start the game when game start conditions are met, processing to progress the started game, processing to end the game when game end conditions are met, or calculation of game results. processing, etc.

The editing processing unit 108 performs editing processing for the player to edit the captured image. The acquisition unit 110 performs a process of acquiring a photographed image photographed by the photographing unit 162. For example, it performs a process of receiving and reading out information about a photographed image from the photographing unit 162. The analysis unit 111 performs analysis processing on the target object. For example, analysis processing such as silhouette analysis of the object is performed based on the photographed image of the object.

The movement processing unit 112 performs processing to move objects such as primitives in virtual space. For example, in this embodiment, a process for setting a virtual space (object space) in which a plurality of objects are arranged is performed. For example, various types of objects representing moving objects (people, robots, cars, trains, airplanes, ships, monsters, animals, etc.), maps (terrain), buildings, spectator seats, courses (roads), trees, walls, water surfaces, etc. A process of arranging and setting an object (an object composed of primitives such as a polygon, a free-form surface, or a subdivision surface) in a virtual space is performed. In other words, the position and rotation angle (synonymous with orientation and direction) of the object in the world coordinate system are determined, and the position (X, Y, Z) and rotation angle (rotation angle around the X, Y, Z axes) are The object is placed. Specifically, the virtual space information storage unit 177 of the storage unit 170 stores object information, which is information such as the position, rotation angle, moving speed, and moving direction of an object in the virtual space, as virtual space information, including object numbers. are stored in association with. For example, a process of updating object information, which is virtual space information, for each frame is performed. The movement processing unit 112 then performs a process of moving objects such as primitives in the virtual space set in this way. For example, it performs processing to update information on objects such as primitives. It also performs motion processing to make objects move.

The texture generation unit 114 performs a process of generating a texture to be mapped to a primitive. For example, processing is performed to generate a texture based on the captured image acquired by the acquisition unit 110.

The image generation unit 120 performs image generation processing. For example, a drawing process is performed based on the results of various processes performed by the processing unit 100, thereby generating an image and displaying it on the display unit 190. Specifically, geometry processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed, and based on the processing results, drawing data (positions of vertices of primitive surfaces) (coordinates, texture coordinates, color data, normal vectors, α values, etc.) are created. Then, based on this drawing data (primitive surface data), the object (one or more primitive surfaces) after perspective transformation (after geometry processing) is drawn in the drawing buffer 178. As a result, a virtual space image visible from the virtual camera in the virtual space is generated. Note that the drawing process performed by the image generation unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

The sound generation unit 130 performs sound generation processing based on the results of various processes performed by the processing unit 100. Specifically, music (music, BGM), sound effects, voices, etc. are generated and output to the sound output unit 192.

The image generation system of this embodiment includes an acquisition section 110, a storage section 170, a movement processing section 112, a texture generation section 114, and an image generation section 120, as shown in FIG.

The acquisition unit 110 acquires a group of photographed images obtained by photographing the object from a plurality of photographing directions using the photographing unit 162. The photographed image group may be, for example, a plurality of still images photographed by a digital camera, or a plurality of images constituting a moving image photographed by a digital camera or a video camera. In this case, the photographed image group is, for example, an image group constituting a moving image encoded by MPEG or the like. That is, the photographed image group may be a movie video such as MPEG. Note that the image generation system of this embodiment may not include the photographing section 162; in this case, the acquisition section 110 acquires a photographed image via the I/F section 194 and the communication section 196.

The storage unit 170 stores the acquired photographed image group. For example, when the acquisition unit 110 acquires a group of photographed images, information on the acquired group of photographed images is saved and stored in the storage unit 170. Specifically, the photographed image information storage section 176 of the storage section 170 stores information on a group of photographed images.

The movement processing unit 112 performs a process of moving the primitive in the virtual space based on the movement control information. For example, based on the movement control information, processing is performed to translate or rotate the primitive in the virtual space. The movement control information is information for controlling movement of primitives. As the movement control information, operation input information input by the player using, for example, the operation unit 160 can be used. Alternatively, movement control information for controlling a series of movements of primitives may be stored in the storage unit 170, and the movement of the primitives may be controlled by reading the stored movement control information from the storage unit 170. In this way, it is possible to realize movement control in which the primitive automatically moves according to the movement control information. As a result, it is possible to realize, for example, an image generation system that allows viewing of a display object (for example, a fish, an animal, etc.) corresponding to the target object.

The texture generation unit 114 generates a texture to be mapped to the primitive based on the captured image selected from the captured image group (one of the first and second captured image groups) according to the movement control information of the primitive. Perform the process to generate. For example, based on the movement control information of the primitive, a photographed image corresponding to the movement control information is read from the storage unit 170 from the photographed image group (one of the first and second photographed image groups). The texture generation unit 114 then generates a texture based on the read captured image. For example, in a photographed image of a target object, a portion of the target object is generated as a texture image. Specifically, the texture is generated by making the background image part of the photographed image transparent and cutting out the object image part. By mapping the texture to a primitive, it becomes possible to display an image of a display object corresponding to the target object. Here, the primitive is, for example, a primitive surface, such as a polygon. Specifically, the primitive is a billboard polygon placed directly facing the virtual camera. Note that the primitive may be a free-form surface, a subdivision surface, or the like.

The image generation unit 120 maps a texture that changes according to movement control information onto a primitive to generate a virtual space image that can be seen from a virtual camera in a virtual space. For example, by mapping a texture based on a photographed image selected from a group of photographed images according to movement control information onto a primitive, the texture mapped onto the primitive changes according to the movement control information of the primitive. That is, from among a group of captured images obtained by photographing an object from a plurality of photographing directions, a captured image that matches the movement control state of the primitive is selected, and a texture generated based on the selected captured image is Map to primitives. By doing so, a texture image corresponding to the movement control state of the primitive is mapped onto the primitive, and a virtual space image in which a display object corresponding to the target object is displayed can be generated.

Furthermore, the movement processing unit 112 performs a process of moving the primitive and the virtual camera according to the movement control information while the positional relationship between the primitive and the virtual camera is fixed. For example, the positional relationship between the primitive and the virtual camera is fixed so that the distance between the primitive and the virtual camera is a constant distance. For example, the positional relationship between the primitive and the virtual camera is fixed so that the primitive faces the virtual camera and the distance between the primitive and the virtual camera is a constant distance. The state in which the primitive directly faces the virtual camera is, for example, a state in which the viewing direction of the virtual camera is orthogonal to the surface of the primitive. Then, the movement processing unit 112 performs a process of moving the primitive and the virtual camera together, with the positional relationship between the primitive and the virtual camera being fixed in this manner. For example, if the movement control information is to translate upward, downward, left, or right in the virtual space, the movement processing unit 112 performs processing to translate the primitive and the virtual camera upward, downward, left, or right. Alternatively, if the movement control information is rotational movement around the X-axis, Y-axis, or Z-axis in the virtual space, the movement processing unit 112 rotates the primitive and the virtual camera around the X-axis, Y-axis, or Z-axis. Perform the processing to

Further, the photographing unit 162 photographs the target object and the background of the target object. The imaging unit 162 may be included in the image generation system, or may be provided separately from the image generation system. The texture generation unit 114 determines whether the color of each pixel of the photographed image corresponds to the color of the background. If it is determined that the pixel has a color corresponding to the background color, a texture is generated by making the pixel of the color corresponding to the background transparent. For example, the texture generation unit 114 determines, for each pixel of the photographed image, whether the color of each pixel corresponds to the background. For example, when the photographing unit 162 photographs an object, the background color is set to a predetermined color such as green or blue. For example, the background color is set to a predetermined color different from the color of the object. For example, if the object has multiple colors, the background color is set to a predetermined color different from any of the multiple colors. For example, when photographing, a background object placed around an object is set to a predetermined color. It is desirable that the background color be set to a color that is complementary to the color of the object. When the texture generation unit 114 determines that the pixel to be processed is a pixel of a color corresponding to the background (for example, green or blue), the texture generation unit 114 performs processing to make the pixel transparent. For example, in alpha compositing, which is semi-transparent processing, by setting the alpha value of the pixel to 0, the pixel becomes a transparent pixel. Note that the above-described processing for each pixel can be realized using a pixel shader or the like.

Furthermore, the texture generation unit 114 selects a captured image from the captured image group and generates a texture based on the selected captured image by performing seek control of the moving image composed of the captured image group based on the movement control information. do. For example, seek control is performed to fast forward or rewind a moving image (movie) of a group of photographed images, and a photographed image corresponding to movement control information is sought and selected. A texture is then generated based on the selected photographed image. The texture generated in this way is mapped to a primitive that moves translationally or rotationally according to the movement control information.

Further, the photographed image group (the first and second photographed image groups) is a group of images in which the photographing direction of the photographing unit 162 relative to the object changes within a photographing direction range of 180 degrees or more. Preferably, as shown in FIG. 9, which will be described later, the photographed image group is a group of images in which the photographing direction of the photographing unit 162 with respect to the object changes within a photographing direction range of 360 degrees. For example, the photographing direction of the photographing unit 162 is directed toward the target object, and the photographing direction is set in the direction in which the target object is gazed. The photographing direction range is a range in which the angle of the photographing direction changes from 0 degrees to β while gazing at the object. In this embodiment, β is set to 180 degrees or more, and preferably β=360 degrees as shown in FIG.

The movement processing unit 112 also performs movement processing of the second primitive in which the relative positional relationship with respect to the primitive changes according to the movement control information. For example, a movement process such as rotationally moving the second primitive based on the primitive is performed. The image generation unit 120 then generates, as a virtual space image, an image that includes the image of the primitive to which the texture has been mapped and the image of the second primitive. For example, the primitive is for displaying an image of a display object corresponding to the target object, and a texture based on a photographed image of the target object is mapped to the primitive. On the other hand, the second primitive is for displaying, for example, an image of an accompanying object of the display object corresponding to the target object. The accompanying object is, for example, an effect of the displayed object, an item in your possession, an accessory, an equipment, a weapon, or an armor, and the image of the second primitive is an effect of the object, an item in your possession, an accessory, an equipment, etc. , images of weapons, armor, etc. The second primitive may be a polygon such as a billboard polygon, a free-form surface, a subdivision surface, or the like. Alternatively, the second primitive may constitute a three-dimensional object representing an appendage of the display object. By using such a second primitive, images of effects, belongings, accessories, equipment, weapons, armor, etc. that do not exist on objects in the real world can be added to the display objects corresponding to the objects and displayed. become able to.

Furthermore, the movement processing unit 112 performs a process of moving the second primitive from the initial placement position of the second primitive set based on the analysis result of the target object, using the primitive as a reference. For example, a process of moving a second primitive from an initial placement position with reference to the primitive is performed based on the movement control information. For example, the analysis unit 111 performs analysis processing on a target that is an object in the real world. For example, the analysis unit 111 performs analysis processing such as silhouette analysis of the object to identify the type, shape, etc. of the object. The movement processing unit 112 then arranges the second primitive at an initial arrangement position according to the type and shape of the object, and rotates or translates the second primitive from the initial arrangement position with respect to the primitive. Perform the moving process. For example, if the object is of the first type or shape, the second primitive is placed at the first initial placement position and the movement process is performed, and the object is of the second type or shape. If it is a shape, the second primitive is placed at the second initial placement position and the movement process is performed.

The image generation unit 120 also performs hidden surface removal processing on the second primitive in accordance with the movement control information. That is, the second primitive rotates or translates in accordance with the movement control information so that the relative positional relationship with respect to the primitive changes. Then, when it is determined based on the movement control information that the accompanying object represented by the second primitive has reached a position where it is hidden from the virtual camera with respect to the display object represented by the primitive, the second primitive Perform hidden surface removal processing. Then, hidden surface removal processing is performed to make the accompanying object represented by the second primitive appear hidden by the display object represented by the primitive. By doing so, it becomes possible to display images of the display object and its accompanying objects without contradiction.

Further, the second primitive image is, for example, an effect image. For example, the second primitive image is an image of an effect (game effect) that is a special graphic representing light, flame, smoke, jetting, flashing, explosion, weapon attack, magic attack, or the like. By using the effect image as the second primitive image in this way, it becomes possible to generate effects that cannot be generated on objects in the real world in the virtual space that is the game space.

The image generation unit 120 also generates, as a virtual space image, an image that includes an image of a primitive and images of a plurality of objects set in the virtual space. Post-effect processing is then performed on the generated virtual space image. For example, post-effect processing such as blur processing, brightness adjustment processing, or hue adjustment processing is performed. Specifically, the image generation unit 120 performs post-effect processing on the virtual space image in which other objects are displayed so that the display object represented by the primitive is displayed without any discomfort. For example, post-effect processing is performed to blur the outline of a display object represented by a primitive, so that the outline that is the boundary between the display object and the virtual space image becomes less noticeable. Furthermore, if there is a difference between the brightness of the displayed object and the brightness of the surrounding virtual space image, the image generation unit 120 performs post-effect processing to reduce this difference. For example, a texture mapped to a primitive is generated based on a photographed image obtained by photographing an object in the real world. For this reason, the brightness of the texture image changes depending on the shooting environment, resulting in a difference between the brightness of the image of the displayed object and the brightness of the virtual space image around the displayed object. The image generation unit 120 performs post-effect processing to reduce the difference.

The acquisition unit 110 also acquires a group of photographed images for other display objects obtained by photographing other objects corresponding to the other display objects from a plurality of photographing directions. That is, a group of photographed images for other display objects, which are images of other objects corresponding to other display objects different from the display object corresponding to the primitive, is obtained. For example, when a display object represented by a primitive is a character, other display objects may be, for example, an enemy character that fights with the character. The texture generation unit 114 then generates a texture for another display object based on a photographed image selected from a group of photographed images for another display object according to movement control information for another display object. The image generation unit 120 then maps the texture for the other display object that changes according to the movement control information for the other display object onto the primitive for the other display object to generate a virtual space image. For example, the movement processing unit 112 performs a process of moving a primitive for another display object in the virtual space in accordance with movement control information for another display object. Then, to the primitive for the other display object that moves in this way, a texture for the other display object that changes according to the movement control information for the other display object is mapped. This makes it possible to generate realistic images of other display objects, such as enemy characters, with less processing load.

The image generation unit 120 also generates an image in which particles are displayed around the primitive as a virtual space image. For example, in virtual space, particles are generated so that they are arranged around primitives to generate a virtual space image. For example, the image generation unit 120 performs effect processing (post-effect processing) to generate particles. The presence of particles around the primitive in this manner makes it possible to generate a virtual space image in which a display object corresponding to the primitive moves within the virtual space without causing any discomfort.

Further, as shown in FIG. 18, which will be described later, the image generation system of this embodiment includes a photographing unit 162 such as a camera, a placement unit 10 (placement unit) in which the object POB is placed, and a rotational movement mechanism 12. be able to. The rotational movement mechanism 12 is a mechanism that performs rotational movement to change the photographing direction of the photographing section 162 with respect to the object POB placed in the placement section 10. In FIG. 18, the rotational movement mechanism 12 realized by a drive unit such as a motor rotates the placement unit 10 like a turntable, thereby changing the imaging direction of the imaging unit 162 with respect to the object POB. By providing such an arrangement section 10 and rotational movement mechanism 12, it becomes possible to obtain a group of photographed images in which the photographing direction of the photographing section 162 with respect to the object POB falls within a photographing direction range of a predetermined angle or more.

Further, the image generation system of this embodiment may include a support section that supports the object while floating in the air. For example, as shown in FIG. 19, which will be described later, by supporting the object POB with the support section 16, the object POB can be photographed by the photographing section 162 while floating in the air. For example, with the legs of the object POB separated from the arrangement section 10, the object POB is supported by the support section 16, and the photographing section 162 photographs the object POB from a plurality of photographing directions to create a group of photographed images. Let's get it. In this way, for example, a plastic model such as a robot that appears as a flying character in an anime or the like can be photographed as an object POB and made to appear in a virtual space.

The acquisition unit 110 also acquires a first group of captured images and a second group of captured images. Note that the number of captured image groups to be acquired is not limited to two, and three or more captured image groups may be acquired. That is, the acquisition unit 110 can acquire the first to Kth photographed image groups (K is an integer of 2 or more). The first group of photographed images is a group of photographed images obtained by photographing the object from a plurality of photographing directions by the photographing unit 162. On the other hand, the second group of photographed images is a group of images obtained by photographing an object with a changed pose, an object with an accompanying object added, or a target whose shape has been deformed from a plurality of photographing directions. For example, by photographing an object (second object) whose pose (posture) of the object in the first photographed image group has been changed by the photographing unit 162 from a plurality of photographing directions, the second photographed image group is obtained. get. Alternatively, an object (second object) in which incidental items such as weapons, armor, equipment, accessories, or belongings are added (equipped or possessed) to the object in the first photographed image group, A second group of photographed images is obtained by photographing from a plurality of photographing directions by the photographing unit 162. Alternatively, the second group of photographed images is obtained by photographing an object (second object) obtained by deforming the object in the first group of photographed images from a plurality of photographing directions using the photographing unit 162. For example, the object is set to a transformation state in which it transforms into different shapes, and images are taken from a plurality of photographing directions to obtain a second group of photographed images. Alternatively, a group of images obtained by processing the first group of captured images may be acquired as the second group of captured images. For example, the first group of captured images may be processed to change the size such as reduction or enlargement, or the first group of captured images may be subjected to image effect processing such as blurring, or the first group of captured images may be A second group of photographed images is obtained by performing processing to change the color, brightness, or contrast of the first group of photographed images, or by performing processing such as horizontally flipping, vertically flipping, or rotating the first group of photographed images. do. The storage unit 170 (captured image information storage unit 176) stores the first and second captured image groups acquired in this manner, and the movement processing unit 112 performs processing to move the primitive in the virtual space. . Then, the texture generation unit 114 generates a texture to be mapped to the primitive from the first and second captured image groups, and the image generation unit 120 maps the generated texture to the primitive to create a virtual camera in the virtual space. Generate a virtual space image that can be seen from For example, the texture generation unit 114 generates a texture based on a photographed image obtained by processing a predetermined procedure from the first and second photographed image groups. Then, the image generation unit 120 creates a virtual space image in which a display object corresponding to the target object is displayed by mapping the texture based on the photographed image obtained by processing a predetermined procedure onto a primitive that moves within the virtual space. generate.

The texture generation unit 114 also selects one of the first and second photographed image groups in accordance with the player's operation input information. Then, a texture is generated based on the photographed images of one of the selected photographed image groups. For example, operation input information input by the player using the operation unit 160 is acquired, and one of the first and second photographed image groups is selected based on this operation input information. For example, if it is determined that the player has performed the first operation based on the operation input information, the first group of captured images is selected, and the map is mapped to the primitive based on the captured images of the first group of captured images. Generate a texture. On the other hand, if it is determined that the player has performed the second operation, a second group of captured images is selected, and a texture to be mapped to the primitive is generated based on the captured images of the second group of captured images. . In this way, a photographed image is selected from among the first and second photographed image groups corresponding to the player's operation input information, a texture is generated, and a display object corresponding to the target object is generated. It can be mapped to primitives for display. Therefore, it is possible to cause a display object to appear in the virtual space, to which a photographed image appropriate according to the player's operation input information is mapped.

The texture generation unit 114 also selects one of the first and second photographed image groups according to the analysis result of the object and the player's operation input information, and selects one of the selected photographed image groups. A texture is generated based on the captured image. For example, the analysis unit 111 performs analysis processing such as silhouette analysis of the object to identify the type, shape, etc. of the object. Then, the texture generation unit 114 selects one of the first and second photographed image groups based on the type and shape of the object, which is the analysis result of the object, and the player's operation input information. do. For example, if the analysis process analyzes that the object is of the first type or the first shape, and it is determined that the player has performed the first operation based on the operation input information, then the first A group of captured images is selected, and a texture to be mapped to the primitive is generated based on the captured images of the first group of captured images. On the other hand, if the analysis process analyzes that the object is of the second type or shape, and it is determined that the player has performed the second operation based on the operation input information, then the second A group of captured images is selected, and a texture to be mapped to the primitive is generated based on the captured images of the second group of captured images. By doing so, it becomes possible to cause a display object to appear in the virtual space, on which an appropriate captured image is mapped according to both the analysis result of the target object and the player's operation input information.

The texture generation unit 114 also generates a texture by combining the captured images of the first captured image group and the captured images of the second captured image group. For example, a composite image is generated by performing a composition process such as α composition of the captured images of the first and second captured image groups, a texture is generated based on the composite image, and it is mapped to a primitive corresponding to the display object. . For example, after mapping a texture based on a captured image of a first captured image group onto a primitive, a texture based on a composite image of captured images of the first and second captured image groups is mapped onto a primitive. After that, textures based on the captured images of the second captured image group are mapped onto the primitives. In this way, it is possible to smoothly switch from the image of the displayed object based on the photographed images of the first photographed image group to the image of the displayed object based on the photographed images of the second photographed image group. It becomes possible to implement motion interpolation processing, etc.

Furthermore, the game processing unit 106 sets game parameters for the game played by the player based on the analysis result of the object, and performs game processing based on the set game parameters. For example, the type, shape, etc. of the object are specified based on the analysis results of the object. Then, based on the type, shape, etc. of the object, game parameters of a display object such as a character corresponding to the object are set. For example, game parameters such as attack power, defense power, durability, movement ability, running ability, special ability, special skill, hit points, magic points, or level are set. Based on the set game parameters, a game in which a display object such as a character appears is processed. For example, a game start process, a game progress process, a game end process, a game score calculation process, etc. are performed.

Further, when receiving the second group of captured images after receiving the first group of captured images, the acquisition unit 110 determines whether the second group of captured images is a different group of captured images from the first group of captured images. to judge. If it is determined that they are different groups of captured images, the second group of captured images is stored in the storage unit 170. That is, when the second group of captured images is received, it is determined whether the second group of captured images is a different group of captured images from the first group of captured images that were captured in the past. Then, when the second group of captured images is a different group of captured images from the first group of captured images, the acquisition unit 110 stores and saves the second group of captured images in the storage unit 170. For example, if the object shown in the second group of captured images is an object whose pose has been changed, an object has been added with an accompanying object, or an object whose shape has been deformed, the second group of captured images may be different from the first group of captured images. It is determined that the captured image group is different from the captured image group of . Even when a third group of captured images is received, it is determined whether the third group of captured images is a different group of captured images from the first group of captured images or the second group of captured images, In some cases, the third captured image group is stored and saved in the storage unit 170. Whether the captured images are different groups can be determined based on, for example, the shape or color of the object appearing in the captured images.

In addition, the editing processing unit 108 performs editing processing to accept editing by the player on the photographed images when generating textures from the first and second photographed image groups. For example, in this embodiment, a texture is generated based on the captured images of the first and second captured image groups, and when the texture is generated, the player can edit the captured images. For example, the player can perform editing such as correcting a photographed image, adding additional display objects such as marks and icons to the photographed image, and transforming the photographed image. A texture is then generated based on the edited captured image and mapped onto a primitive that moves in virtual space.

2. Method of this embodiment Next, the method of this embodiment will be described in detail.

2.1 Overview First, an overview of the method of this embodiment will be explained. A method called photogrammetry has been known for some time, and with this photogrammetry, objects in the real world are scanned in 3D based on images taken from multiple shooting directions, and can be placed in virtual space. Generate a 3D model. However, in 3D scanning, there is a limit to the resolution of captured images, so if the object has a detailed shape, it is difficult to generate a 3D model that faithfully reproduces this detailed shape. . In addition, the process of generating a 3D model by 3D scanning requires a very heavy processing load on the computer, so after photographing the object, a 3D model corresponding to the object appears in virtual space in a short period of time. There is also the problem that it is difficult to do so.

Therefore, in this embodiment, instead of performing 3D scan analysis processing based on the photographed image of the target object, a texture is generated using the photographed image of the target object. Then, by mapping the texture generated based on the photographed image onto a primitive in the virtual space, a virtual space image in which a display object corresponding to the target object is displayed is generated. Specifically, a primitive realized by a polygon or the like is moved within the virtual space based on the movement control information. As the movement control information, for example, player operation input information can be used. Alternatively, it may be movement control information that moves a primitive along a predetermined trajectory. Then, using the movement control information used to control the movement of the primitive, an appropriate photographed image is selected from among the plurality of photographed images of the object. The texture generated based on the selected photographed image is then mapped onto the primitive that moves based on the movement control information to generate a virtual space image that is an image visible from the virtual camera in the virtual space. That is, by mapping the texture to the primitive, an image of a displayed object such as a character corresponding to the target object is generated, and a virtual space image including the image of this displayed object is generated. By doing so, it becomes possible to generate a virtual space image including an image of a display object in which the shape of the object is faithfully reproduced, without performing heavy processing such as 3D scanning.

Specifically, as shown in FIG. 2A, in this embodiment, the photographing unit 162 photographs the object POB from a plurality of photographing directions. In FIG. 2(A), the object POB is a plastic model of a robot. This object POB is placed on a placement section 10 that is a turntable, and the placement section 10 is rotated using a rotational movement mechanism 12 realized by a motor or the like. For example, the arrangement section 10 is rotated 360 degrees during a photographing time of about 30 to 60 seconds. Further, as a background of the object POB, a monochromatic background portion 20 such as a green screen or a blue screen is arranged. By doing so, the object POB can be photographed by the photographing unit 162 in a 360-degree photographing direction range, for example as shown in FIG. can be obtained. Specifically, by photographing a movie using the photographing unit 162, a group of photographed images, which are moving images (movies) photographed in a 360-degree photographing direction range, can be obtained.

In this embodiment, after such photographing is completed, the character CH (displayed object in a broad sense) corresponding to the object POB is placed in the virtual space in a short time of one action, as shown in FIG. 2(B). The character CH is made to appear so that the player can play a game using this character CH. For example, in FIG. 2B, three-dimensional objects OB1, OB2, and OB3 such as rocks are arranged in a virtual space that is a three-dimensional object space, and an enemy character CHE, which is an enemy robot, also appears. The player can move the character CH as shown in A1 and A2 by operating the operation unit 160 such as a game controller. Further, in the image of the character CH, which is a robot, an image of a vernier injection flame VN, which is an auxiliary device, is also displayed.

Note that although the following explanation will mainly be based on an example in which the object is a plastic model of a robot, the present embodiment is not limited to this. The object may be, for example, a plastic model or model of a ship such as a battleship, an airplane such as a fighter jet, a tank, a car, or a monster. Further, the object is not limited to a plastic model or a model, and may be any object that exists in the real world. For example, objects include dolls, figurines, real-world vehicles such as cars, motorcycles, and automobiles, everyday items such as household appliances, stationery, and watches, works of art such as sculptures and paintings, and constructions such as buildings and bridges. Various objects (subjects) such as objects can be assumed.

Next, the image generation method of this embodiment will be explained in detail using FIGS. 3(A) to 7. First, as shown in FIG. 3(A), the object POB is placed on the placement section 10, which is a turntable. Then, in a photographing time of 30 to 60 seconds, the object POB is photographed by the photographing section 162 with the background section 20 such as a green screen as the background while rotating the placement section 10 by 360 degrees as shown in B1 and B2. Then, moving image data obtained by, for example, movie shooting by the shooting unit 162 is taken into the processing device 200. This moving image corresponds to a group of photographed images obtained by photographing the object POB from a plurality of photographing directions by the photographing unit 162.

Note that although FIG. 3A shows an example in which the processing device 200 that implements the image generation system of this embodiment is a PC (personal computer), this embodiment is not limited to this. The processing device 200 that implements the image generation system may be a commercial game device as shown in FIG. 18, which will be described later, or may be another type of device such as a home game device. For example, the processing device 200 may be a portable communication terminal such as a smartphone or a tablet PC. In this case, the photographing unit 162 can be realized by a built-in camera of a portable communication terminal.

A group of photographed images, which are moving images acquired by photographing by the photographing section 162, are stored in the storage section 170 of the processing device 200. For example, data of a moving image, which is a group of images shot in a 360-degree shooting direction range, is automatically saved in a designated folder of the storage unit 170, for example. FIG. 3(B) is an example of a photographed image IM included in the photographed image group (moving image). This photographed image IM is a photographed image when the rear side, which is the back side of the object POB, is photographed by the photographing unit 162 in FIG. 3(A). The photographed image IM includes an image of the object POB and an image of the background BG formed by the background section 20 in FIG. 3(A). The background BG is, for example, a monochromatic background such as a green background or a blue background. Note that in FIG. 3A, a group of photographed images is acquired by performing movie shooting of the object POB by the photographing unit 162, but for example, while changing the photographing direction of the photographing unit 162, the photographing image group is acquired by taking a time-lapse photograph of the object POB. A group of captured images may be obtained by performing the following steps.

In this embodiment, as shown in FIG. 4(A), the texture TEX generated based on the photographed image IM of FIG. 3(B) is mapped onto the primitive PM. In this case, as shown in FIG. 4(B), the background BG in FIG. 3(B) is transparently processed in real time, and the texture TEX in which only the character CH corresponding to the object POB is displayed is mapped to the primitive PM. be done. That is, as schematically shown in FIG. 4(B), among the pixels of the texture TEX, pixels corresponding to the background BG are set to be transparent, and a texture TEX is generated in which only the pixels corresponding to the character CH are cut out. be done. The character CH is a display object that appears in the game corresponding to the object POB. Since the image of the character CH is an image based on the photographed image IM explained in FIGS. 3(A) and 3(B), the detailed shape of the object POB is more faithfully reproduced than when using 3D scanning. The image will be

Furthermore, the primitive PM is placed in a virtual space that is a three-dimensional object space, and various objects such as objects OB1, OB2, and OB3 representing rocks are placed in this virtual space. In this embodiment, this primitive PM moves in the virtual space according to the movement control information. Specifically, the primitive PM rotates or translates in the virtual space using operation input information input by the player using the operation unit 160 as movement control information. Alternatively, the primitive PM may be rotated or translated in the virtual space based on movement control information that automatically moves the primitive PM. Then, as described later, a texture TEX is generated based on the captured image selected according to the movement control information from among the captured images of the moving image captured in FIGS. 3(A) and 3(B), The generated texture TEX is mapped onto the primitive PM. This makes it possible to map the texture TEX that changes according to the movement control information onto the primitive PM to generate a virtual space image that can be seen from a virtual camera in the virtual space. According to the present embodiment, the object POB, such as a plastic model of a robot, appears in the virtual space, which is the game space, in its original form. Therefore, players can experience virtual reality as if the object POB, such as a plastic model, that they own actually appears in the virtual space, making it possible to realize an unprecedented type of image generation system. It becomes possible.

FIG. 5 is a diagram showing the relationship between primitive PM and virtual camera VC in virtual space. Objects OB1 to OB3 representing rocks, an enemy character CHE which is an enemy robot, and the like are arranged in the virtual space. VM is the view volume of the virtual camera VC, and corresponds to the display range (drawing range) of the virtual camera VC.

As shown in FIG. 5, in this embodiment, the positional relationship between the primitive PM and the virtual camera VC is fixed. For example, while the positional relationship between the primitive PM and the virtual camera VC is fixed, the primitive PM and the virtual camera VC are moved according to movement control information. For example, the rail RL connecting the primitive PM and the virtual camera VC is moved according to the movement control information. Specifically, in FIG. 5, a billboard polygon is used as the primitive PM. For example, the distance between the primitive PM and the virtual camera VC is constant, and the primitive PM is arranged to directly face the virtual camera VC. For example, the primitive PM is arranged such that the distance between the primitive PM and the virtual camera VC is constant, and the viewing direction VL of the virtual camera VC and the surface of the primitive PM are perpendicular to each other. In this way, even when the primitive PM is placed in a three-dimensional virtual space, it is possible to create an effect that prevents the primitive PM from looking flat from the player's perspective, making it appear as if the character is a three-dimensional display object. It becomes possible to provide the player with a virtual reality feeling in which the CH appears to exist in a virtual space.

Further, in this embodiment, as shown in FIG. 6, the texture TEX image mapped to the primitive PM is changed according to the movement control information of the primitive PM. Specifically, in accordance with the player's operation input to the operation unit 160, while fixing the positional relationship between the virtual camera VC and the primitive PM, using the position of the virtual camera VC as a fulcrum (rotation center), C1, C2, C3, The primitive PM is rotated as shown in C4. That is, the rail RL connecting the virtual camera VC and the primitive PM is rotated using the position of the virtual camera VC as a fulcrum. In the case of translational movement, the virtual camera VC and the primitive PM are translated as one unit. That is, the rail RL connecting the virtual camera VC and the primitive PM is translated. For example, when an operation input instructing the left direction is performed, the primitive PM is rotationally moved in the direction shown by C1 using the position of the virtual camera VC as a fulcrum. When an operation input instructing the right direction is performed, the primitive PM is rotationally moved in the direction shown by C2 using the position of the virtual camera VC as a fulcrum. Similarly, when an operation input indicating a downward direction or an upward direction is performed, the primitive PM is rotationally moved in the directions shown in C3 and C4, respectively, using the position of the virtual camera VC as a fulcrum. By doing so, movement processing of the primitive PM is realized in accordance with operation input information that is movement control information.

In conjunction with such movement of the primitive PM, the texture TEX mapped to the primitive PM is changed. Specifically, by playing back a video of the object POB in a 360-degree shooting direction range as shown in Figures 3(A) and 3(B) in conjunction with the movement of the primitive PM. , changes the texture TEX mapped to the primitive PM in conjunction with the movement of the primitive PM. For example, when the player performs an operation input instructing the left direction to rotate the primitive PM using the position of the virtual camera VC as a fulcrum as shown in C1 of FIG. The video is played back as shown in the playback direction DR. For example, video playback may be performed in which the image of the object POB (character CH) taken from the rear side as shown in D1 in FIG. 6 changes to the image taken of the object POB from the rear left side as shown in D2. conduct. Furthermore, moving image playback is performed such that the image is changed to an image taken of the object POB from the front left side as shown in D3, and the image is changed to an image taken of the object POB from the front side as shown in D4. The photographed images shown in D1, D2, D3, and D4 constitute a group of photographed images of this embodiment. In this way, it is possible to generate a virtual space image that looks as if the character CH, which is a three-dimensional display object, is rotating and moving in the virtual space, while using texture TEX based on a two-dimensional photographed image. Become.

Further, in this embodiment, as shown in FIG. 7, effect processing is also performed to realize a 3D effect representation of, for example, the injection flame VN of a vernier or the beam BM fired by a beam cannon. In this embodiment, in order to realize such effect processing, the relative positional relationship with respect to the primitive PM changes according to the movement control information, as will be explained later in FIGS. 13(A) to 13(C). Prepare primitives PMV and PMB. Primitives PMV and PMB are second primitives, for example polygons. An image (texture) representing the injection flame VN is drawn on the primitive PMV, and an image (texture) representing the beam BM is drawn on the primitive PMB. Then, in accordance with the movement control information (operation input information) for moving the primitive PM, the relative positional relationship of the primitives PMV and PMB with respect to the primitive PM is changed as shown in E1 and E2 of FIG. Then, an image including an image of the primitive PM to which the texture TEX has been mapped, and images of the primitives PMV and PMB (second primitive) is generated as a virtual space image.

For example, when the primitive PM rotates, the primitive PMV for the injection flame VN is rotated in conjunction with the rotation as shown in E1 of FIG. That is, as shown in F1 of FIG. 7, the primitive PMV on which the injection flame VN is drawn is rotated so that the injection flame VN appears to be ejected from the vernier auxiliary device of the character CH drawn on the texture TEX. Furthermore, when the primitive PM rotates, the primitive PMB for the beam BM is rotated in conjunction with the rotation as shown in E2 of FIG. That is, as shown in F2 of FIG. 7, the primitive PMB on which the beam BM is drawn is rotated and moved so that the beam BM appears to be fired from the beam cannon of the character CH drawn on the texture TEX.

In this manner, in this embodiment, the primitives PMV and PMB that express the injection of the injection flame VN, the emission of the beam BM, etc. are moved in conjunction with the playback of the moving image as shown in D1 to D4 in FIG. This makes it possible to realistically express effects such as injection flame VN and beam BM. In addition, as will be described later, by controlling the drawing order of primitive PMV and PMB from the position of the virtual camera VC, it is possible to realize the appearance and gameplay of a 3D model, even though it is a 2D model. ing. That is, in this embodiment, as shown in FIGS. 3A and 3B, a virtual space image including an image of the character CH corresponding to the object POB is created based on a photographed image of the object POB such as a plastic model. However, the injection flame VN and beam BM cannot be expressed by the object POB in the real world. Therefore, the injection flame VN and beam BM are expressed using primitives PMV and PMB (second primitives) whose relative positional relationship with respect to the primitive PM changes. Then, the relative positional relationship of the primitives PMV and PMB with respect to the primitive PM is changed so as not to contradict the video playback of the character CH as shown in D1 to D4 in FIG. This makes it possible to express effects of the injection flame VN and beam BM that cannot be expressed using the object POB in the real world.

2.2 Texture Mapping According to Movement Control Information Next, details of the method of this embodiment will be described. FIG. 8 is a flowchart of virtual space image generation processing based on a group of captured images. First, a group of photographed images obtained by photographing the object POB from a plurality of photographing directions is acquired (step S1). For example, by shooting a movie with the shooting unit 162 as shown in FIG. The acquired photographed image group is then stored in the storage unit 170 (step S2). That is, data of a group of captured images, which are moving images, is stored and saved in the memory of the processing device 200. Next, a texture TEX is generated based on the photographed image selected according to the movement control information of the primitive PM (step S3). For example, based on movement control information (operation input information), a corresponding photographed image is selected from a group of photographed images such as D1 to D4 in FIG. Generate a texture TEX like this. Then, the texture TEX that changes according to the movement control information is mapped onto the primitive PM that moves according to the movement control information to generate a virtual space image that can be seen from the virtual camera in the virtual space (step S4). For example, as shown in C1 to C4 in FIG. 6, depending on the movement control information (operation input information), the primitive PM rotates around the position of the virtual camera VC as a fulcrum, or the virtual camera VC and the primitive PM move as a unit. to move in translation. Then, the texture TEX that changes in conjunction with the movement of the primitive PM is mapped onto the primitive PM. For example, when the primitive PM rotates to the left with the position of the virtual camera VC as a fulcrum (rotation center) as shown in C1 of FIG. , a captured image corresponding to the rotation angle of the primitive PM is selected, and a texture TEX based on the selected captured image is mapped onto the primitive PM. This makes it possible to generate a realistic virtual space image in which the three-dimensional model character CH corresponding to the object POB appears to be moving within the virtual space, even though a two-dimensional captured image is used.

Further, in this embodiment, the photographed image group (the first and second photographed image groups) is a group of images in which the change in the photographing direction of the photographing unit 162 with respect to the object POB is within a photographing direction range of 180 degrees or more. . For example, in FIG. 9, the object POB is photographed by the photographing unit 162 in the photographing direction range RDR in which the change in photographing direction DCM with respect to the object POB is 360 degrees. For example, in FIG. 9, the angle of the photographing direction DCM of the photographing unit 162 is 0 degrees when photographing the rear of the object POB, and the angle of the photographing direction DCM when photographing the left side of the object POB is 0 degrees. The angle is 90 degrees. Also, the angle of the photographing direction DCM when photographing the front of the object POB is 180 degrees, and the angle of the photographing direction DCM when photographing the right side of the object POB is 270 degrees. . The photographing direction range RDR in FIG. 9 is a range in which the photographing direction DCM of the photographing section 162 changes from 0 degrees to 360 degrees. However, for example, the shooting direction range RDR may be set to 180 degrees such that the change in shooting direction DCM is 0 to 90 degrees and 360 degrees to 270 degrees, or the shooting direction range RDR may be set to 180 degrees to 360 degrees. It may be set in the direction range.

In this way, by using a group of photographed images in which the photographing direction of the photographing unit 162 with respect to the object POB changes within a photographing direction range of 180 degrees or more, even when the primitive PM rotates or moves in various directions, it is possible to It becomes possible to generate a texture based on an appropriate corresponding photographed image and map it to the primitive PM.

Note that although FIGS. 3A and 9 show the case where the photographing direction DCM is rotated, for example, with the vertical direction along the main axis of the object POB as the rotation axis, the present embodiment is not limited to this. For example, a group of photographed images may be acquired by photographing the object POB while rotating the photographing direction DCM with the horizontal direction, which is a direction perpendicular to the main axis of the object POB, as the rotation axis. In this way, even when rotating the primitive PM in the direction of C3 or C4 in FIG. 6, for example, it becomes possible to map the texture TEX based on an appropriate captured image according to the rotational movement onto the primitive PM. Furthermore, in FIG. 3A, by rotating the object POB, a group of photographed images in the photographing direction range RDR as shown in FIG. By rotating around the center, a group of photographed images in the photographing direction range RDR may be acquired. That is, the imaging unit 162 is moved on a circle centered on the object POB in the imaging direction DCM such that the object POB is watched.

Further, when photographing is performed in a plurality of photographing directions as shown in FIG. 9, it is desirable that the number of photographing directions, that is, the number of images in the photographed image group, be as large as possible. For example, when the number of frames of images per second is p (for example, p=30, 60, etc.), it is desirable that the number of images in the photographed image group corresponding to the number of photographing directions is p or more. That is, the object POB is photographed in at least p photographing directions, and a photographed image group including at least p photographed images is obtained. In this way, a moving image with a smooth and appropriate texture can be played for at least one second and mapped to the primitive PM.

Furthermore, in the present embodiment, a texture to be mapped to the primitive PM is generated based on a captured image selected from a captured image group (first and second captured image groups) according to movement control information of the primitive PM. . Specifically, for example, a captured image is selected from the captured image group by performing seek control of a video composed of a group of captured images based on movement control information, and a texture is generated based on the selected captured image. .

For example, in FIG. 10, the captured images IM1 to IMj are a captured image group, and from this captured image group, a texture is generated based on the captured image IMi selected according to the movement control information of the primitive PM, and is mapped to the primitive PM. Ru. Here, i and j are integers satisfying 1≦i≦j. For example, j is the number of images in the photographed image group, and it is desirable that j≧p, where p is the number of frames of images per second as described above.

In FIG. 10, seek control of a moving image composed of a group of photographed images IM1 to IMj is performed based on movement control information. For example, in the case of rewinding seek control, seek control is performed in the direction DS1 in FIG. 10, and a photographed image according to the movement control information is selected from the photographed image group IM1 to IMj. On the other hand, in the case of fast-forward seek control, seek control is performed in the direction DS2, and a photographed image according to the movement control information is selected from the photographed image group IM1 to IMj. For example, assume that seek numbers SN1 to SNj are associated with photographed images IM1 to IMj. In this case, for example, one of the seek numbers SN1 to SNj is selected according to the movement control information of the primitive PM, a captured image associated with the selected seek number is selected, a texture is generated, and the primitive Mapped to PM. For example, when seek number SNi is selected based on the movement control information, captured image IMi associated with seek number SNi is selected, texture is generated, and mapped to primitive PM. In this way, it is possible to effectively utilize the seek control of a video composed of a group of captured images, select an appropriate captured image corresponding to the movement control information of the primitive PM, generate a texture, and map it to the primitive PM. It becomes like this.

Further, in this embodiment, processing is performed to move the primitive PM and the virtual camera VC in accordance with movement control information while the positional relationship between the primitive PM and the virtual camera VC is fixed. For example, in FIGS. 11A to 11C, a texture depicting an image of a character CH is mapped to a primitive PM.

The image of the character CH is actually a two-dimensional image, but in order to make the explanation easier to understand, the image of the character CH is shown schematically in order to make it easier to understand. It is described in The same applies to the following explanation.

In FIGS. 11A to 11C, the distance between the virtual camera VC and the primitive PM is constant and fixed. That is, the distance between the viewpoint position PS of the virtual camera VC and the surface of the primitive PM is constant. Moreover, the primitive PM is arranged so as to directly face the virtual camera VC, and the primitive PM is arranged so that the viewing direction VL of the virtual camera VC is orthogonal to the surface of the primitive PM. As described above, in this embodiment, the primitive PM and the virtual camera VC are moved in the virtual space while the positional relationship between the primitive PM and the virtual camera VC is fixed. For example, a billboard polygon is used as the primitive PM. By doing so, it becomes possible to generate an image that looks like a three-dimensional image in a pseudo manner as an image of a display object such as a character CH that is displayed by mapping the texture to the primitive PM.

As shown in C1 of FIG. 6, for example, when the player inputs an operation to instruct the left direction, the viewpoint position of the virtual camera VC is changed as shown in FIGS. 11(A) to 11(B). The primitive PM rotates counterclockwise using PS as a fulcrum (rotation center). At this time, the distance between the virtual camera VC and the primitive PM is constant, and the viewing direction VL of the virtual camera VC is perpendicular to the surface of the primitive PM. With the positional relationship between the primitive PM and the virtual camera VC fixed in this manner, the primitive PM and the virtual camera VC rotate and move based on the movement control information indicating the left direction as shown in C1 of FIG. ing. The image of the character CH mapped onto the primitive PM at this time is an image obtained by rotating the character CH so that the direction changes counterclockwise, as shown in D2 in FIG. For example, the image of the character CH is an image of the object POB corresponding to the character CH viewed from the left rear side. In this way, when the player issues an operation instruction to the left, an image of the character CH that has turned counterclockwise will be displayed as a virtual space image. It becomes possible to display an appropriate image of the character CH according to the movement control information.

In addition, as shown in C2 of FIG. 6, when the player inputs an operation to direct the right direction, for example, the viewpoint position of the virtual camera VC is changed as shown in FIGS. 11(A) to 11(C). The primitive PM rotates clockwise using the PS as a fulcrum. At this time, the distance between the virtual camera VC and the primitive PM is constant, and the viewing direction VL of the virtual camera VC is perpendicular to the surface of the primitive PM. With the positional relationship between the primitive PM and the virtual camera VC fixed in this manner, the primitive PM and the virtual camera VC rotate and move based on the movement control information indicating the right direction as shown in C2 of FIG. ing. The image of the character CH mapped onto the primitive PM at this time is an image obtained by rotating the character CH so that the direction changes clockwise. For example, the image of the character CH is an image of the object POB corresponding to the character CH viewed from the right rear. In this way, when the player issues an operation instruction to the right, an image of the character CH that has turned clockwise will be displayed as a virtual space image. It becomes possible to display an appropriate image of the character CH according to the movement control information.

Further, in this embodiment, as shown in FIGS. 3A and 3B, the photographing unit 162 photographs the object POB and the background BG of the object POB. For example, by photographing the background portion 20 in FIG. 3(A), a monochromatic background BG is photographed as shown in FIG. 3(B). Then, it is determined whether the color of each pixel in the photographed image corresponds to the color of the background. If it is determined that the pixel has a color corresponding to the background color, a texture is generated by making the pixel of the color corresponding to the background transparent.

FIG. 12 shows a flowchart of the process of making pixels of a color corresponding to the background color transparent. The processing in FIG. 12 can be realized by, for example, a pixel shader. First, the color of the pixel to be processed is determined (step S11). Then, it is determined whether the color of the pixel to be processed corresponds to the color of the background (step S12). For example, in the case of a green screen, if the color of the pixel to be processed is green, it is determined that the color corresponds to the background color. In the case of a blue background, if the color of the pixel to be processed is blue, it is determined that the color corresponds to the background color. If it is determined that the color of the pixel to be processed corresponds to the color of the background, the pixel to be processed is set to be transparent (step S13). Taking α compositing processing as an example, when it is determined that the color of the pixel to be processed corresponds to the color of the background, the α value of the pixel to be processed is set to zero. Then, it is determined whether all pixels of the photographed image have been processed (step S14), and if all pixels have not been processed, the next pixel is set as the pixel to be processed (step S15), and step Return to S11. On the other hand, if all pixels of the photographed image have been processed, the processing ends.

By performing the processing as shown in FIG. 12, it becomes possible to generate a texture TEX as shown in FIG. 4(B) from the captured image IM as shown in FIG. 3(B). This makes it possible to generate an appropriate virtual space image in which other objects in the virtual space are displayed in the background BG. Further, the processing shown in FIG. 12 can be realized by a pixel shader based on a shader program, and can be realized by simple processing with a light processing load. Therefore, as shown in FIGS. 2(A) and 2(B), after photographing the object POB, with one action, a display object such as a character CH corresponding to the object POB appears in the virtual space. This can be easily realized.

Note that the background color does not need to be a fixed color, and may be set variably. For example, the background color is variably set depending on the color of the object POB. For example, the background color may be set to a color that does not exist in the object POB, or a complementary color to the color of the object POB may be set as the background color. Specifically, in FIGS. 2(A) and 2(B), a background portion 20 whose color can be variably set is used. For example, the color of the background portion 20 is variably set by projecting light of a predetermined color as the background color using a projector. Alternatively, the color of the background section 20 can be variably set by using a display device such as a curved display device as the background section 20 and displaying a screen of a predetermined color as the background color. By doing so, the background color can be set to an appropriate color according to various objects POB.

Further, in this embodiment, a movement process of the second primitive is performed in which the relative positional relationship with respect to the primitive changes according to the movement control information. Then, an image including the image of the primitive to which the texture has been mapped and the image of the second primitive is generated as a virtual space image.

For example, in FIG. 13A, primitives PMV and PMB are arranged as second primitives based on the position of primitive PM. As explained in FIG. 7, the primitive PMV is a primitive for displaying an image of the vernier injection flame VN, and the primitive PMB is a primitive for displaying an image of the beam BM fired by the beam cannon. Note that the images of the injection flame VN and beam BM may be generated using three-dimensional objects, and in this case, the primitives PMV and PMB are polygons and the like that constitute the objects.

In FIGS. 13A to 13C, the process of moving the primitive PM according to the movement control information is performed as described in FIGS. 11A to 11C. Furthermore, the image (texture) of the character CH to be mapped to the primitive PM is also generated using an appropriate captured image according to the movement control information. In FIGS. 13A to 13C, the movement processing of the primitives PMV and PMB is performed so that the relative positional relationship with respect to the primitive PM changes according to the movement control information.

For example, in FIG. 13A, an image in which the character CH is facing in the line-of-sight direction VL of the virtual camera VC is mapped to the primitive PM. Specifically, as in the case where the photographing direction DCM is 0 degrees in FIG. 9, a texture generated based on a photographed image taken from directly behind the object POB is mapped to the primitive PM as an image of the character CH. . In this case, a primitive PMV for the injection flame VN is placed directly behind the character CH. Furthermore, since the character CH has a beam cannon in his right hand, for example, a primitive PMB for the beam BM is arranged in the front right direction of the character CH. As a result, as shown in FIG. 7, it becomes possible to generate an appropriate virtual space image in which the injection flame VN is injected from the position of the vernier, which is an auxiliary device, and the beam BM is ejected from the position of the beam cannon.

On the other hand, in FIG. 13(B), as shown in C1 of FIG. 6, the player has given an operation instruction to the left, which causes the player to rotate counterclockwise (counterclockwise) with the position of the virtual camera VC as a fulcrum. ), the primitive PM is rotating. With such rotational movement of the primitive PM, the image of the character CH mapped to the primitive PM also becomes an image whose direction has changed counterclockwise, as shown in D2 in FIG. The relative positional relationship of the primitives PMV and PMB with respect to the primitive PM also changes in conjunction with the image of the character CH changing its direction counterclockwise. That is, the relative positional relationship of the primitive PMV with respect to the primitive PM is changed so that the primitive PMV is located behind the character CH which has turned counterclockwise. Specifically, the primitive PMV is rotated counterclockwise with respect to the primitive PM. Further, the relative positional relationship of the primitive PMB with respect to the primitive PM is changed so that the primitive PMB is located in the front right direction of the character CH which has turned counterclockwise. Specifically, the primitive PMB is rotated counterclockwise with respect to the primitive PM. By doing so, it becomes possible to generate an appropriate virtual space image in which the injection flame VN is injected from the vernier position and the beam BM is ejected from the beam cannon position.

In addition, in FIG. 13(C), as shown in C2 of FIG. 6, the player gives an operation instruction to the right, which causes the player to rotate clockwise around the position of the virtual camera VC as a fulcrum. Primitive PM is rotating and moving. With such rotational movement of the primitive PM, the image of the character CH mapped to the primitive PM also becomes an image whose direction has changed clockwise. The relative positional relationship of the primitives PMV and PMB with respect to the primitive PM also changes in conjunction with the image of the character CH turning clockwise. That is, the primitive PMV is rotated clockwise with respect to the primitive PM as a reference so that the primitive PMV is located behind the character CH whose direction has changed clockwise. Furthermore, the primitive PMB is rotated clockwise with respect to the primitive PM as a reference so that the primitive PMB is positioned in the front right direction of the character CH whose direction has changed clockwise. By doing so, it becomes possible to generate an appropriate virtual space image in which the injection flame VN is injected from the vernier position and the beam BM is ejected from the beam cannon position.

Furthermore, in this embodiment, hidden surface removal processing for the second primitive is performed in accordance with movement control information. For example, in FIG. 13B, the image of the beam BM is an image that should be hidden behind the character CH when viewed from the virtual camera VC. For example, in the positional relationship as shown in FIG. 13(A), the image of the beam BM is an image that should be displayed without being hidden by the character CH. However, when the primitive PM is rotated as shown in FIG. 13(B) and the primitive PMB is rotated counterclockwise with respect to the primitive PM, the image of the beam BM of the primitive PMB is hidden by the character CH and becomes invisible. The image becomes the image that should be displayed. Therefore, in this case, hidden surface elimination processing is performed on the primitive PMB for the beam BM. By performing hidden surface removal processing on the primitive PMB in this manner, the image of the beam BM is hidden from view from the virtual camera VC, making it possible to generate an appropriate virtual space image. That is, normally, hidden surface removal processing is performed by determining the context (depth value) of the primitives PM, PMV, and PMB as seen from the virtual camera VC. In this regard, in the present embodiment, the hidden surface removal process is performed by determining the front-and-back relationship between the character CH mapped to the primitive PM, the beam BM, and the injection flame VN based on movement control information. By doing so, it becomes possible to generate a virtual space image in which hidden surfaces have been appropriately removed.

Further, in the present embodiment, processing is performed to move the second primitive from the initial placement position of the second primitive set based on the analysis result of the target object POB, using the primitive PM as a reference. For example, in FIG. 14, the injection position of the vernier injection flame VN and the emission position of the beam BM are specified by performing analysis processing such as silhouette analysis of the object POB. That is, the injection position of the injection flame VN and the emission position of the beam BM differ depending on the type and shape of the object POB. Therefore, by performing analysis processing such as silhouette analysis based on the photographed image of the object POB, the type and shape of the object POB are specified, and the injection position of the injection flame VN and the emission position of the beam BM are specified. Then, as shown in FIG. 14, the injection position of the identified injection flame VN is set to the initial arrangement position PI1 of the primitive PMV for the injection flame VN. Further, the identified beam BM emission position is set to the initial arrangement position PI2 of the primitive PMB for the beam BM. Then, from these initial placement positions PI1 and PI2, as shown in FIGS. 13A to 13C, processing is performed to move the primitives PMV and PMB based on the primitive PMV. That is, with PI1 and PI2 as initial placement positions, the primitives PMV and PMB are rotationally moved with respect to the primitive PMV. In this way, the primitives PMV and PMB are placed at the appropriate initial placement positions PI1 and PI2 according to the analysis results of the target object POB, and when the primitive PM moves according to the movement control information, the primitives are linked to the movement. The image of the injection flame VN and the image of the beam BM can be displayed at appropriate positions.

In addition, by attaching a marker to the injection position of the injection flame VN and the firing position of the beam BM in the target object POB, and performing recognition processing of this marker in the analysis process of the target object POB, the initial placement position of the second primitive can be determined. may be determined. That is, based on the result of the marker recognition process, the initial arrangement position PI1 of the primitive PMV for the injection flame VN and the initial arrangement position PI2 of the primitive PMB for the beam BM are determined.

Furthermore, in this embodiment, the images of the injection flame VN and beam BM, which are images of the primitives PMV and PMB, are effect images. That is, in this embodiment, the object POB is actually photographed by the photographing unit 162 to generate a texture and display the image of the character CH corresponding to the object POB, but effects such as the injection flame VN and the beam BM are The image cannot be photographed by actual photographing by the photographing unit 162. Therefore, in this embodiment, images of effects such as the injection flame VN and the beam BM are generated using primitives PMV and PMB that are arranged relatively to the primitive PM. In this way, it becomes possible to display an effect image that cannot be photographed by actual photography by the photographing unit 162 along with the image of the character CH corresponding to the object POB.

Further, in this embodiment, an image including a primitive image and images of a plurality of objects set in the virtual space is generated as a virtual space image, and post-effect processing is performed on the generated virtual space image. For example, in FIG. 15, an image is generated that includes an image of the character CH, which is an image of the primitive PM, and images of rock objects OB1, OB2, OB3 set in the virtual space, and objects of the enemy character CHE, as the virtual space image. ing. Post-effect processing is then performed on the virtual space image generated in this way. For example, post-effect processing such as blur processing, brightness adjustment processing, or hue adjustment processing is performed. For example, post-effect processing is performed on the virtual space image in which other objects OB1, OB2, OB3, etc. are displayed so that the image of the character CH represented by the primitive PM is displayed without any discomfort. For example, post-effect processing is performed to blur the outline of the character CH represented by the primitive PM, so that the outline that is the boundary between the character CH and the virtual space image becomes less noticeable. Furthermore, if there is a difference between the brightness of the image of the character CH and the brightness of the surrounding virtual space image, post-effect processing is performed to reduce this difference. By doing so, it becomes possible to generate a high-quality, realistic virtual space image in which the image of the character CH generated based on the photographed image blends seamlessly into the surrounding virtual space.

Further, in this embodiment, a group of photographed images for another display object obtained by photographing another target object corresponding to the other display object from a plurality of photographing directions is acquired. Then, in accordance with the movement control information for the other display object, a texture for the other display object is generated based on a photographed image selected from a group of photographed images for the other display object. Then, a virtual space image is generated by mapping the texture for another display object that changes according to the movement control information for the other display object onto the primitive for the other display object. Here, the other display object is, for example, an enemy character.

For example, in FIG. 16, a virtual space image is generated that includes images of enemy characters CHE1 and CHE2, which are other display objects. In this case, a group of photographed images obtained by photographing objects corresponding to the enemy characters CHE1 and CHE2 from a plurality of photographing directions is obtained. For example, by photographing plastic models of enemy robots corresponding to enemy characters CHE1 and CHE2 from multiple photographing directions using a method similar to that shown in FIGS. Obtain a group of captured images (for display objects). Then, textures TEXE1 and TEXE2 for the enemy characters CHE1 and CHE2 are generated based on the photographed images selected from the group of photographed images for the enemy characters CHE1 and CHE2 in accordance with the movement control information for the enemy characters CHE1 and CHE2. That is, the enemy characters CHE1 and CHE2 move within the virtual space based on the movement control information for the enemy characters CHE1 and CHE2. Based on the movement control information for the enemy characters CHE1 and CHE2, a photographed image is selected from a group of photographed images for the enemy characters CHE1 and CHE2, and textures TEXE1 and TEXE2 for the enemy characters CHE1 and CHE2 are generated. By doing this, it becomes possible to generate textures TEXE1 and TEXE2 that reflect movement control information, as in the case of the player's character CH. Then, as shown in FIG. 16, a virtual space image is generated by mapping the generated textures TEXE1 and TEXE2 onto primitives PME1 and PME2 for enemy characters CHE1 and CHE2.

In this way, for the enemy characters CHE1 and CHE2, as in the case of the character CH, an effect can be created in which the plastic model of the enemy robot, which is the object corresponding to the enemy characters CHE1 and CHE2, appears in the virtual space. It becomes possible. This makes it possible to realize an unprecedented game in which the plastic model of the player's robot and the plastic model of the enemy robot compete against each other in virtual space. Further, the images of the enemy characters CHE1 and CHE2 can also be made to reflect the movement control information for the enemy characters CHE1 and CHE2, making it possible to generate more realistic images. For example, in FIG. 16, for the enemy character CHE1, an image is generated in which the enemy character CHE1 faces to the right forward based on the primitive PME1, and an image of the enemy character CHE1 facing toward the character CH is generated as a virtual space image. become. Furthermore, for the enemy character CHE2, an image is generated that looks to the left front with reference to the primitive PME2, and an image of the enemy character CHE2 facing the character CH is generated as a virtual space image. This makes it possible to improve the player's sense of virtual reality.

Further, in this embodiment, an image in which particles are displayed around the primitive PM is generated as a virtual space image. For example, in FIG. 17, particles PT are generated around the primitive PM to which the image of the character CH is mapped. The particle PT display process is a process for expressing an event made up of moving particles, and for example, the movement of the particles PT is controlled by physical simulation. For example, more precise particle control can be achieved by setting environmental conditions such as initial velocity, weight, gravity, and air resistance when particles PT are generated. For example, a buffer for managing the vertex coordinates of particles PT is provided in the storage unit 170, and the behavior of particles PT is controlled by recursively performing shader calculations.

That is, in this embodiment, a two-dimensional captured image of the object POB is used to generate an image of the character CH that appears to be a pseudo three-dimensional image. On the other hand, other objects in the virtual space are represented by three-dimensional models. Therefore, unless some measures are taken, a sense of incongruity will occur between the image of the character CH, which is a pseudo three-dimensional image, and the image of other objects, which are three-dimensional models, and the player's sense of virtual reality will be affected. There is a risk of damage. In this regard, if particles PT are generated around the primitive PM to which the image of the character CH is mapped as shown in FIG. This creates a three-dimensional sense of space. This makes it possible to provide the player with a virtual reality feeling as if the three-dimensional model character CH is moving in a three-dimensional virtual space.

FIG. 18 shows an example of application of the image generation system of this embodiment to a commercial game device 210. The image generation system shown in FIG. 18 is provided with a photographing section 162, a placement section 10 in which the object POB is placed, and a rotational movement mechanism 12 in addition to a business game device 210 as a main device. Specifically, the image generation system shown in FIG. 18 is provided with a photographing casing 220, and this photographing casing 220 is provided with the photographing section 162, the arrangement section 10, and the rotational movement mechanism 12. The photographing casing 220 is also provided with a background section 20. The rotational movement mechanism 12 is a mechanism that performs rotational movement to change the photographing direction of the photographing section 162 with respect to the object POB placed in the placement section 10. In FIG. 18, the rotational movement mechanism 12 rotates the arrangement section 10, thereby changing the direction in which the imaging section 162 takes pictures of the object POB. The game device 210 (main body side casing) and the photographing casing 220 are connected to each other by wire or wirelessly, for example, so that the game device 210 can control the rotation of the arrangement section 10 by the rotational movement mechanism 12, for example. ing.

In the image generation system of FIG. 18, the player first places an object POB, such as a plastic model of a robot, which the player desires to photograph, in the placement section 10 of the photographing casing 220. When the player performs an operation instructing photography on the game device 210, the rotation movement mechanism 12 rotates the arrangement section 10, and the photographing section 162 captures the object in a 360-degree photographing direction range, as shown in FIG. Photograph the POB. Then, the data of the moving image (shot image group) obtained by shooting is transferred to the game device 210 and stored in the storage unit 170 of the game device 210. As shown in FIGS. 2(A) and 2(B), the player can play a game in which a character CH corresponding to the object POB appears in the virtual space with one action from photographing the object POB. become. As a result, players will be able to have a character CH that corresponds to an object POB such as a plastic model that they own appear in the virtual space and enjoy a game using the character CH, realizing an unprecedented game. becomes possible.

Further, as shown in FIG. 19, it is desirable to provide a support section 16 that supports the object POB while floating in the air. In FIG. 19 , one end of a rod-shaped support portion 16 is attached to the placement portion 10 . A hole is provided in the buttocks of the object POB, which is a plastic model of the robot, and by inserting the other end of the rod-shaped support 16 into this hole, the object POB is suspended in the air. It can be supported by In this way, it becomes possible to make a character CH in a floating pose appear as the character CH corresponding to the object POB, and it becomes possible to meet the various demands of the players.

Note that in FIG. 19, the other end of the rod-shaped support portion 16 is inserted into the hole in the buttock of the object POB to support the object POB in a suspended state, but the support method of this embodiment is not limited to this. For example, the object POB may be suspended in the air by a thin transparent linear member such as a piano wire, so that the object POB is supported while floating in the air. Alternatively, by providing a hole in the head of the object POB and inserting the other end of the support section 16 into the hole in the head, the object POB can be suspended and supported in the upside down state. You can. In this case, it is sufficient to vertically invert the photographed images of the acquired photographic image group.

2.3 Generation of Texture from First and Second Groups of Photographed Images Above, the case where there is only one group of photographed images has been mainly described. Below, a method of generating a texture based on captured images of a plurality of captured image groups such as the first and second captured image groups, mapping it to a primitive PM, and generating a virtual space image will be described. FIG. 20 is a flowchart of virtual space image generation processing based on the first and second photographed image groups.

First, a first group of photographed images obtained by photographing the object from a plurality of photographing directions is obtained by the photographing unit 162 (step S21). That is, a first group of photographed images (first moving image) is acquired by the photographing method explained with reference to FIGS. 3(A) and 3(B). Next, a second group of photographed images is obtained by photographing the object with a changed pose, the object with an accompanying object added, or the object with a deformed shape from a plurality of photographing directions (step S22). . For example, as shown in FIGS. 21(A) and 21(B), which will be described later, the object POB in which the pose of the object in the first photographed image group has been changed is ) to obtain a second group of captured images (second moving image). Alternatively, as shown in FIG. 22(A), the object POB in which a weapon such as a cannon CN or an accessory such as armor is added to the object in the first photographed image group is captured from multiple photographing directions. By photographing, a second group of photographed images is acquired. Alternatively, as shown in FIG. 22(B), a second group of photographed images is obtained by photographing the object POB, which is a shape-deformed object in the first group of photographed images, from a plurality of photographing directions. Alternatively, as shown in FIG. 23, a second group of captured images is obtained by processing the first group of captured images. Then, the first and second photographed image groups acquired in this way are stored and saved in the storage unit 170 (step S23).

Next, a texture to be mapped to the primitive is generated from the first and second photographed image groups (step S24). For example, a texture is generated based on the captured images of the first and second captured image groups using the method described in FIGS. 4(A), 4(B), FIG. 12, etc. Then, the generated texture is mapped onto a primitive to generate a virtual space image that can be seen from the virtual camera in the virtual space (step S25). For example, by performing processing to move a primitive in virtual space and mapping the texture generated from the first and second captured image groups to the primitive moving in virtual space, the image includes an image in which the texture is mapped to the primitive. Generate a virtual space image.

In this way, a texture can be generated using not only the first group of captured images but also the second group of captured images, and a virtual space image including an image of a primitive to which the generated texture is mapped can be generated. It becomes like this. For example, you can process not only a photographed image of the object, but also a photographed image of the object with a changed pose, a photographed image of the object with appendages added, a photographed image of the object with deformed shape, or a photographed image of the target. A texture is generated based on the photographed image, and a virtual space image including an image of a primitive to which the texture is mapped can be generated. Therefore, it is possible to make a display object corresponding to the target object appear in the virtual space, and it is also possible to generate virtual space images based on captured images of the target object in various forms.

For example, FIG. 21(A) is an example of the object POB. This object POB is a plastic model of a robot, and is placed in the placement section 10. By rotating the arrangement unit 10 using the rotational movement mechanism 12, a group of photographed images of the object POB photographed in a plurality of photographing directions can be obtained. As explained with reference to FIG. 3B, the captured images of this captured image group are images in which the object POB is reflected and the background BG formed by the background section 20 is reflected.

In FIG. 21(B), the object POB whose pose has been changed is placed in the placement unit 10. That is, the object POB in FIG. 21(B) is in a different pose from that in FIG. 21(A). The object POB in FIG. 21(B) may be the object POB in FIG. 21(A) whose pose is changed by bending the joints, etc., or the object POB in FIG. 21(A) may be different from the object POB in FIG. It may be an object (second object) prepared separately by the player. For example, a first group of captured images is obtained by arranging an object POB in a basic pose as shown in FIG. 21(A) in the placement unit 10 and photographing it from a plurality of photographing directions. Further, a second group of photographed images is obtained by arranging the object POB in a predetermined pose as shown in FIG. 21(B) in the arrangement section 10 and photographing it from a plurality of photographing directions. Furthermore, three or more photographed image groups, such as a third and fourth photographed image group, may be obtained by photographing the object POB with different poses.

Further, in FIG. 22(A), an object POB to which an accompanying cannon CN is added is placed in the placement section 10. That is, the object POB in FIG. 22(A) has a cannon CN added thereto, which is not present in FIG. 21(A). The object POB in FIG. 22(A) may be the object POB in FIG. 21(A) with an accompanying object such as a cannon CN added, or the object POB in FIG. 21(A) may be the same as the object POB in FIG. 21(A). may be an object (second object) with an accompanying object prepared separately by the player. For example, a first group of photographed images is obtained by arranging an object POB without a cannon CN as shown in FIG. 21(A) in the arrangement section 10 and photographing it from a plurality of photographing directions. Further, a second group of photographed images is obtained by placing an object POB equipped with a cannon CN as shown in FIG. 22(A) in the placement section 10 and photographing from a plurality of photographing directions. Furthermore, by photographing the object POB to which different accompanying objects are added, three or more photographed image groups, such as a third and fourth photographed image group, may be obtained. Incidental items added to the object POB are not limited to the cannon CN, but include weapons other than the cannon CN, armor, equipment (equipment other than weapons and armor), and accessories (accessories, decorations, etc.) , or personal belongings (bags, watches, etc.).

Further, in FIG. 22(B), an object POB whose shape has been deformed due to transformation or the like is placed in the placement section 10. That is, the object POB in FIG. 22(B) has been deformed into a shape different from that in FIG. 21(A). For example, the robot's shape has changed from a ground robot to an airplane-shaped robot. The object POB in FIG. 22(B) may be the object POB in FIG. 21(A) whose shape is deformed by changing parts etc., or the object POB in FIG. 21(A) may be the same as the object POB in FIG. may be an object (second object) prepared separately by the player. For example, a first group of photographed images is obtained by arranging a normal type object POB as shown in FIG. 21(A) in the arrangement section 10 and photographing it from a plurality of photographing directions. Further, a second group of photographed images is obtained by arranging the object POB, which has been deformed into a special shape as shown in FIG. 22(B), in the arrangement section 10 and photographing it from a plurality of photographing directions. Furthermore, three or more photographed image groups, such as the third and fourth photographed image groups, may be acquired by photographing the object POB that has been deformed into third and fourth different shapes.

Further, in FIG. 23, a second group of captured images is obtained by processing the first group of captured images. For example, the second group of photographed images is obtained by performing an enlargement process to change the size of a display object such as a character appearing in the photographed images of the first group of photographed images so as to be stretched in the horizontal direction. For example, processing that makes the display mode of objects such as characters that appear in the captured images of the first captured image group different from the display mode of displayed objects such as characters that appear in the captured images of the second captured image group. is being carried out. Alternatively, the first group of captured images may be subjected to image effect processing such as blurring, or the first group of captured images may be processed to change the color, brightness, or contrast of the first group of captured images. The second group of photographed images may be obtained by performing processing such as horizontally flipping, vertically flipping, or rotating the photographed image group. Alternatively, the second group of photographed images may be obtained by processing a display object such as a character so that it appears to be destroyed by an attack from an enemy.

Further, in this embodiment, as shown in FIG. 24, one of the first and second photographed image groups is selected in accordance with the player's operation input information. For example, if it is determined that the player has made the first operation input, the first group of captured images from the first and second groups of captured images is selected. On the other hand, if it is determined that the player has made the second operation input, the second group of captured images is selected from the first and second groups of captured images. Then, a texture is generated based on the captured images of one of the selected captured image groups, and the generated texture is mapped to a primitive to generate a virtual space image that can be seen from the virtual camera. In this way, an appropriate group of captured images is selected according to the player's operation input information, and an image mapped with a texture based on the captured images of the selected group of captured images is transferred to the displayed object corresponding to the target object. It can be generated as an image.

For example, when the player is performing a normal operation input (first operation input) such as moving the character, the player can obtain an advantage by photographing the object POB in a normal pose as shown in FIG. 21(A). The first photographed image group is selected. On the other hand, when the player performs an operation input to attack an enemy with a weapon (second operation input), by photographing the object POB in a pose to attack with a weapon as shown in FIG. 21(B), The obtained second photographed image group is selected. This makes it possible to generate a virtual space image that includes an image of a character in a pose of attacking with a weapon. Alternatively, if the player performs an operation input to attach the cannon CN (second operation input), the player can obtain the desired result by photographing the object POB equipped with the cannon CN as shown in FIG. 22(A). The second captured image group is selected. This makes it possible to generate a virtual space image that includes an image of a character equipped with a cannon CN. Alternatively, if the player performs an operation input (second operation input) that transforms (transforms) into a special shape, the object POB transformed into a special shape as shown in FIG. 22(B) is photographed. The second photographed image group obtained by this method is selected. This makes it possible to generate a virtual space image that includes an image of a character transformed into a special shape.

Furthermore, in this embodiment, one of the first and second photographed image groups is selected according to the analysis result of the object and the operation input information of the player, and the selected one of the photographed image groups is Generate a texture based on the captured image. In this way, an appropriate group of photographed images is selected according to not only the player's operation input information but also the analysis results of the object, and an image to which textures based on the photographed images of the selected group of photographed images are mapped is created. , it becomes possible to generate an image of a display object corresponding to the target object. FIG. 25 is a flowchart illustrating virtual space image generation processing using such a method.

First, an analysis process of the object is performed based on the photographed image etc. (step S31). For example, the type and shape of the object is identified by analyzing the silhouette of the object. Also, the player's operation input information is determined (step S32). For example, based on the operation input information input by the player using the operation unit 160, it is determined what kind of operation input the player has made. Then, one of the first and second photographed image groups is selected according to the analysis result of the object and the player's operation input information (step S33). For example, the type of the object is specified based on the analysis result of the object, and it is determined whether the player has performed a specific operation input prepared for that type of object. Then, a texture is generated based on the captured images of one of the selected captured image groups, and the generated texture is mapped to generate a virtual space image that can be seen from the virtual camera in the virtual space (steps S34, S35). .

For example, if the object is a plastic model of a robot, since there are multiple types of plastic models of robots, it is determined by silhouette analysis etc. which type of plastic model of the robot was photographed. In the case of a plastic model of the first type of robot, it is assumed that the game is set such that when the player performs the first operation input, the corresponding character performs a special technique. In this case, the analysis process based on the photographed image identifies that the plastic model of the robot that is the object is of the first type, and when the player performs the first operation input, the special technique is executed. Select the second group of photographed images to be used when unrolling. The texture generated based on the captured images of the second captured image group is then mapped onto the primitive to generate a virtual space image. By doing so, an image in which a character corresponding to the plastic model of the first type of robot performs a special technique can be generated, and the sense of virtual reality can be improved.

Furthermore, in this embodiment, processing for moving primitives in virtual space is performed based on movement control information. Then, as shown in FIG. 26, a texture is generated based on a photographed image selected from one of the first and second photographed image groups according to the movement control information. The texture that changes according to the movement control information is then mapped onto the primitive to generate a virtual space image. In this way, it becomes possible to generate a virtual space image using the method explained in FIGS. be able to generate.

Furthermore, in the present embodiment, a texture is generated by combining the captured images of the first captured image group and the captured images of the second captured image group.

For example, in FIG. 27A, captured images IMX1 to IMXj, which are a first group of captured images, and captured images IMY1 to IMYj, which are a second group of captured images, are acquired. Then, the captured image IMXi of the first captured image group and the captured image IMYi of the second captured image group are combined to generate a composite image IMXY. For example, a composition process such as α composition is performed to generate a composite image IMXY. A texture is then generated based on the composite image IMXY and mapped onto the primitive. As a result, a virtual space image including an image of the primitive is generated. In this way, a texture that reflects both the captured images of the first captured image group and the captured images of the second captured image group is generated, and the image to which the generated texture is mapped is applied to the object. It becomes possible to generate an image of the corresponding display object.

For example, in FIG. 27(B), a first group of captured images is a group of captured images when the object is in the first pose, and a second group of captured images is a group of captured images when the object is in the second pose. Assume that the image group is a group of captured images for the case. The first pose is, for example, the pose shown in FIG. 21(A), and the second pose is, for example, the pose shown in FIG. 21(B). As explained in FIGS. 24 and 25, based on the player's operation input information, etc., a character image is generated with a texture based on the first group of captured images, and a character image with a texture based on the second group of captured images is generated. Suppose that we perform processing to switch to image generation. By doing so, it becomes possible to perform a process of switching from a virtual space image in which an image of the character in the first pose is displayed to a virtual space image in which an image of the character in the second pose is displayed. For example, it becomes possible to perform a process of switching from a virtual space image in which an image of a character in the pose of FIG. 21(A) is displayed to a virtual space image in which an image of the character in the pose of FIG. 21(B) is displayed. In this case, during the switching period of the switching process, a texture based on the composite image IMXY of the captured images of the first and second captured image groups is generated and mapped to the primitive. In this way, an image of the character in a pose that interpolates the first pose and the second pose can be displayed during the switching period. For example, an image of a character in an intermediate pose, such as interpolating the pose of FIG. 21(A) and the pose of FIG. 21(B), can be displayed during the switching period. This enables processing such as motion interpolation, and enables smooth switching of character poses.

Note that the first captured image group is captured by capturing the object POB as shown in FIG. 21(A), and the second captured image is captured by capturing the object POB whose shape is deformed as shown in FIG. A switching process as shown in FIG. 27(B) may be performed using a composite image of the photographed images of the photographed image group. In this way, it becomes possible to generate a virtual space image in which the shape of the character changes smoothly.

Further, in this embodiment, game parameters of a game played by a player are set based on the analysis result of the object, and game processing is performed based on the set game parameters. FIG. 28 shows a flowchart for performing such processing.

First, an analysis process of the object is performed based on the photographed image and the like (step S41). For example, the type and shape of the object is identified by analyzing the silhouette of the object. Then, based on the analysis result of the object, game parameters of the game played by the player are set (step S42). For example, game parameters such as attack power, defense power, durability, movement ability, running ability, special ability, special skill, hit point, magic point, level, etc. of a character corresponding to the object are set. For example, initial parameter values, which are basic values of game parameters, are set. Then, game processing is executed based on the set game parameters (step S43). For example, a game in which a character or the like appears is processed based on the set game parameters. For example, a game start process, a game progress process, a game end process, a game score calculation process, etc. are performed.

In this way, it is possible to display a display object such as a character based on a photographed image of the target object, and to set game parameters used for processing a game in which a display object such as a character appears based on the analysis result of the target object. become able to. This makes it possible to create a game in which a display object corresponding to an object appears in virtual space, and also allows game parameters for the game's processing to be set based on the type of object, etc., which is the analysis result of the object. This makes it possible to create games that are more interesting than ever before.

Note that game parameters may be set based not only on analysis results such as silhouette analysis of objects but also on player information. For example, the values of game parameters set as initial values are changed based on the player's personal information.

Furthermore, in the present embodiment, when a second group of captured images is received after receiving the first group of captured images, it is determined whether the second group of captured images is a different group of captured images from the first group of captured images. to judge. Then, when it is determined that the captured images are different groups, the second captured image group is stored in the storage unit 170. For example, when an object is photographed by the photographing unit 162, the second photographed image group (second moving image) is a different photographic image from the first photographed image group (first moving image) photographed in the past. Determine whether it is an image group (video). Then, the second captured image group (second moving image) is stored in the storage unit 170, provided that the captured images are different groups. For example, if the second group of captured images is a group of captured images of the same object as the first group of captured images, it is not stored in the storage unit 170. FIG. 29 is a flowchart illustrating the storage process of such a group of photographed images.

First, a first group of captured images is received and stored in the storage unit 170 (step S51). Then, it is determined whether the second group of photographed images has been accepted (step S52). For example, it is determined whether a second group of photographed images has been acquired and accepted by photographing by the photographing unit 162. When the second group of captured images is accepted, it is determined whether the second group of captured images is a different group of captured images from the first group of captured images (step S53). If it is determined that the second group of captured images is different from the first group of captured images, the second group of captured images is stored in the storage unit 170 (step S54).

By performing the above-described processing, it is possible to prevent a situation in which a group of similar photographed images (moving images) is wasted in the storage unit 170. On the other hand, if the object appearing in the second group of captured images is an object whose pose has been changed from that of the object in the first group of captured images, an object to which an incidental object has been added, or an object whose shape has been deformed, It is determined that the second group of captured images is a different group of captured images from the first group of captured images. By doing this, as shown in FIG. 20, an appropriate texture is generated from the first and second photographed image groups and mapped to the primitive, and a virtual space in which a display object corresponding to the target object is displayed. It will be possible to generate images.

Furthermore, in this embodiment, when generating textures from the first and second groups of captured images, editing by the player on the captured images is accepted. That is, editing processing is performed to enable the player to edit the captured image. For example, as shown in FIG. 30, editing by the player is accepted for a photographed image selected from the first and second photographed image groups. For example, the player can perform editing such as correcting a photographed image, adding additional display objects such as marks and icons to the photographed image, and transforming the photographed image. Then, a texture is generated based on the edited photographic image, and the generated texture is mapped to a primitive, thereby generating a virtual space image including an image of the primitive.

In this way, instead of using the captured image of the object itself, the texture is generated using the captured image after the player has performed the desired editing, and the image of the primitive to which the texture is mapped is displayed. become able to. As a result, when displaying an image of a display object such as a character corresponding to a target object using a primitive image, the display object image can be displayed based on the corrected captured image, or additional display of marks, icons, etc. It becomes possible to display an image of a display object to which objects are added.

Further, in the present embodiment, common points may be consumed for photographing an object by the photographing unit 162 and for playing a game using a character or the like corresponding to the object. That is, the cost necessary for photographing the object and the cost necessary for playing the game are made common. For example, the player may be allowed to photograph the object by consuming points earned through game play. For example, game play is charged, and players pay real currency or virtual currency such as in-game currency to play the game. When the player obtains points in the game, the photographing unit 162 is permitted to photograph the object on the condition that the acquired points are consumed. For example, when a player plays a game for the first time, the player is allowed to take one shot of an object such as a plastic model that the player owns. The player then plays a game in which a character or the like corresponding to the photographed object appears. In this game, the player obtains points, and when the points reach a certain value or more, the player is allowed to take another picture. As a result, the player can, for example, create a plastic model (object in a broad sense) with different poses as shown in FIG. 21(B), a plastic model equipped with a weapon as shown in FIG. 22(A), or a plastic model equipped with a weapon as shown in FIG. It becomes possible to photograph a plastic model whose shape has been deformed as shown in B). Since characters corresponding to these plastic models can appear in the game and the game can be played, it is possible to create a game that does not get boring even when played over and over again.

Furthermore, in this embodiment, various correction processes and modification processes may be performed when generating textures from captured images. For example, textures may be generated from captured images using machine learning such as AI (Artificial Intelligence). For example, captured images obtained by a player photographing an object may be stored and accumulated in the storage unit 170, and machine learning such as AI may be performed based on the accumulated captured images. The learned model generated by machine learning may then be used to perform correction processing or correction processing on the photographed image to generate a texture, and the generated texture may be mapped to a primitive.

Although the present embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications can be made without substantively departing from the novelty and effects of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, in the specification or drawings, a term (plastic model, character, etc.) that is described at least once together with a different term with a broader meaning or the same meaning (object, display object, etc.) is not used anywhere in the specification or drawings. Can be replaced with a different term. Furthermore, the processing for acquiring a group of captured images, the processing for moving primitives, the processing for generating textures, the processing for generating virtual space images, etc. are not limited to those described in this embodiment. Included in the scope of disclosure.

POB object, PM, PMV, PMB, PME1, PME2 primitive,
TEX, TEXE1, TEXE2 texture, PT particle,
CH character, CHE, CHE1, CHE2 enemy character, CN cannon,
OB1~OB3 Object, IM photographed image, BG background,
RDR shooting direction range, VC virtual camera, VL line of sight direction,
PS viewpoint position, RL rail, VM view volume,
VN injection flame, BM beam, DCM shooting direction,
10 arrangement part, 12 rotation movement mechanism, 16 support part, 20 background part,
100 processing unit, 106 game processing unit, 108 editing processing unit, 110 acquisition unit,
111 analysis section, 112 movement processing section, 114 texture generation section,
120 image generation section, 130 sound generation section, 160 operation section, 162 photographing section,
170 storage unit, 176 photographed image information storage unit, 177 virtual space information storage unit,
178 drawing buffer, 180 information storage medium, 190 display unit, 192 sound output unit,
194 I/F section, 195 portable information storage medium, 196 communication section,
200 processing device, 210 game device, 220 photographing casing

Claims (15)

an acquisition unit that acquires a group of photographed images obtained by photographing the object from a plurality of photographing directions by the photographing unit;
a storage unit that stores the acquired photographic image group;
a movement processing unit that performs processing to move the primitive in the virtual space based on the movement control information;
A photographed image that is a two-dimensional image is selected from the group of photographed images based on the movement control information of the primitive, and an image of the object appearing in the selected photographed image is generated as a texture to be mapped to the primitive. A texture generation section;
By mapping the texture generated by the captured image selected based on the movement control information to the primitive, the texture that changes according to the movement control information is mapped to the primitive, and the texture is created in the virtual space. an image generation unit that generates a virtual space image visible from the virtual camera;
including;
The movement processing section
An image generation system characterized by performing a process of moving the primitive and the virtual camera according to the movement control information while the positional relationship between the primitive and the virtual camera is fixed. In claim 1,
The movement processing section
An image generation system characterized in that the positional relationship between the primitive and the virtual camera is fixed so that the primitive faces the virtual camera and the distance between the primitive and the virtual camera is a constant distance. In claim 1 or 2,
The movement processing section
Based on the movement control information, the primitive is rotationally moved about the position of the virtual camera while fixing the positional relationship between the primitive and the virtual camera, or the positional relationship between the primitive and the virtual camera is fixed. An image generation system characterized in that the primitive and the virtual camera are translated as one unit. an acquisition unit that acquires a group of photographed images obtained by photographing the object from a plurality of photographing directions by the photographing unit;
a storage unit that stores the acquired photographic image group;
a movement processing unit that performs processing to move the primitive in the virtual space based on the movement control information;
A photographed image that is a two-dimensional image is selected from the group of photographed images based on the movement control information of the primitive, and an image of the object appearing in the selected photographed image is generated as a texture to be mapped to the primitive. A texture generation section;
By mapping the texture generated by the captured image selected based on the movement control information to the primitive, the texture that changes according to the movement control information is mapped to the primitive, and the texture is created in the virtual space. an image generation unit that generates a virtual space image visible from the virtual camera;
including;
The movement processing section
performing movement processing of a second primitive whose relative positional relationship with respect to the primitive changes according to the movement control information;
The image generation unit includes:
An image generation system characterized in that an image including an image of the primitive to which the texture is mapped and an image of the second primitive is generated as the virtual space image. In claim 4,
The movement processing section
An image generation system characterized by performing a process of moving the second primitive based on the primitive from an initial arrangement position of the second primitive that is set based on an analysis result of the target object. In claim 4 or 5,
The image generation unit includes:
An image generation system characterized in that hidden surface removal processing of the second primitive is performed in accordance with the movement control information. In any one of claims 4 to 6,
An image generation system characterized in that the image of the second primitive is an effect image. In any one of claims 4 to 7,
The movement processing section
When the primitive rotates counterclockwise around the position of the virtual camera, the second primitive rotates counterclockwise with respect to the primitive with the primitive as a reference, and the primitive rotates counterclockwise with respect to the primitive. An image generation system characterized in that, when the camera position is rotated clockwise around the rotation center, the second primitive is rotated clockwise relative to the primitive with the primitive as a reference. In any one of claims 1 to 8,
The texture generation unit is
By performing seek control of the moving image constituted by the group of captured images based on the movement control information, the captured image is selected from the group of captured images, and the texture is generated based on the selected captured image. An image generation system characterized by: In any one of claims 1 to 9,
The image generation system is characterized in that the group of photographed images is a group of images in which a change in the photographing direction of the photographing unit with respect to the object is within a photographing direction range of 180 degrees or more. In any one of claims 1 to 10,
The acquisition unit includes:
Obtaining a group of photographed images for other display objects obtained by photographing other objects corresponding to the other display objects from a plurality of photographing directions,
The texture generation section includes:
Generating a texture for another display object based on a photographed image selected from a group of photographed images for the other display object in accordance with movement control information for the other display object;
The image generation unit includes:
An image characterized in that the virtual space image is generated by mapping a texture for the other display object that changes according to movement control information for the other display object onto a primitive for the other display object. generation system. In any one of claims 1 to 11,
The image generation unit includes:
An image generation system characterized in that an image in which particles are displayed around the primitive is generated as the virtual space image. Photography department and
a placement section where the object is placed;
a rotational movement mechanism that performs a rotational movement for changing the imaging direction of the imaging unit with respect to the object placed in the placement unit;
a support part that supports the object while floating in the air;
an acquisition unit that acquires a group of photographed images obtained by photographing the object from a plurality of photographing directions by the photographing unit;
a storage unit that stores the acquired photographic image group;
a movement processing unit that performs processing to move the primitive in the virtual space based on the movement control information;
a texture generation unit that generates a texture to be mapped to the primitive based on a captured image selected from the captured image group according to the movement control information of the primitive;
an image generation unit that maps the texture that changes according to the movement control information onto the primitive to generate a virtual space image visible from a virtual camera in the virtual space;
An image generation system comprising: an acquisition unit that acquires a group of photographed images obtained by photographing the object from a plurality of photographing directions by the photographing unit;
a storage unit that stores the acquired photographic image group;
a movement processing unit that performs processing to move the primitive in the virtual space based on the movement control information;
A photographed image that is a two-dimensional image is selected from the group of photographed images based on the movement control information of the primitive, and an image of the object appearing in the selected photographed image is generated as a texture to be mapped to the primitive. A texture generation section;
By mapping the texture generated by the captured image selected based on the movement control information to the primitive, the texture that changes according to the movement control information is mapped to the primitive, and the texture is created in the virtual space. As an image generation unit that generates a virtual space image that can be seen from a virtual camera,
make the computer work,
The movement processing section
A program characterized in that the program performs a process of moving the primitive and the virtual camera in accordance with the movement control information while the positional relationship between the primitive and the virtual camera is fixed. an acquisition unit that acquires a group of photographed images obtained by photographing the object from a plurality of photographing directions by the photographing unit;
a storage unit that stores the acquired photographic image group;
a movement processing unit that performs processing to move the primitive in the virtual space based on the movement control information;
A photographed image that is a two-dimensional image is selected from the group of photographed images based on the movement control information of the primitive, and an image of the object appearing in the selected photographed image is generated as a texture to be mapped to the primitive. A texture generation section;
By mapping the texture generated by the captured image selected based on the movement control information onto the primitive , the texture that changes according to the movement control information is mapped onto the primitive, and the virtual space As an image generation unit that generates a virtual space image that can be seen from a virtual camera,
make the computer work,
The movement processing section
performing movement processing of a second primitive whose relative positional relationship with respect to the primitive changes according to the movement control information;
The image generation unit includes:
A program for generating, as the virtual space image, an image including an image of the primitive to which the texture has been mapped and an image of the second primitive.

Images