JP2011215919A - Program, information storage medium and image generation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program, an information storage medium, and an image generation system or the like, allowing generation of an image in which the movement, or the like, of a possessed object or a portion of a player is reflected.SOLUTION: The image generation system includes an image information acquisition part acquiring image information from an image sensor; an information acquisition part performing image processing based on the image information, and acquiring at least one piece of direction information, color information and shape information regarding the possessed object or the portion of the player as information for rendering; and an image generating portion performing rendering processing, based on the information for the rendering that is at least the one piece of the direction information, the color information and the shape information, and generating an image.

Description

  The present invention relates to a program, an information storage medium, an image generation system, and the like.

  In recent years, game devices capable of performing game operations using a game controller with a built-in motion sensor have been gaining popularity in place of game controllers having conventional operation buttons and direction keys. According to the game device having such an operation interface, an intuitive operation input by the player is possible, and the game operation can be simplified. As a conventional technique of a game apparatus that enables such an intuitive interface, there is a technique disclosed in Patent Document 1, for example.

  However, in a game controller with a built-in motion sensor, the operation target of the player is the game controller, and the game operation cannot be performed using any player's possession or the player's hand or body. For this reason, an intuitive game operation in which the movement of the player's hand or the like is directly reflected on the screen cannot be realized, and there is a problem that improvement of the player's virtual reality is insufficient.

JP 2009-89629 A

  According to some aspects of the present invention, it is possible to provide a program, an information storage medium, an image generation system, and the like that can generate an image reflecting the player's belongings and the movement of a part.

  According to one aspect of the present invention, an image information acquisition unit that acquires image information from an image sensor and image processing based on the image information are performed, and direction information, color information, and shape information about a player's belongings or parts are stored. An information acquisition unit that acquires at least one as rendering information, and an image that generates an image by performing a rendering process based on the rendering information that is at least one of the direction information, the color information, and the shape information The present invention relates to an image generation system including a generation unit. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to one aspect of the present invention, at least one of direction information, color information, and shape information about an object or part of a player is acquired as rendering information by image processing based on image information from an image sensor. Then, an image is generated using the acquired rendering information. In this way, it becomes possible to reflect the direction information, color information, or shape information of the player's belongings or parts acquired by the image processing based on the image information in the rendering process. It is possible to generate an image reflecting the movement of the camera.

  In one aspect of the present invention, the image generation unit sets light source information of a light source arranged and set in an object space based on the rendering information, and rendering using the light source in which the light source information is set Processing may be performed to generate an image visible from the virtual camera in the object space.

  In this way, it is possible to generate an image by causing a light source that reflects the player's belongings and the movement of the part to appear in the object space, and it is possible to generate an image with a display mode that has not existed before. Become.

  In one aspect of the present invention, a character control unit that controls a character corresponding to the player is included (a computer is functioned as the character control unit), and the information acquisition unit receives the image information from the image sensor. Based on the acquired skeleton information and the position information of the character, the image generation unit acquires skeleton information that identifies the movement of the player that can be seen from the image sensor, based on the acquired skeleton information and the position information of the character. The arrangement position of the light source may be specified.

  In this way, the arrangement position of the light source in the object space can be set by effectively utilizing the skeleton information of the player, so that the processing efficiency can be improved.

  In one aspect of the present invention, the image information acquisition unit acquires depth information in which a depth value is set at each pixel position as the image information from the image sensor, and the information acquisition unit includes the depth information. Based on the above, at least one of the direction information and the shape information of the possessed item or the part of the player may be acquired as the rendering information.

  By using the depth information in this way, it is possible to efficiently obtain rendering information such as the player's belongings and part direction information.

  In the aspect of the invention, the information acquisition unit acquires a plurality of first to N-th direction information (N is an integer of 2 or more) as the direction information of the rendering information based on the depth information. The image generation unit may generate an image by performing a rendering process based on the first to Nth direction information that is the rendering information.

  In this way, an image can be generated by performing rendering processing reflecting the 1st to Nth direction information of the player's belongings or parts, so that it is possible to generate various images and images with high rendering effects. become.

  In one aspect of the present invention, the image generation unit sets a rendering range based on shape information of the possessed object or the part acquired based on the depth information, and performs the rendering process in the set rendering range. You may go.

  In this way, an image can be generated by setting a rendering range that reflects the shape information of the belongings or parts of the player, so that it is possible to generate a variety of images and images with a high effect.

  In one aspect of the present invention, the information acquisition unit acquires skeleton information that identifies an action of the player visible from the image sensor based on the image information from the image sensor, and the acquired skeleton information. The direction information about the possessed item or the part of the player may be acquired based on the information.

  In this way, it becomes possible to detect the movement of the player's belongings and parts based on the skeleton information and acquire the direction information of the belongings and parts, and the skeleton information can be effectively used.

  In one aspect of the present invention, the information acquisition unit acquires skeleton information that identifies an action of the player visible from the image sensor based on the image information from the image sensor, and the acquired skeleton information. The processing area of the image processing for acquiring the rendering information may be set based on the above.

  In this way, since it is only necessary to perform image processing in the specified processing region, it is possible to reduce the processing load of image processing, and it is possible to acquire rendering information with a small processing load.

  In one aspect of the present invention, a virtual camera control unit that controls a virtual camera arranged and set in the object space is included (a computer is caused to function as the virtual camera control unit), and the information acquisition unit includes the player The position information of the belongings or the part is acquired, and the virtual camera control unit sets the virtual camera in the object space based on the position information of the belongings or the part, and the image generation unit May generate an image visible from the virtual camera in the object space.

  In this way, the virtual camera can be arranged and set at a position in the object space corresponding to the position information of the player's belongings or parts, and an image that can be seen from the virtual camera in the object space can be generated. Accordingly, an image such as head tracking can be generated by a simple process, and various images and images with high rendering effects can be generated.

  In one aspect of the present invention, the virtual camera control unit arranges the virtual camera at a position corresponding to the position information of the possession or the part, and corresponds to the direction information of the possession or the part. You may perform control which orient | assigns the direction of the said virtual camera to the direction to do.

  In this way, the direction of the virtual camera can be directed in a direction corresponding to the direction of the player's belongings or parts, and an image that more reflects the movement of the player can be generated.

  In one embodiment of the present invention, the information acquisition unit acquires skeleton information that identifies the play action visible from the image sensor based on the image information from the image sensor, and the acquired skeleton information. The position information of the belongings or the part may be acquired based on the information.

  In this way, the position information of the player's belongings or parts can be acquired by effectively utilizing the skeleton information, and the virtual camera can be arranged and set at the position in the object space corresponding to the position information.

  According to another aspect of the present invention, an image information acquisition unit that acquires image information from an image sensor, and image processing based on the image information are performed, so that at least direction information and shape information about a player's belongings or parts are included. An information acquisition unit that acquires one as hit calculation information, a hit calculation unit that performs hit calculation processing based on the acquired hit calculation information, and generates an image based on the result of the hit calculation processing And an image generation system including an image generation unit. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to another aspect of the present invention, at least one of direction information, color information, and shape information about a player's belongings or parts is acquired as hit calculation information by image processing based on image information from an image sensor. The Then, hit calculation processing is performed using the acquired hit calculation information, and an image is generated based on the result of the hit calculation processing. In this way, it becomes possible to reflect the direction information, color information, or shape information of the player's possession or part acquired from the image information in the hit calculation process, and the movement of the player's possession or part etc. A reflected image can be generated.

  In another aspect of the present invention, the hit calculation unit may perform the hit calculation process on the target object based on a trajectory direction specified by the direction information of the hit calculation information.

  In this way, it is possible to determine whether or not a bullet or the like hits the target object using the trajectory direction specified by the direction information of the hit calculation information.

  In another aspect of the present invention, the image information acquisition unit acquires depth information in which a depth value is set at each pixel position as the image information from the image sensor, and the information acquisition unit includes the depth information. Based on the information, at least one of the direction information and the shape information of the possessed item or the part of the player may be acquired as the hit calculation information.

  By using the depth information in this way, it is possible to efficiently acquire hit calculation information such as the player's belongings and the direction information of the part.

  Moreover, in another aspect of the present invention, the information acquisition unit includes a plurality of first to Nth direction information (N is an integer of 2 or more) as the direction information of the hit calculation information based on the depth information. The hit calculation unit may perform the hit calculation process based on the first to Nth direction information that is the hit calculation information.

  In this way, an image can be generated by performing hit calculation processing reflecting the 1st to Nth direction information of the player's belongings or parts, so that it is possible to generate various images and images with high rendering effects. It becomes possible.

  In another aspect of the present invention, the hit calculation unit sets a hit range based on the possessed object acquired based on the depth information or the shape information of the part, and based on the set hit range, Hit calculation processing may be performed.

  In this way, it is possible to set the hit range reflecting the shape information of the belongings or parts of the player and perform hit calculation processing to generate an image.

  In another aspect of the present invention, the information acquisition unit acquires skeleton information that identifies the action of the player as seen from the image sensor based on the image information from the image sensor, and the acquired skeleton Based on the information, the direction information about the possessed item or the part of the player may be acquired.

  In this way, it becomes possible to detect the movement of the player's belongings and parts based on the skeleton information and acquire the direction information of the belongings and parts, and the skeleton information can be effectively used.

  In another aspect of the present invention, the information acquisition unit acquires skeleton information that identifies the action of the player as seen from the image sensor based on the image information from the image sensor, and the acquired skeleton Based on the information, a processing area for the image processing for obtaining the hit calculation information may be set.

  In this way, since it is only necessary to perform image processing in the specified processing region, it is possible to reduce the processing load of the image processing and to acquire hit calculation information with a small processing load.

  According to another aspect of the present invention, an image information acquisition unit that acquires image information from an image sensor, and information acquisition that acquires image information based on the image information and acquires position information about a player's belongings or parts. A virtual camera control unit that controls a virtual camera arranged and set in the object space, and an image generation unit, the virtual camera control unit is based on the position information of the possession or the part, The virtual camera is arranged and set in the object space, and the image generation unit is related to an image generation system that generates an image visible from the virtual camera in the object space. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to another aspect of the present invention, the virtual camera is arranged and set at a position in the object space corresponding to the position information of the player's belongings or parts so that an image visible from the virtual camera can be generated in the object space. Become. Accordingly, an image such as head tracking can be generated by a simple process, and various images and images with high rendering effects can be generated.

1 is a configuration example of an image generation system according to the present embodiment. 2A and 2B are explanatory diagrams of a method for acquiring color image information and depth information using an image sensor. FIGS. 3A and 3B are explanatory diagrams of a method for acquiring direction information and the like of the player's belongings as rendering information. 4A and 4B are explanatory diagrams of rendering processing using the acquired rendering information. FIGS. 5A and 5B are explanatory diagrams of a technique for reflecting shape information and color information of the player's belongings and the like in the rendering process. Explanatory drawing of the method of calculating | requiring the skeleton information of a player based on depth information. Explanatory drawing of the method of specifying the arrangement position etc. of a light source based on the skeleton information of a player. FIG. 8A and FIG. 8B are explanatory diagrams of another method for acquiring rendering information. FIGS. 9A and 9B are explanatory diagrams of a technique for acquiring a plurality of direction information as rendering information. FIG. 10A and FIG. 10B are explanatory diagrams of a method for acquiring direction information such as a player's belongings using the player's skeleton information. FIG. 11A and FIG. 11B are explanatory diagrams of a method for setting a processing area for image processing using skeleton information of a player. 12A to 12C are explanatory diagrams of a method for arranging and setting a virtual camera based on position information of a player's belongings and the like. FIGS. 13A and 13B are explanatory diagrams of a method for acquiring direction information of the player's belongings and the like as hit calculation information. The flowchart explaining the process of this embodiment. The flowchart explaining the process of this embodiment. The flowchart explaining the process of this embodiment. The flowchart explaining the process of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows an example of a block diagram of an image generation system (game device) of the present embodiment. Note that the configuration of the image generation system of the present embodiment is not limited to that shown in FIG. 1, and various modifications may be made such as omitting some of the components (each unit) or adding other components. .

  The operation unit 160 is for a player to input operation data, and functions thereof are direction instruction keys, operation buttons, analog sticks, levers, various sensors (such as an angular velocity sensor and an acceleration sensor), a microphone, or a touch panel type. This can be realized with a display.

  The operation unit 160 includes an image sensor realized by, for example, a color image sensor or a depth sensor. Note that the function of the operation unit 160 may be realized only by the image sensor.

  The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, and the like, and its function can be realized by a RAM (DRAM, VRAM) or the like. Then, the game program and game data necessary for executing the game program are held in the storage unit 170.

  An information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and functions thereof by an optical disk (CD, DVD), HDD (hard disk drive), memory (ROM, etc.), and the like. realizable. The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, in the information storage medium 180, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by an LCD, an organic EL display, a CRT, a touch panel display, an HMD (head mounted display), or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The auxiliary storage device 194 (auxiliary memory, secondary memory) is a storage device used to supplement the capacity of the storage unit 170, and can be realized by a memory card such as an SD memory card or a multimedia card.

  The communication unit 196 communicates with the outside (for example, another image generation system, a server, or a host device) via a wired or wireless network, and functions as a communication ASIC, a communication processor, or the like. It can be realized by hardware and communication firmware.

  Note that a program (data) for causing a computer to function as each unit of the present embodiment is obtained from an information storage medium of a server (host device) via an information storage medium 180 (or storage unit 170, auxiliary storage) via a network and communication unit 196. May be distributed to the device 194). Use of an information storage medium by such a server (host device) can also be included in the scope of the present invention.

  The processing unit 100 (processor) performs game processing, image generation processing, sound generation processing, and the like based on operation data from the operation unit 160, a program, and the like. The processing unit 100 performs various processes using the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, GPU, etc.), ASIC (gate array, etc.), and programs.

  The processing unit 100 includes an image information acquisition unit 102, an information acquisition unit 104, a game calculation unit 108, a hit calculation unit 110, an object space setting unit 112, a character control unit 114, a virtual camera control unit 118, an image generation unit 120, and sound generation. Part 130. The information acquisition unit 104 includes a skeleton information acquisition unit 105, and the character control unit 114 includes a movement processing unit 115 and a motion processing unit 116. The image generation unit 120 includes a light source setting unit 122. Various modifications may be made such as omitting some of these components or adding other components.

  The image information acquisition unit 102 acquires image information from the image sensor. For example, information on an image captured by the image sensor is stored in the image information storage unit 171 of the storage unit 170. Specifically, information on the color image captured by the color image sensor of the image sensor is stored in the color image information storage unit 172, and information on the depth image captured by the depth sensor of the image sensor is stored in the depth information storage unit 173. Is done. The image information acquisition unit 102 reads the image information from the image information storage unit 171 to acquire the image information.

  The information acquisition unit 104 acquires rendering information and hit calculation information based on the image information acquired by the image information acquisition unit 102.

  The game calculation unit 108 performs game calculation processing. Here, as a game calculation, a process for starting a game when a game start condition is satisfied, a process for advancing the game, a process for calculating a game result, or a process for ending a game when a game end condition is satisfied and so on.

  The hit calculation unit 110 performs hit calculation processing. For example, hit check processing or hit effect processing is performed. Specifically, the orbital direction of the bullet or the like and the hit check of the target object are performed, and when it is determined that the hit has occurred, a process of generating a hit effect is performed.

  The object space setting unit 112 performs an object space setting process in which a plurality of objects are arranged. For example, various objects (polygon, free-form surface or subdivision surface) representing display objects such as characters (people, animals, robots, cars, ships, airplanes, etc.), maps (terrain), buildings, courses (roads), trees, walls, etc. The object is configured to place and set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects. Specifically, the object data storage unit 175 of the storage unit 170 stores object data that is data such as the position, rotation angle, moving speed, and moving direction of the object (part object) in association with the object number. . The object space setting unit 112 performs a process of updating the object data for each frame, for example.

  The character control unit 114 controls a character that moves (moves) in the object space. For example, the movement processing unit 115 included in the character control unit 114 performs a process of moving a character (model object, moving object). For example, the character is moved in the object space based on operation information input by the player through the operation unit 160, a program (movement algorithm), various data, and the like. Specifically, a simulation process for sequentially obtaining character movement information (position, rotation angle, speed, or acceleration) every frame (for example, 1/60 second) is performed. Note that a frame is a unit of time for performing a movement process, a motion process, and an image generation process.

  The motion processing unit 116 included in the character control unit 114 performs motion processing (motion reproduction, motion generation) that causes the character to perform motion (animation). This motion processing can be realized by reproducing the motion of the character based on the motion data stored in the motion data storage unit 176.

  Specifically, the motion data storage unit 176 stores the position or rotation angle of each bone (each part object constituting the character) constituting the skeleton of the character (model object) around the three axes of the child bone relative to the parent bone. Motion data including the rotation angle). The model data storage unit 177 stores model data of a model object representing a character. The motion processing unit 116 reads this motion data from the motion data storage unit 176 and moves each bone (part object) constituting the skeleton based on the read motion data (by deforming the skeleton shape). Play the character's motion.

  The virtual camera control unit 118 performs control processing of a virtual camera (viewpoint, reference virtual camera) for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. Specifically, processing for controlling the position (X, Y, Z) or rotation angle (rotation angle about the X, Y, Z axis) of the virtual camera (processing for controlling the viewpoint position, the line-of-sight direction or the angle of view) I do.

  For example, when a character is photographed from behind using a virtual camera, the position or rotation angle (the direction of the virtual camera) of the virtual camera is controlled so that the virtual camera follows changes in the position or rotation of the character. In this case, the virtual camera can be controlled based on information such as the character position, rotation angle, or speed obtained by the movement processing unit 115. Alternatively, the virtual camera may be controlled to rotate at a predetermined rotation angle or to move along a predetermined movement path. In this case, the virtual camera is controlled based on the virtual camera data for specifying the position (movement path) or rotation angle of the virtual camera.

  The image generation unit 120 performs drawing processing based on the results of various processing (game processing and simulation processing) performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. Based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vector, α value, etc.) are created. Then, based on this drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is converted into image information in units of pixels such as a drawing buffer 179 (frame buffer, work buffer, etc.). Draw in a buffer that can be stored. Thereby, an image that can be seen from the virtual camera (given viewpoint) in the object space is generated. Note that the rendering process can be realized by a vertex shader process or a pixel shader process.

  The light source setting unit 122 included in the image generation unit 120 performs light source setting processing. For example, light source information such as the light source direction, light source color, and light source position is set. The light source information is stored in the light source information storage unit 178.

  Note that the image generation unit 120 may generate a so-called stereoscopic image. In this case, the left-eye virtual camera and the right-eye virtual camera are arranged and set using the position of the reference virtual camera and the inter-camera distance. The image generation unit 120 generates a left-eye image that can be seen from the left-eye virtual camera in the object space and a right-eye image that can be seen from the right-eye virtual camera in the object space. Then, using these left-eye image and right-eye image, stereoscopic vision may be realized by an eye separation spectacle method, a naked eye method using a lenticular lens, or the like.

  The sound generation unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates game sounds such as BGM, sound effects, or sounds, and outputs the game sounds to the sound output unit 192.

  In this embodiment, when the image information acquisition unit 102 acquires image information from the image sensor, the information acquisition unit 104 performs image processing based on the image information. Then, at least one of the direction information, the color information, and the shape information about the possession or part of the player (viewer) is acquired as rendering information.

  Here, the possession is a real world object possessed by the player. The part is a part of the player's body, such as a hand, a leg, an arm, a thigh, a head, or a chest. The player's possession or part direction information is information indicating the direction of the possession or part, such as the direction of the possession or part or the normal direction of the surface of the possession or part. The color information of the belonging or part is information representing the color of the belonging or part. The shape information of the belonging or part is information representing the shape of the possession or part (information that schematically represents the shape), such as a planar shape such as a polygon or a circle, a polyhedron, a sphere, a cylinder, or a cone. Three-dimensional shape. The rendering information is information used for image rendering processing.

  The image generation unit 120 performs rendering processing (shading processing) based on rendering information (shading information) that is at least one of the direction information, color information, and shape information, and generates an image. For example, the rendering process is performed by setting the direction information, color information, and shape information of the belongings or parts as parameters of the rendering process, and for example, an image that can be seen from the virtual camera in the object space is generated.

  Specifically, the image generation unit 120 sets light source information (light source attribute information) of a light source arranged and set in the object space based on the rendering information. For example, the direction of the light source is set based on the direction information of the belonging or part, the light source color is set based on the color information, and the light source shape is set based on the shape information. Then, the image generation unit 120 performs a rendering process using the light source in which the light source information is set in this way, and generates an image that can be seen from the virtual camera in the object space. For example, shading processing is performed based on the light source information of the light source and the normal vector of the polygon (primitive surface) of the object to generate an image. In this case, shading processing such as Gouraud shading and Phong shading is performed using various illumination models such as Lambert, Phong, and Blinn.

  The character control unit 114 controls a character corresponding to the player. Here, the character becomes an object that is actually displayed in the third person viewpoint, but becomes a virtual object that is not displayed on the display unit 190 in the first person viewpoint.

  Then, the information acquisition unit 104 acquires the player's skeleton information based on the image information from the image sensor. For example, the skeleton information acquisition unit 105 of the information acquisition unit 104 performs image processing based on image information such as depth information and obtains skeleton information of the player. The obtained skeleton information is stored in the skeleton information storage unit 174.

  Here, the skeleton information is information for specifying, for example, the action of the player as seen from the image sensor. Specifically, this skeleton information has a plurality of joint position information corresponding to a plurality of joints of the player, and each joint position information is constituted by three-dimensional coordinate information. What connects each joint becomes a bone, and a skeleton is formed by connecting a plurality of bones.

  When the skeleton information is acquired in this way, the image generation unit 120 specifies the arrangement position of the light source in the object space based on the acquired skeleton information and the position information of the character. For example, based on the position information (representative position information) of the character in the world coordinate system of the object space and the joint position information of the skeleton in the local coordinate system (position information of the joint corresponding to the possession or part), Specify the location.

  The image information acquisition unit 102 acquires depth information and color image information as image information from the image sensor. This depth information is image information in which a depth value (depth value) is set at each pixel position, for example. Based on the acquired depth information, the information acquisition unit 104 acquires at least one of direction information and shape information of the belongings or parts of the player as rendering information. For example, the direction and shape of the player's belongings and parts are determined based on the depth information and color image information, and acquired as rendering information.

  In this case, the information acquisition unit 104 may acquire a plurality of first to Nth direction information (N is an integer of 2 or more) as direction information of the rendering information based on the depth information. For example, a plurality of pieces of direction information are acquired as direction information of the player's belongings and parts. Then, the image generation unit 120 performs rendering processing based on the first to Nth direction information that is the rendering information, and generates an image. That is, a rendering process reflecting a plurality of direction information is performed, and an image to be displayed on the display unit 190 is generated. In other words, the image generation unit 120 sets the rendering range based on the possessed or part shape information acquired based on the depth information, performs the rendering process in the set rendering range, Generate. Taking a rendering process using a light source as an example, an irradiation range of light source light, which is a rendering range, is set based on the shape information of an object or part, and a rendering process (shading process) is performed in the irradiation range.

  Further, the information acquisition unit 104 may acquire skeleton information based on image information from the image sensor, and may acquire direction information about the player's belongings or parts based on the acquired skeleton information. For example, the direction connecting the first and second joints corresponding to the hand of the skeleton is set as direction information about the player's belongings or parts.

  Further, the information acquisition unit 104 may set a processing area for image processing for acquiring rendering information based on the acquired skeleton information. For example, the position of the player's belongings or part is specified based on the skeleton information, and an area of a predetermined size (area smaller than the screen) including the specified position is set as the processing area. The information acquisition unit 104 performs image processing using the depth information and color image information in the set processing region, and acquires direction information, color information, and shape information of the player's belongings or parts.

  Further, when the information acquisition unit 104 acquires the position information of the possessed item or part of the player, the virtual camera control unit 118 sets the virtual camera in the object space based on the position information of the possessed item or part. Also good. Specifically, the virtual camera is arranged at a position corresponding to the position information of the player's belongings or part, and the direction of the virtual camera (gaze direction) is directed to the direction corresponding to the direction information of the possession or part. I do. Alternatively, the direction of the virtual camera may be controlled to face the center of the screen. Then, the image generation unit 120 generates an image that can be seen from the virtual camera in the object space. By doing so, it is possible to generate an image such as head tracking. In this case, the information acquisition unit 104 may acquire the position information of the player's belongings or parts based on, for example, skeleton information.

  Further, the information acquisition unit 104 performs image processing based on image information from the image sensor, and acquires at least one of direction information and shape information about the player's belongings or parts as hit calculation information. The hit calculation unit 110 performs hit calculation processing based on the acquired hit calculation information. For example, hit calculation processing (hit check processing) reflecting the direction and shape of the player's belongings or parts is performed. Specifically, based on the trajectory direction (hit direction) specified by the direction information of the hit calculation information, hit calculation processing is performed on the target object (an object that becomes a hit target of hit objects such as bullets). Then, the image generation unit 120 generates an image based on the result of the hit calculation process. For example, an image of a hit object such as a bullet is generated, or when a hit object hits a target object, an image for producing a hit effect is generated.

  Further, when the image information acquisition unit 102 acquires depth information as image information, the information acquisition unit 104 obtains at least one of direction information and shape information of the player's belongings or parts based on the acquired depth information. You may acquire as information for hit calculation. For example, based on depth information and color image information, the player's belongings and the direction and shape of the part are determined and acquired as hit calculation information.

  When the information acquisition unit 104 acquires a plurality of first to Nth direction information as the direction information of the hit calculation information based on the depth information, the hit calculation unit 110 includes the first to Nth direction information. Based on the above, hit calculation processing is performed. That is, hit calculation processing reflecting a plurality of pieces of direction information is performed, and an image to be displayed on the display unit 190 is generated. For example, hit objects such as bullets fly in a plurality of directions, and hit calculation processing with the target object is performed. In other words, the hit calculation unit 110 sets a hit range based on the possessed or part shape information acquired based on the depth information, and performs hit calculation processing based on the set hit range. For example, when the target object is located within the hit range set by the shape of the belonging or part, it is determined that a bullet or the like has hit the target object.

  In the hit calculation process, the information acquisition unit 104 acquires direction information about the player's belongings or parts based on the skeleton information as in the rendering process. Further, based on the skeleton information, a processing area for image processing for obtaining hit calculation information is set.

  Further, when the information acquisition unit 104 performs image processing based on the image information and acquires position information about the possessed item or part of the player, the virtual camera control unit 118 uses the object information based only on the acquired position information. A virtual camera may be arranged and set in the space. In this case, the direction in which the virtual camera faces may be controlled so that it always faces in a certain direction or toward the center of the screen.

2. Next, the method of this embodiment will be described in detail.

2.1 Rendering processing based on direction information of belongings Generally, game devices perform game operations by operating buttons and levers of a game controller. There is also a game apparatus in which a motion controller (six-axis sensor) is provided in the game controller, and a player performs a game operation by moving the game controller itself.

  However, such a game device with an operation interface requires a game controller for the game operation, and cannot implement a game operation that directly reflects a gesture such as a player's hand movement. I can not improve one more.

  Therefore, in this embodiment, an operation interface that allows a player to input an operation based on image information captured by the image sensor is employed.

  For example, in FIG. 2A, an image sensor ISE realized by a depth sensor (infrared sensor or the like) or a color image sensor (RGB sensor such as a CCD or CMOS sensor) is installed at a position corresponding to the display unit 190. The image sensor ISE is installed so that, for example, its imaging direction (optical axis direction) is a direction from the display unit 190 toward the player PL, and color image information of the player PL viewed from the display unit 190 side, Image depth information. The image sensor ISE may be built in the display unit 190 or may be prepared as an external component.

  Then, using the image sensor ISE in FIG. 2A, color image information and depth information as shown in FIG. 2B are acquired. For example, with color image information, color information of the player PL and the surrounding landscape can be obtained. On the other hand, in the depth information, the depth values (depth values) of the player PL and the surrounding scenery are obtained as gray scale values, for example. For example, the color image information is image information in which a color value (RGB) is set at each pixel position, and the depth information is image information in which a depth value (depth value) is set at each pixel position. The image sensor ISE may be a sensor in which a depth sensor and a color image sensor are provided separately, or may be a sensor in which a depth sensor and a color image sensor are combined.

  Various methods can be assumed as publicly known depth information acquisition methods. For example, by projecting light such as infrared rays from an image sensor ISE (depth sensor), the depth information is obtained by detecting the reflection intensity of the projected light and the time (Time Of Flight) until the projected light returns, The shape of an object such as the player PL viewed from the position of the image sensor ISE is detected. Specifically, the depth information is expressed as grayscale data that becomes brighter for an object close to the image sensor ISE and darker for an object far from the image sensor ISE, for example. The depth information acquisition method can be variously modified. For example, an image sensor such as a CMOS sensor may be used to acquire depth information (distance information to the subject) simultaneously with color image information. . Further, the depth information may be acquired by a distance measuring sensor using ultrasonic waves or the like.

  In the present embodiment, image processing based on image information from the image sensor ISE is performed, and direction information, color information, shape information, or the like of the possessed item or part of the player PL is acquired as rendering information.

  For example, in FIG. 3A, the player PL has a plate-like (cuboid, rectangular) possession TH, and the surface of the possession TH faces the display unit 190 side. In this case, in this embodiment, image information such as color image information and depth information as shown in FIG. 3B is acquired by the image sensor ISE.

  Then, based on the color image information of FIG. 3 (B), the color information and shape information of the belonging TH (or part) are specified. For example, according to the color image information of FIG. 3B, the color of the belonging TH, the rectangular parallelepiped shape of the possession TH, or the rectangular shape of the surface can be specified. Further, the direction DTH (normal direction) of the surface of the belonging TH can be specified by looking at how the image is parsed.

  Further, the direction information (DTH) and shape information (rectangular, rectangular) of the belonging TH (or part) can also be specified using the depth information of FIG. That is, according to the depth information, it is possible to extract the three-dimensional shapes of the player PL and the belongings TH when viewed from the image sensor ISE. Therefore, for example, the direction DTH (normal direction) in which the surface of the possession TH faces can be specified based on the depth information (such as the slope of the depth value) in the area of the rectangular surface of the possession TH. Moreover, the rectangular parallelepiped shape and the rectangular shape of the possession TH can be specified based on the depth information.

  In this embodiment, the rendering process is performed using the direction information, color information, or shape information of the belonging TH specified in this way as rendering information. In the following description, the case where the rendering process and the hit calculation process are performed using the direction information, color information, or shape information of the belongings TH of the player PL will be mainly described. The rendering process and the hit calculation process may be performed using the direction information, color information, or shape information of the part.

  For example, in FIG. 4A, the character CH corresponds to the player PL, and is an object that moves in the object space by the operation of the player PL. When the character CH is in the third person viewpoint, the whole body (or a part of the body including the head) is displayed on the screen SC of the display unit 190. On the other hand, in the first person viewpoint, the character CH is a virtual object that is not displayed on the screen SC. In the case of the first person viewpoint, only a part of the character CH's hand or the like may be displayed. That is, the first person viewpoint is a viewpoint setting in which at least the head of the character CH is not displayed.

  In FIG. 4A, the light source LS is set at a position corresponding to the possessed item TH of the player PL. That is, in FIG. 3A, the player PL has the possession TH in the right hand, and correspondingly, in FIG. 4A, the light source LS such as a flashlight light is arranged and set in the right hand of the character CH. Then, the light source direction DLS of the light source LS is set in a direction corresponding to the direction DTH of the belongings TH acquired in FIG. Then, the shading process of the target object TOB is performed by the light RY irradiated in the light source direction DLS, and an image is rendered. As a result, as shown in FIG. 4B, an image shaded by the light RY from the light source LS or the like is generated and displayed on the display unit 190.

  In the present embodiment, an image reflecting the color information and shape information of the belongings TH is generated. For example, in FIG. 5A, since the belonging TH has a disk shape, a light source LS having a circular light irradiation surface is set, and an image of the light RY corresponding to the light source LS or a shading image by the light RY. Is generated. Moreover, since the color of the possession TH is red, the color of the light RY is also red, and the image of the target object TOB irradiated with the light RY is also an image irradiated with red light.

  On the other hand, in FIG. 5B, since the belonging TH has a rectangular parallelepiped shape, a light source LS having a rectangular light irradiation surface is set, and an image of the light RY corresponding to the light source LS and a shading image by the light RY are displayed. Generated. Further, since the color of the possession TH is blue, the color of the light RY is also blue, and the image of the target object TOB irradiated with the light RY is also an image irradiated with blue light.

  In this embodiment, as shown in FIG. 6, skeleton information for specifying the operation of the player PL is acquired based on image information such as depth information shown in FIG. In FIG. 6, position information (three-dimensional coordinates) of the skeleton joints C0 to C19 is acquired as the skeleton information. These joints C0 to C10 correspond to the joints of the player PL reflected on the image sensor ISE. When the whole body of the player PL is not reflected on the image sensor ISE, skeleton information having position information on joints only for the reflected portion may be generated.

  For example, if the depth information in FIG. 3B is used, the three-dimensional shape of the player PL or the like viewed from the image sensor ISE side can be specified. Further, by combining color image information, a region of a part such as the face of the player PL can be specified by face image recognition or the like. Therefore, each part of the player PL is estimated based on the information on the three-dimensional shape and the joint position of each part is estimated. Then, based on the two-dimensional coordinates of the pixel position of the depth information corresponding to the estimated joint position and the depth information set to the pixel position, the three-dimensional coordinate information of the joint position of the skeleton is obtained, as shown in FIG. Get skeleton information.

  By using such skeleton information, the operation of the player PL gesture and the like can be specified in real time, and an unprecedented operation interface environment can be realized. Further, this skeleton information has a high affinity with the motion data of the character CH arranged in the object space. Accordingly, the character CH can be moved in the object space by using the skeleton information as, for example, motion data.

  In this embodiment, the arrangement position of the light source LS is specified using this skeleton information. Specifically, in FIG. 7, the position information PRF (representative position information) of the character CH corresponding to the player PL is obtained. For example, in the present embodiment, the movement of the player PL instructing the movement of the character CH is detected by the image sensor ISE, or the character CH is moved in the object space based on operation information from the game controller. By this movement calculation, the position information PRF of the character CH in the world coordinate system of the object space is sequentially obtained. Based on the position information PRF in the world coordinate system and the joint position information in the local coordinate system of the skeleton information obtained as shown in FIG. 6, the position of the hand of the character CH corresponding to the player PL is determined. Can be sought. Then, as shown in FIG. 7, the obtained position of the hand or the like is set as the arrangement position PLS of the light source LS. Then, rendering processing (shading processing) is performed using the light source LS arranged in the PLS.

  In this way, the arrangement position PLS of the light source LS can be specified by effectively utilizing the skeleton information that specifies the operation of the player. Then, the light source LS is arranged at the position of the belonging TH held by the player PL, and an image is generated as if the surroundings of the character CH corresponding to the player PL are illuminated by the light from the light source LS. become able to.

  In the above description, the method for acquiring the direction information, shape information, and the like of the belongings and parts of the player PL using the depth information shown in FIG. 3B has been described, but the present embodiment is not limited to this. For example, only the color image information may be used to acquire the player's belongings, part direction information, and the like.

  For example, in FIG. 8A, a member on which a special image pattern is drawn is prepared as the possession TH of the player PL. In the belonging TH of FIG. 8A, the surface is divided into four areas, and different image patterns are drawn for the respective areas. Then, the player PL possesses the possessed item TH on which such a special image pattern is drawn, and images it with the image sensor ISE, for example, to obtain color image information as shown in FIG. 8B. Can do. Then, based on the color image information, the direction information of the player's belongings TH is specified. That is, the direction information (direction of the surface) of the belonging TH may be specified based on the arrangement position of the image pattern in the color image information, the degree of parsing of the image of the possession TH, and the like.

  Moreover, in this embodiment, you may acquire several direction information (1st-Nth direction information) as direction information, such as belongings TH.

  For example, in FIG. 9A, only one direction DTH is acquired as the direction information of the belonging TH. That is, the normal direction of the surface of the belonging TH is acquired as the direction DTH.

  On the other hand, in FIG. 9B, a plurality of directions DTH1, DTH2, and DTH3 (first to Nth direction information) are acquired as the direction information of the belonging TH. When a plurality of directions DTH1 to DTH3 are acquired as the direction information of the belonging TH in this way, the rendering process may be performed using the plurality of directions DTH1 to DTH3. In other words, the rendering process is performed using the rendering range RRD set by the plurality of directions DTH1 to DTH3.

  For example, it is assumed that the belonging TH is formed of a member that can freely deform its shape. In this case, in the shape before deformation in FIG. 9A, spotlight light in one irradiation direction corresponding to the direction DTH is emitted from the light source LS in FIG. 4A.

  On the other hand, when the belonging TH is bent and deformed by the player PL as shown in FIG. 9B, light that diffuses in a range corresponding to a plurality of directions DTH1 to DTH3 is emitted from the light source LS in FIG. Try to irradiate.

  If it does in this way, it will become possible to generate the light of a different irradiation form in a game picture using one possession TH. As a result, various game effects are possible, and the effects of the game effects can be improved.

  In order to reflect the shape deformation of the belongings TH as shown in FIGS. 9A and 9B in the game image, it is desirable to use depth information as shown in FIG. That is, if the depth information is used, information in the depth direction of the belonging TH can be obtained, so that the shape deformation as shown in FIGS. 9A and 9B can be easily recognized.

  For example, before the shape deformation of FIG. 9A, one direction DTH is obtained as the direction information of the belonging TH, and an image of spot light in the light source direction corresponding to this direction DTH can be generated. On the other hand, when the player bends the belonging TH, the depth value at each position on the surface of the possession TH is inclined when viewed from the image sensor ISE. Therefore, a plurality of directions DTH1 to DTH3 can be acquired using this depth information as shown in FIG. 9B. As a result, the rendering process is performed with the light in the irradiation ranges (rendering range and shading range) corresponding to the plurality of directions DTH1 to DTH3, and the object is shaded by the image of the light in the irradiation range or the light in the irradiation range. Attached images can be generated. Therefore, when the player PL transforms the possession TH into various shapes, the light image and the irradiation position image in the game image also change variously, so that a game image with a high effect can be generated. Become.

  In the present embodiment, the direction information about the player's belongings and parts may be acquired based on the skeleton information acquired as shown in FIG.

  For example, in FIG. 10A, the movement of the player PL is acquired as information on the skeleton SK. In this case, based on the positions of the joints C6, C7, etc. corresponding to the hand of the player PL, the direction of the belongings TH, etc. is specified, and the direction is set as the light source direction DLS. Similarly, in FIG. 10B, the direction of the belongings TH is specified based on the positions of the joints C6, C7, etc. corresponding to the hand of the player PL, and the direction is set as the light source direction DLS. In this case, the positions of the joints C6 and C7 can be acquired based on the skeleton information.

  In this way, when the player PL changes the direction in which the hand faces and changes the direction in which the belonging TH faces, the change in the direction can be detected based on the skeleton information. That is, it can be detected based on the positions of the joints C6, C7, etc. in the skeleton information. Then, according to the change in the direction in which the belonging TH faces, the light source direction DLS also changes as shown in FIGS. 10A and 10B, and the irradiation direction of the light RY from the light source LS also changes. To come. Therefore, when the player PL changes the direction of the hand and changes the direction of the belonging TH, the direction of light in the game image also changes. Thus, the player PL can see various directions desired by the player PL with light. In other words, it is possible to effectively utilize the skeleton information acquired using the depth information, thereby realizing an unprecedented game effect.

  In the present embodiment, a processing area for image processing may be specified using skeleton information acquired as shown in FIG.

  For example, in FIG. 11A, the position of the joint C7 or the like corresponding to the hand of the player PL is specified based on the skeleton information. Then, the image region including the specified position such as C7 is set as the image processing region RPC, and image processing is performed on the depth information and color image information in the set processing region RPC. That is, in FIG. 3 (A) and FIG. 3 (B), the color image information and depth information in the set processing region RPC are obtained when obtaining the direction information, color information, and shape information of the belongings and parts of the player PL. Then, image processing is performed.

  In this way, it is not necessary to perform image processing in all areas, and image processing can be performed in a processing area RPC smaller than the screen size. Therefore, it is possible to reduce the processing load of image processing, and it is possible to acquire rendering information such as belongings and part direction information with a small processing load.

  According to the method of the present embodiment described above, when the player moves with his belongings or moves a part such as a hand, the direction information, color information and shape information are acquired as rendering information. Then, a rendering process using the rendering information is performed, and an image to be displayed on the player is generated. Accordingly, the movement of the player is intuitively reflected in the game image, and a game image that has never been possible can be generated.

  In this case, if the player holds an appropriate belonging around him and moves it, a light source reflecting the shape and color of the belonging is set in the object space, and an image of light from the light source, An image in which an object is illuminated with the light is generated. For example, as shown in FIGS. 5A and 5B, if the object is a red object, a red light image or an irradiation image thereof is generated. If the object is a blue object, a blue light image is generated. And its irradiation image is generated. Accordingly, it is possible to produce a game effect that could not be realized in the conventional game.

  If the skeleton information described with reference to FIG. 6 is used, the arrangement position of the light source can be easily specified as shown in FIG. 7, and the light source direction can be easily set as shown in FIGS. 10A and 10B. Can be identified. Furthermore, as described with reference to FIGS. 11A and 11B, the processing area of the image processing for specifying the direction information of the belongings can be limited to a narrow area, and the processing load can be reduced.

  Moreover, if the direction information of the possession is acquired using the depth information as in the present embodiment, the shape of the possession or the part changes as shown in FIGS. 9A and 9B. In addition, the light irradiation range and the like are changed, and various game images can be generated.

2.2 Arrangement of virtual camera based on position information of belongings A technique called head tracking has been known. In this head tracking, the direction of the player's face and the direction of the line of sight are detected by a sensor or the like, and an image corresponding to the direction that the player faces is generated and displayed on the display unit.

  However, in the conventional head tracking method, it is necessary for the player to wear a device for detecting the direction of the face and the direction of the line of sight on the head, which is complicated for the player. Also, there is a problem that the device for detecting the orientation of the face becomes large-scale.

  Therefore, in this embodiment, the position of the player's belongings or parts is detected from the image information from the image sensor, and a virtual camera is arranged at the detected position to generate an image similar to head tracking.

  For example, in FIG. 12A, based on the image information from the image sensor ISE, the position information of the possessed item TH (or a part such as a hand) of the player PL is acquired. In FIG. 12A, the player holds the belonging TH such as a tissue box as a camera.

  Then, as shown in FIG. 12B, the virtual camera VC is arranged in the object space based on the acquired position information of the belonging TH (or part). That is, the virtual camera VC is arranged at a position in the object space corresponding to the position of the belonging TH in the real world. Then, an image that can be seen from the virtual camera VC in the object space is generated and displayed on the display unit 190.

  In this way, as shown in FIG. 12C, when the player PL looks at the personal belongings TH like a camera and looks into the screen SC of the display unit 190, an image visible from the camera is displayed. It becomes like this. Therefore, an image of the place is displayed by looking at the place desired by itself in the game space using a camera as in head tracking. Note that an image that can be viewed from the virtual camera VC may be displayed on a sub screen provided on the main screen of the display unit 190.

  Further, the direction of the virtual camera VC may be set, for example, always in a direction orthogonal to the virtual screen or in a direction toward the center of the virtual screen. Alternatively, the direction in which the belonging TH (or part) faces is detected based on the image information from the image sensor ISE as described above, and the direction (gaze direction) of the virtual camera VC is set in a direction corresponding to the detected direction. May be.

  Further, the position information of the belonging TH may be obtained based on the skeleton information described with reference to FIG. Specifically, as described with reference to FIG. 7, the position of the character CH's hand and the like are obtained based on the position PRF of the character CH and the skeleton information, and the virtual camera VC is arranged at that position. In this way, the arrangement position of the virtual camera VC can be obtained by a simple process.

2.3 Hit calculation process based on direction information of belongings The above has described the case where rendering processing is performed using the direction information of the player's belongings or parts as rendering information. As the calculation information, hit calculation processing may be performed.

  For example, in FIG. 13A, image processing based on the image information from the image sensor ISE is performed, and at least one of the direction information and the shape information about the possessed item TH (or part) of the player PL is used as hit calculation information. To be acquired. For example, the direction DTH of the belongings TH is acquired as hit calculation information.

  In this case, in the present embodiment, hit calculation processing is performed based on the hit calculation information. For example, in FIG. 13B, the character CH corresponding to the player PL moves in the object space and encounters an enemy target object TOB. At this time, as shown in FIG. 13 (A), when the player PL points the possession TH in the direction of the target object TOB, as shown in FIG. 13 (B), the weapon possessed by the character CH corresponding to the possession TH. A bullet is fired from the AOB and fired at the target object TOB. That is, bullets (handguns and ray gun bullets) are fired in the trajectory direction DHI specified by the direction DTH of the possessed TH in FIG. Then, hit calculation processing for performing a hit check between the bullet trajectory direction DHI and the target object TOB is performed, and an image is generated based on the result of the hit calculation processing. Specifically, when it is determined that the bullet has hit the target object TOB, an effect image indicating that the bullet has hit the target object TOB is generated.

  In this way, the player can take appropriate possessions around him and point it in the direction of the target object so that bullets are fired from weapons such as handguns and ray guns that correspond to the possessions. , Hit the enemy target object. Therefore, an intuitive and easy-to-understand operation interface environment can be provided to the player. Further, it is possible to realize a shooting game as shown in FIGS. 13A and 13B without using a dedicated game controller.

  In addition, according to the present embodiment, hit calculation processing reflecting the player's belongings and the shape of the part can also be realized. For example, each shape of the belongings or parts and each weapon are associated in advance. Then, the shape of the player's belonging or part is recognized based on the image information from the image sensor. Then, a weapon is selected based on the result of shape recognition, an image of the selected weapon is displayed on the game image, and a bullet from the weapon is fired onto the target object. For example, if the object has a short shape, the character is held by a handgun, and an image in which a bullet is fired from the handgun is generated. On the other hand, if the possession has a long shape, the character is caused to possess a rifle, and an image in which a bullet is fired from the life is generated. In this way, various game images can be generated according to the shape of the belongings possessed by the player, and an unprecedented game effect can be realized.

  In the hit calculation method shown in FIGS. 13A and 13B, as in the case of FIG. 3B, the direction information of the player's belongings or parts based on the depth information from the image sensor. It is desirable to acquire shape information as hit calculation information. In this way, it becomes possible to specify the direction information and shape information of the belongings and parts more accurately and efficiently.

  Similarly to the cases of FIGS. 9A and 9B, a plurality of direction information (first to Nth direction information) is obtained as the direction information of the hit calculation information based on the depth information from the image sensor. The hit calculation processing may be performed based on the plurality of pieces of direction information obtained. In other words, a hit range (RRD in FIG. 9B) is set based on the possessed or part shape information acquired based on the depth information, and hit calculation processing is performed based on the set hit range. I do.

  In this way, when the possession has a shape as shown in FIG. 9A, for example, a bullet is fired in one direction, and when it has a shape as shown in FIG. An image in which a bullet is fired in the direction of can be generated. Alternatively, in the case of a light gun, if the possession has the shape of FIG. 9 (A), one elongated light beam is emitted, and in the case of the shape of FIG. 9 (B), it extends to the RRD hit range. Such an image can be generated.

  Similarly to the case of FIGS. 10A and 10B, the player's skeleton information may be acquired, and the direction information about the player's belongings or parts may be acquired based on the acquired skeleton information. Good.

  In this way, for example, when the joints C6 and C7 of the skeleton information are in the positions as shown in FIG. 10A, bullets are fired in the downward direction DLS, and the joints C6 and C7 are shown in FIG. ), It is possible to generate an image in which a bullet is fired in the upward direction DLS. Accordingly, the trajectory direction of the bullet can be specified by a simple process using the skeleton information, and the process can be made efficient.

  Similarly to the cases of FIGS. 11A and 11B, the skeleton information of the player is acquired, and based on the acquired skeleton information, a processing area for image processing for acquiring hit calculation information is set. It may be set. For example, when the player has his belongings, a processing area including the position of the player's hand is set based on the skeleton information. Then, image processing is performed on the depth information and color image information in the set processing region, and direction information and shape information of the possessed item (or part) are specified.

  In this way, the processing area for image processing can be limited to a small size, so that the processing load can be reduced.

2.4 Detailed Processing Example Next, a detailed processing example of the present embodiment will be described with reference to the flowcharts of FIGS. FIG. 14 is a flowchart of a process for generating an image using the direction information of the player's belongings as rendering information.

  First, as described in FIGS. 3A and 3B, image information such as color image information and depth information is acquired by the image sensor (step S1). Then, based on the acquired depth information and the like, the player's skeleton information is obtained (step S2).

  Next, as described with reference to FIGS. 11A and 11B, the position of the player's belongings or a part such as a hand is specified based on the skeleton information, and the image region including the specified position is subjected to image processing. (Step S3). Then, based on the depth information, color image information, and the like in the set processing region, direction information, color information, shape information, and the like of the player's belongings or parts are obtained as rendering information (step S4). Then, a rendering process is performed using the obtained rendering information to generate an image to be displayed on the display unit.

  FIG. 15 is a flowchart of processing for generating an image by setting the light source direction based on the direction information of the player's belongings.

  Steps S11 to S14 in FIG. 15 are the same as steps S1 to S4 in FIG. After obtaining the rendering information in step S14, the arrangement position of the light source is set based on the character position information and skeleton information as described in FIG. 7 (step S15). The light source direction is set based on the direction information of the player's belonging or part, and the light source color is set based on the color information of the possession or part (steps S16 and S17). Then, rendering processing (shading processing) is performed using the light source for which the light source information is set, and an image is generated (step S18). As a result, an image as shown in FIG. 4B can be generated.

  FIG. 16 is a flowchart of processing for generating an image by setting a virtual camera based on the position information of the player's belongings.

  Steps S21 and S22 in FIG. 16 are the same as steps S1 and S2 in FIG. After the skeleton information is obtained in step S22, as described with reference to FIG. 12A, the position of the player's belongings or a part such as a hand is specified based on the skeleton information (step S23). Then, as described in FIG. 12B, the virtual camera is arranged at a position in the object space corresponding to the specified position (step S24). Then, an image that can be seen from the virtual camera in the object space is generated (step S25). As a result, an image such as head tracking can be generated by a simple process.

  FIG. 17 is a flowchart of processing for generating an image by performing hit calculation processing based on the direction information of the player's belongings.

  Steps S31, S32, and S33 in FIG. 17 are the same as steps S1, S2, and S3 in FIG. After the processing area of the image processing is set in step S33, based on the depth information, color image information, etc. in the set processing area, as described with reference to FIG. Direction information, shape information, and the like are obtained as hit calculation information (step S34). Then, as described with reference to FIG. 13B, the trajectory direction of the bullet is set based on the direction information of the player's belongings or parts (step S35).

  Next, a hit determination process between the trajectory of the bullet and the target object is performed (step S36). Then, it is determined whether or not the bullet has hit the target object (step S37). If the bullet hits, a hit effect image is generated (step S38).

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Also, a method for acquiring direction information of a player's belongings or parts, a method for acquiring skeleton information, a rendering process based on rendering information, a hit calculation process based on hit calculation information, a position of a player's belongings or parts A method for setting a virtual camera based on information is not limited to that described in the present embodiment, and methods equivalent to these are also included in the scope of the present invention. The present invention can be applied to various games. Further, the present invention is applied to various image generation systems such as a business game system, a home game system, a large attraction system in which a large number of players participate, a simulator, a multimedia terminal, a system board for generating a game image, and a mobile phone. it can.

ISE image sensor, PL player, SC screen, TH belongings,
DTH, DTH1 to DTH3 Direction of belongings, CH character,
LS light source, DLS light source direction, RY ray, TOB target object,
SK skeleton, C0-C19 joint, RPC processing area, VC virtual camera,
100 processing unit, 102 image information acquisition unit, 104 information acquisition unit,
105 skeleton information acquisition unit, 108 game calculation unit, 110 hit calculation unit,
112 object space setting unit, 114 character control unit, 115 movement processing unit,
116 motion processing unit, 118 virtual camera control unit, 120 image generation unit,
122 light source setting unit, 130 sound generation unit, 160 operation unit,
170 storage unit, 171 image information storage unit, 172 color image information storage unit,
173 depth information storage unit, 174 skeleton information storage unit,
175 Object data storage unit, 176 Motion data storage unit,
177 Model data storage unit, 178 Light source information storage unit, 179 Drawing buffer,
180 information storage medium, 190 display unit, 192 sound output unit, 194 auxiliary storage device,
196 Communication Department

Claims (23)

An image information acquisition unit for acquiring image information from the image sensor;
An information acquisition unit that performs image processing based on the image information to acquire at least one of direction information, color information, and shape information about the player's belongings or parts as rendering information;
As an image generation unit that generates an image by performing a rendering process based on the rendering information that is at least one of the direction information, the color information, and the shape information,
A program characterized by causing a computer to function. In claim 1,
The image generation unit
Based on the rendering information, the light source information of the light source arranged and set in the object space is set, the rendering process using the light source in which the light source information is set is performed, and the image seen from the virtual camera in the object space A program characterized by generating In claim 2,
As a character control unit for controlling the character corresponding to the player,
Make the computer work,
The information acquisition unit
Based on the image information from the image sensor, obtain skeleton information that identifies the action of the player visible from the image sensor;
The image generation unit
A program for specifying an arrangement position of the light source in the object space based on the acquired skeleton information and position information of the character. In any one of Claims 1 thru | or 3,
The image information acquisition unit
As the image information from the image sensor, depth information in which a depth value is set at each pixel position is acquired,
The information acquisition unit
A program characterized in that, based on the depth information, at least one of the direction information and the shape information of the possessed item or the part of the player is acquired as the rendering information. In claim 4,
The information acquisition unit
Based on the depth information, a plurality of first to Nth direction information (N is an integer of 2 or more) is acquired as the direction information of the rendering information,
The image generation unit
A program that performs rendering processing based on the first to Nth direction information that is the rendering information, and generates an image. In claim 4,
The image generation unit
A program that sets a rendering range based on shape information of the possessed item or the part acquired based on the depth information, and performs the rendering process in the set rendering range. In any one of Claims 1 thru | or 6.
The information acquisition unit
Based on the image information from the image sensor, skeleton information that identifies the action of the player visible from the image sensor is acquired, and based on the acquired skeleton information, the possession or the part of the player A program for acquiring the direction information. In any one of Claims 1 thru | or 7,
The information acquisition unit
Based on the image information from the image sensor, obtains skeleton information that identifies the action of the player that is visible from the image sensor, and based on the obtained skeleton information, obtains the rendering information. A program characterized by setting a processing area for image processing. In any one of Claims 1 thru | or 8.
As a virtual camera control unit that controls virtual cameras placed and set in the object space,
Make the computer work,
The information acquisition unit
Obtaining position information of the possession or the part of the player;
The virtual camera control unit
Based on the position information of the belongings or the part, the virtual camera is arranged and set in the object space,
The image generation unit
A program for generating an image visible from the virtual camera in the object space. In claim 9,
The virtual camera control unit
The virtual camera is arranged at a position corresponding to the position information of the possessed object or the part, and control is performed to direct the direction of the virtual camera in a direction corresponding to the direction information of the possessed object or the part. A program characterized by In claim 9 or 10,
The information acquisition unit
Based on the image information from the image sensor, obtains skeleton information that identifies the action of the player visible from the image sensor, and based on the obtained skeleton information, the position information of the belongings or the part The program characterized by acquiring. An image information acquisition unit for acquiring image information from the image sensor;
An information acquisition unit that performs image processing based on the image information, and acquires at least one of direction information and shape information about a player's possession or part as hit calculation information;
A hit calculation unit that performs a hit calculation process based on the obtained hit calculation information;
As an image generation unit that generates an image based on the result of the hit calculation process,
A program characterized by causing a computer to function. In claim 12,
The hit calculation unit
A program for performing the hit calculation processing on a target object based on a trajectory direction specified by the direction information of the hit calculation information. In claim 12 or 13,
The image information acquisition unit
As the image information from the image sensor, depth information in which a depth value is set at each pixel position is acquired,
The information acquisition unit
A program characterized in that, based on the depth information, at least one of the direction information and the shape information of the possessed item or the part of the player is acquired as the hit calculation information. In claim 14,
The information acquisition unit
Based on the depth information, a plurality of first to N-th direction information (N is an integer of 2 or more) is acquired as the direction information of the hit calculation information,
The hit calculation unit
A program for performing the hit calculation processing based on the first to Nth direction information as the hit calculation information. In claim 14,
The hit calculation unit
A program characterized in that a hit range is set based on shape information of the possessed item or the part acquired based on the depth information, and the hit calculation process is performed based on the set hit range. In any of claims 12 to 16,
The information acquisition unit
Based on the image information from the image sensor, skeleton information that identifies the action of the player visible from the image sensor is acquired, and based on the acquired skeleton information, the possession or the part of the player A program for acquiring the direction information. In any of claims 12 to 17,
The information acquisition unit
Based on the image information from the image sensor, acquires skeleton information that identifies the action of the player visible from the image sensor, and acquires the hit calculation information based on the acquired skeleton information. A program for setting a processing area for the image processing. An image information acquisition unit for acquiring image information from the image sensor;
An information acquisition unit that performs image processing based on the image information to acquire position information about a player's belongings or parts;
A virtual camera control unit for controlling a virtual camera arranged and set in the object space;
As an image generator,
Make the computer work,
The virtual camera control unit
Based on the position information of the belongings or the part, the virtual camera is arranged and set in the object space,
The image generation unit
A program for generating an image visible from the virtual camera in the object space.   A computer-readable information storage medium, wherein the program according to any one of claims 1 to 19 is stored. An image information acquisition unit for acquiring image information from the image sensor;
An information acquisition unit that performs image processing based on the image information to acquire at least one of direction information, color information, and shape information about the player's belongings or parts as rendering information;
An image generation unit configured to generate an image by performing a rendering process based on the rendering information that is at least one of the direction information, the color information, and the shape information;
An image generation system comprising: An image information acquisition unit for acquiring image information from the image sensor;
An information acquisition unit that performs image processing based on the image information, and acquires at least one of direction information and shape information about a player's possession or part as hit calculation information;
A hit calculation unit that performs a hit calculation process based on the obtained hit calculation information;
An image generation unit that generates an image based on the result of the hit calculation process;
An image generation system comprising: An image information acquisition unit for acquiring image information from the image sensor;
An information acquisition unit that performs image processing based on the image information to acquire position information about a player's belongings or parts;
A virtual camera control unit for controlling a virtual camera arranged and set in the object space;
An image generation unit,
The virtual camera control unit
Based on the position information of the belongings or the part, the virtual camera is arranged and set in the object space,
The image generation unit
An image generation system generating an image visible from the virtual camera in the object space.